On June 22, the MCPS Board of Education received a briefing from staff led by Maria Navarro the Chief Academic Officer about the kinds of assessments students are given in different grades and different schools. This review was undertaken at the direction of the Maryland State Superintendent of Education, Lilian Lowrey who asked Maryland school districts to look for options to reduce the assessment burden on the students and the system. This is important work that is worth attention and follow-through.
While presumably other Maryland districts are going through this exercise, the MCPS staff presented a comprehensive and polished briefing with various options that are being considered at different levels of the system. This is an important area for the individual schools and also for the ways that MCPS is managed. The only people who think that educational assessments are easy are those who have little experience with them. They are hard and it was clear that the team that briefed the Board on June 22 was aware of how hard this area is to consider comprehensively. In some cases, what MCPS is considering could be very important for the system and should be encouraged. Some areas raised questions for me and in a few area MCPS may want to think about how their ideas might be implemented given some of the changes going on in the field. This is not to say what was presented was wrong, but rather that the directions articulated by the MCPS staff relate to other movements in the field.
The observations, recommendations, and questions that follow are given with the recognition that this review seems very well done and well ahead of many of the majority of school districts in the United States: large and small.
Three Notable Shifts
In this work I see MCPS is tapping into three important shifts in the field of educational data. These shifts have not always been highlighted by the staff as special, but MCPS’ process of evaluating different options for reducing the testing burden is revealing ideas that align with some new movements in educational research and policy.
Shift to more continuous and embedded nature of assessments
In some of the recommendations for restructuring middle school assessments the idea of distributing the exams across multiple class sessions rather than in a single summative block was proposed. This is like what we see in new research into using the data feeds from digital technologies that can provide multiple data points (some from assessments and many from other activity) that can be used quickly and on a more immediate basis. If MCPS does experiment with parsing out assessments over multiple periods then they will be preparing for this future. At the same time, this shift bring’s complications. Much of the statistical practice that is used in education is based on rigid control of testing conditions. Once data is decoupled from the end-measures and made more continuous then how the information can be used (it’s reliability) for important consequences changes as well. These intermediate/continuous measures are much harder to use for high-stakes because they are usually less statistically reliable.
Focus on range of student experiences
Maria Navarro, the Chief Academic Officer discussed the use of different kinds of data, including surveys to understand the “range of student experiences” in response to comments by Dr. Docca who was asking how these changes would impact children of color. Many educators have long recognized that learning performance is only one important factor to attend to. What is newer now is that people in the data movement are developing different kinds of measures of student experience and that the kinds of surveys that MCPS and other districts are utilizing today, including Harvard’s Tripod project (http://tripoded.com/), are getting at these kinds of data. The coming years should see more and more of these kinds of data and the response of Dr. Navarro may be repeated in future meetings and conversations about different kinds of students. The Board should continue to ask for more data about the range of student experiences beyond test scores. I believe that it is in this kind of information that the solutions to these challenges will likely be found.
Alignment and Coherence
Dr. Lang, the Associate Superintendent for Curriculum and Instruction discussed the issue of alignment of assessments. It was presented as alignment in terms of policies and other assessments like PARCC, but the issue is much bigger and the field is just now beginning to wake up to the challenges of connecting different kinds of assessments and other forms of information (see the first recommendation below). This is an area where educational data is different from data in other fields and where education is a more challenging field to implement data-driven solutions than other fields. While it is easy to bring two different scores for the same subject together (ex: math), ensuring that those scores are measuring the same things requires a lot of technical work. This is an increasing challenge for education where more and more technologies that are easy to access such as Khan Academy of Dreambox Learning that can provide data on student work and show that data in tables, charts, and dashboards. The visual displays don’t tell educators whether or not two data points can be comparable or how they relate. This has contributed to some educators distrusting educational data because this lack of alignment translates into incoherence in terms of what to do with students.
Three Recommendations
I have three recommendations for MCPS in this process and in going forward with their work on assessments and data.
Broaden concept of data
In a number of places in the briefing the term data and assessments were used interchangeably. This is clearly consistent with most ways policies use these terms, but this needs to change. Assessments have been seen both in education school and policy circles as the most valid data—the hard data—that is needed to evaluate educational performance. The problem is assessment data is widely variable in it quality and in how clearly represents the things that matter most like instruction. In many parts of the curriculum (ex: social studies, life science, advanced literacy) there are major gaps in quality assessments.
The field has begun to broaden its conception of data to still feature assessments in a central role, but to acknowledge that other kinds of evidence from surveys, from digital tools, and especially artifacts that teachers also use in their daily work. These teacher tools are also developing very quickly and so the kinds of information they can produce and how it can be analyzed is developing rapidly.
In broadening the kinds of evidence considered as core data beyond assessments, the ways the information are used can also be thought of broadly. Several years ago the use of data and other information in educator conversations was less valued than it is today. A few years ago, the idea of data-driven decision making and using test results to directly inform instruction was more popular than it is today largely because these ideas of using assessments as a primary lever of reform have not worked. Assessment data is rarely at a granularity and specificity that can be used in the practice of teaching. Rather, these data can help teachers see their work at the level of a school and the long-term trajectories of students. Professional learning communities looking at a range of information, including the assignments themselves and examples of student work alongside assessments are considered structures that can actually enable meaningful change. The idea of the central office collecting assessments from the schools as was suggested in the June 22 meeting may seem to some educators a case of central office controlling local processes. However, if the view of assessments is broad (ex: beyond test scores) and if the uses are also broad (ex: to understand practice variation as well as measure student learning) then those on the front line may have less concerns and see this sharing of information about assessments as part of organizational learning and professional development.
Guard Against Unreasonable Expectations
Many people looking at the use of assessments and data as levers of change in education hold very logical, simple, and often unworkable impressions of what these tools can do. Assessments are critical and important, but even the state of the art assessments are often weak when it comes to making change. They must be used in concert with coherent commitments and organizational approaches to ensuring all students learn. As MCPS moves forward with reviewing their assessment strategies, policymakers should be realistic about the limits of this part of the system. There are no silver bullets in education and assessments are a problematic area.
Network with other districts
MCPS is often a few steps ahead of the rest of the nation, but the work they are doing to come up with better ways to use assessments are not unique to this county. Almost all districts should be looking into this and many large ones in Maryland are as well. I recommend MCPS connect with these others and perhaps develop a common conversation group to share ideas, concerns, and best practices.
Three Questions
This effort by the Board and administration has been one of the more important initiatives in my view because assessments are deeply connected to just about everything that is in the core of school system functions. It doesn’t involve busses and building services, but almost anything involving teachers and school leaders is increasingly related to assessment. I offer three questions.
How will this be sustained?
Once the school year begins and after the task for the state superintendent is completed, how will this reflection be sustained? How will this not become another historical exercise and instead a part of a continuous improvement process. As groundbreaking as this kind of review is (well begun with its broad involvement of stakeholders) the next steps may be more complicated and require leadership because they will involve making changed. As Mr. Barclay noted on June 22, MCPS is without a full-time permanent superintendent. How will MCPS continue while searching for a new leader and how will the next superintendent continue this initiative into the next school year and beyond? It would seem the board will need to continue to focus on this topic and maintain an engaged dialog because the topics will remain complex and connected to all the important work MCPS does. It is possible that the Board itself will need to take on some of the work that might otherwise be part of the job of a superintendent.
Where are The Assessments Professionals?
The review that was presented on June 22 showed broad stakeholder communications with the inclusion of the union and various formal and informal structures. This is important not only because the success of any effort this large will benefit from buy in, but also because these stakeholders have important perspectives that can really inform the system’s leadership. At the same time there are others who specialize in assessments, including assessments like the PARCC and MSA, who could help MCPS understand the implications of possible decisions. The University of Maryland and George Washington University are both close and include experts in measurement and assessment who might be able to inform the planning for next steps. How is MCPS taking advantage of these local experts in their process of review?
Where are Technology and Shared Accountability Offices?
One of the features of most large school systems (like many companies of old) is that they tend to be walled off into different groups or silos with stronger internal communication than external. This can lead to inefficiencies and lack of a coherence as each unit pursues its own agenda. Strong top-down leadership can get different groups marching to the same beat, but often this comes with downsides, especially in socially oriented organizations like school systems.
The June 22 briefing was led by the Chief Academic Officer and Associate Superintendent for Curriculum and Instruction. This makes sense because this part of the system is responsible for instructional tools and planning. To treat these matters as technical matters first might not put the initiative on as strong an instructionally-relevant footing, which needed. At the same time, the Office of Shared Accountability, which manages the testing process and has many of the technical experts with statistical knowledge important for interpreting assessments, should also be involved in this process. Similarly, the Chief Technology Officer could have an important role in the shaping of new solutions as many of the newer kinds of assessments that can provide more relevant and actionable information run on new platforms that the technology office will manage. At some point, these two areas of MCPS will need to be principally engaged in any future discussions of MCPS assessments.
An Important Opportunity to Learn From
The MCPS BOE is right to follow through on this discussion and to ask staff to continue to make it a priority. It is important for MCPS’ future. MCPS is like just about every other US district today that has a patchwork of assessment systems that the senior staff who presented this briefing appropriately classify as those that are A) externally required and B) that the district has some control over. They also correctly describe the important interrelationships between these data and related areas of practice such as communications, grading, scheduling, and aligning with state and federal policy initiatives. MCPS is unlike many districts in the way they went about this review. While the trigger for this review seems to have been the Maryland State Department of Education’s (MDSE) interest in lowering the testing burden on students and teachers, this review is an important opportunity to look at one of the more challenging and important aspects of the district’s operation. School systems are run by a combination of procedure, intuition/professional judgment, and data. This combination forms a kind of “nervous system” for the districts and assessments often are the most important visible part of this nervous system. Assessments are more than an end measure of student learning, they touch just about every aspect of the work that gets done in MCPS from policy evaluation to teaching.
This kind of review has little precedence in school reform. Very few districts like MCPS have undertaken this kind of evaluation and so there few examples of how to do this. MCPS’s approach has included a combination of information gathering from inside and outside the system as they have begun to frame several different kinds of approaches to reducing testing to free up instructional time. It is comparatively easy to look professional in a routine and practiced task, but much more challenging when the challenge is new. My view is that te MCPS staff have taken strong steps this new area and in the process shown a mature process of developing a set of alternatives by iterating through input and analysis. Most school systems do not have the capacity for this kind of exploratory work. Other districts embarking on a similar review would have a lot to learn by watching how MCPS has approached this task.