Photo of standardized test from Shutterstock.

Standardized test scores in DC have risen significantly in the seven years since schools came under mayoral control, according to a recent study, and it’s not just because of an increase in affluent students. But while math scores have gone up steadily, literacy scores have largely stalled after an early jump.

While DC officials have touted increases in test scores as a sign that education reforms are working, critics have argued that DC’s changing demographics are behind the improvements. They say an influx of more affluent students has driven up the scores while the gap between those students and lower-income minority students has remained as wide as ever.

But a recent independent study concludes that low-income and minority students have improved their scores as well. Controlling for factors like race and income, it concludes that less than 10% of the increase in overall scores is due to DC’s changing demographics.

A division of the American Institutes for Research called CALDER did the report, which is one of a series evaluating the effects of DC’s education reform efforts since the school system came under mayoral control in 2007. The statute that abolished DC’s local school board and handed control to the mayor also required independent assessments of how the new regime was working.

The report on student achievement concluded that more affluent DC students had larger test score gains than low-income ones, which were defined as students receiving reduced-price lunch. And more affluent black students improved more than low-income ones.

On the other hand, improvements among black and Hispanic students were larger than those for white students, probably because they had more room to grow.

But when researchers controlled for the effects of differences like race and income, they found increases across all categories, especially in math.

Proficiency rates are different from actual test scores

How can that conclusion be squared with claims that scores for poor and minority students have remained stagnant or gotten worse? It depends on whether you look at proficiency rates or actual test scores.

After students take DC’s standardized test, the DC CAS, their scores put them in one of four categories: below basic, basic, proficient, and advanced. Usually what’s reported is the “proficiency rate,” which is the percentage of students who have scored in the proficient or advanced categories.

The proficiency rate can be useful in highlighting disparities between schools. But it overlooks students who have moved up from below basic to basic, or who have improved their scores but not enough to move up from one category to the next.

The CALDER report was able to capture those changes because researchers looked at actual scores rather than categories. According to Umut Ozek, the report’s lead author, that approach provides a more accurate picture of student growth.

A portion of DC students also take another test, the National Assessment of Educational Progress (NAEP), every year. That test found no significant reduction in DC’s achievement gap in 2013, but Ozek says the NAEP’s samples of racial and economic subgroups are very small, which makes its findings questionable.

Ozek’s team also concluded that both DCPS and the charter sector saw roughly similar gains in test scores. Those results remained the same when the team focused just on low-income students in each sector, Ozek said.

There are caveats to reading scores, as well as other research limitations

The report included a number of caveats, including one about gains in literacy scores. About 50% of the increase in reading scores occurred in the first three years the study covered, from the 2006-07 school year to the 2008-09 school year.

Those years were not only the first years the schools were under mayoral control, they were also the first years that students took the DC CAS. And, Ozek says, it’s possible the reason for the jump was that teachers and students were adjusting to the new test. When the study team excluded the first year of results from their analysis, the rate of growth became flatter, providing some evidence for the “adjustment hypothesis.”

In other words, once teachers figured out what the test was looking for, they were able to better prepare students to take it. But the lack of improvement in later years suggests that teachers and students have hit a wall. The report also cautions that there have been allegations of cheating on the DC CAS.

On the other hand, Ozek says the report’s findings are bolstered by similar results on the NAEP tests. The NAEP is widely regarded as cheat-proof and difficult to prepare for.

Other caveats in the report include the fact that test scores provide only an approximation of actual student learning. And to the extent that the scores do show that reform has been working, the study can’t tell us which of the various changes since 2007 are responsible for the improvement.

We need a new approach to literacy

One academic connected with the study has argued that the results indicate that DC has generally been heading in the right direction. While that may be true for math, the stagnation in reading scores in both the charter and DCPS sectors is a cause for concern, especially among low-income and minority students.

Generally, it’s harder to close the achievement gap in literacy for those students, probably because literacy skills largely rest on the kind of vocabulary and background knowledge that affluent students are more likely to acquire outside of school.

And while math skills are important, students who lack literacy skills are at a tremendous disadvantage when it comes to learning almost all subjects. Poor reading comprehension can even interfere with students’ ability to do math word problems.

It’s good to know that scores for low-income and minority students have gone up, even if that increase isn’t enough to show up in proficiency rates. But the stagnation in literacy scores is particularly troubling because DCPS has made literacy one of its key areas of focus.

Maybe it’s time for both DCPS and the charter sector to try something new when it comes to helping low-income students acquire the reading and writing skills that form the foundation of a meaningful education.