Photo by gak on Flickr.

DC students’ scores grew dramatically on national standardized tests, more than almost every state. This is good news, but it’s hard to know whether the gains actually mean we’re educating students better, or just that more wealthy families are sending their kids to DC schools.

The Washington Post editorial board isn’t hesitating to claim credit on behalf of its agenda. “School reform in the District is working. That is the unassailable message” of the news, the board writes. “The NAEP is the gold standard or, to use [US Secretary of Education Arne] Duncan’s description, ‘irrefutable.’”

I hope school reform does work, and in particular, that it actually helps struggling students from disadvantaged backgrounds get a better education. But good as the NAEP may be, there’s nothing “unassailable” or “irrefutable” about the Post’s conclusion thus far.

This is “misnaepery”

That’s because the NAEP, the National Assessment of Education Progress, just measures 4th, 8th, and 12th graders every two years. The problem is that one year’s 4th graders might be very different from the last. Since wealthier students tend to do better on standardized tests, one reason for the rise in DC’s test scores could be that more affluent kids are taking the NAEP.

Steven Glazerman coined the term “misnaepery” to describe using NAEP data to draw conclusions beyond what the scores really show.

It’s not that DC’s test scores haven’t gone up — they have. And it’s always better to have higher scores than lower ones. But it’s misusing statistics to say that this proves there’s been a change in school quality, or that any particular leader deserves the credit.

What do we know?

We don’t have data showing that the public school population in DC, as opposed to the general population, has become wealthier since 2011. But, as Emma Brown reported in the Post, “The demographics of test-takers in the District has shifted during the past two decades, with the proportion of white and Hispanic students growing as the proportion of black students has fallen.”

It’s possible that some of those Hispanic students are learners of English as a second language, the one group that failed to show gains on the NAEP. But presumably many of those white students are relatively affluent.

And while it’s good news that the scores for all racial groups rose, the gulf between those groups remains wide. As Brown wrote, “Black and Hispanic students made gains, and achievement gaps between them and white students narrowed slightly in some subjects and grade levels. But the gaps widened or remained the same in others.”

There’s no free lunch for these statistics

It’s also become more difficult to focus on the trends among students who qualify for free and reduced-price lunch (FRL), the most common method of identifying poorer kids of any ethnicity. The reason, as Brown explained, is that DC has changed its FRL rules.

Instead of requiring families to turn in forms showing household income, DC now provides free lunch to all students at any school where at least 40% of students are in foster care, homeless, or receiving welfare benefits. That could mean that some wealthier students at those schools are now lumped into the FRL category.

Brown quotes the head of the agency that administers NAEP as saying that the rule change is “masking whatever is actually happening.” The official, Brown says, “cautioned against drawing conclusions about the progress of poor children in the District based on the 2013 test results.”

Officials are touting that students who performed in the bottom 10% of test-takers have shown the most significant gains since the last round of NAEP tests. Because test scores are largely correlated with income, those are likely to be the kids at the bottom of the socioeconomic scale, which could be a ray of hope. But it’s still not possible to tell from this data whether the actual poor students are doing better, or if this is more “cohort replacement” where this year’s bottom 10% is different, and wealthier, than last year’s.

We still have the wrong data

Maybe DC schools have improved, and we all certainly hope so. But to be sure that’s the case, we need “longitudinal” data, which compares the same students from year to year instead of starting over with a new cohort. And unfortunately, that’s not what the NAEP scores give us.

With all the testing that’s happening, testing which often distracts from actual instruction, you’d think that we should be able to actually draw meaningful policy conclusions. It’s a shame that there’s so much data but not the data we need.

We do know that the number of higher-income families and students living in DC has increased. That can be good if we have the right policies to ensure that the rising tide lifts all boats instead of just a few. But we can’t simply skip that necessary step.

Scores can go up if schools get better. Scores can also go up if wealthy families push out poor ones from the city or displace poor families from high-performing schools. We need to make sure any new education policies are responsible for rising test scores rather than the change in our population before celebrating too loudly.