Photo by Gilpin2010 on Flickr.
People make a lot of decisions based on school test scores. Parents select schools for their children. Administrators fire principals and close schools. But few realize that they are using the wrong scores to make these important decisions.
Most people use the “percent proficiency” score, which measures what percentage of the students in a school are proficient in math or reading using the DC CAS test. They should also be using the Median Growth Percentile (MGP) which is based on the CAS scores. The Office of the State Superintendent of Education (OSSE) calculates MGP, but few people look at it.
That can cause problems. Recently the Public Charter School Board actually closed one of the top performing schools in the city based on MGP, because its percent proficiency scores were too low. Where does your local school rank based on MGP scores? See below.
Proficiency measures demographics, MGP measures value added by a school
Most parents compare schools by looking at the percentage of students in the school who tested proficient in math and reading. It’s understandable why they do that, as DC Public Schools prominently displays this information on their web site.
The DCPS web site shows that 85% of students tested proficient in math and 82% tested proficient in reading. Deal ranks as having the 10th highest test scores in the city, the highest of any middle school.
But what does that mean? How much of this comes from actual great teaching at Deal, and how much from the fact that Deal draws from some of the most affluent parts of the District?
What really matters is how much a school helps its students advance. Percent proficiency doesn’t tell you this on its own. For example, Janney Elementary feeds into Deal Middle School. 90% of students at Janney test proficient in math and 93% test proficient in reading. Does that mean Janney students are going downhill when they attend Deal? Probably not.
What would happen if a housing development opened next door to Deal, and the city attached a sizable affordable housing requirement to the development which drew some families with middle schoolers who were previously farther behind? Deal’s proficiency percentages would go down the following year. Does that mean that the quality of instruction got worse at Deal? Probably not that, either.
These scenarios illustrate the problem with using static data to try to understand the quality of instruction at a school. What you want to look at is longitudinal data.
Longitudinal data tracks the performance of the same students over time, to measure the value added by different schools and classrooms over the entire schooling of a student. MGP uses longitudinal data.
The National Academy of Sciences argues against using the most prominent source of static test scores — the federal NAEP test — to draw conclusions on the causal effect of school reforms for exactly this reason.
Yet nearly every piece of advocacy research arguing for or against school reform makes the mistake of using static data. That’s why they draw different conclusions using the same data, leaving parents confused and frustrated.
Ranking DC schools by MGP scores reveals some surprises
Here’s how MGP works. If a school has an MGP of 60, that means that the students in that school scored better than 60% of the students citywide who had similar test scores in previous years.
The top 5 schools by MGP are not in the top 5, or even the top 10, when ranked by percent proficiency.
|School||MGP Rank||MGP: Math||MGP: Rdg.||% Prof Rank||% Prof: Math||% Prof: Reading||Grades||Ward|
|Thurgood Marshall Academy PCS||#1||81.4||75.1||#19||81.4||75.1||9-12||8|
|KIPP PCS: AIM||#2||75.7||68.4||#24||85.0||59.3||5-8||8|
|DC Prep Edgewood MS||#4||76.3||65.3||#15||88.8||71.9||4-8||5|
|KIPP PCS: KEY||#5||71.7||69.0||#23||78.4||66.7||5-8||7|
Deal Middle School, the 10th ranked school by test scores and top ranked middle school, ranks 38th by MGP. Deal has a math MGP of 59.9 and a reading MGP of 58.5.
Does that mean that KIPP middle schools are better than Deal? Many parents would say no. And MGP doesn’t measure everything important about a school on its own. But this reveals how parents often use test scores — as an indicator of the quality of the other students, not the quality of the instruction.
Public Charter School Board closes 2nd best school in DC based on 2012 MGP
Septima Clark charter school had the 7th highest MGP in the city in 2012. The all-boys elementary school in Anacostia had a 2012 Reading MGP of 77, the 2nd highest in the city.
That means that Septima Clark students scored better than 77% of students in the city whose scores in the previous year were the same as the previous year’s scores of the Septima Clark’s students.
However, the Public Charter School Board (PCSB) ranked Septima Clark as a “mid-performing school” in its Charter Performance Report. The PCSB indicated that it would close the school, which forced Septima Clark to merge with Achievement Prep. One parent told ABC, “Either the board was misinformed, had no idea what was going on or just deliberately did not care”.
When asked why they closed one of the top-performing schools in the city, PCSB Executive Director Scott Pearson said “Growth is one of many important indicators of school quality, but we caution the use of it in isolation”. Pearson pointed to a high churn rate of students at Septima Clark, which indicates to him that parents are dissatisfied.
Pearson said that the PCSB “weighs growth as a factor, along with proficiency, attendance, re-enrollment, and whether students can read by third grade (a predictor for future successes such as high school graduation and college completion). We were pleased to see Septima Clark PCS had a strong showing in growth last year, but previous year’s growth scores were not as strong, and its proficiency is one of the lowest in the city.”
MGP scores should be a larger factor in assessing schools
OSSE has provided MGP scores for 2 years. Nonetheless, advocates on all sides of the education reform dialogue continue to use non-longitudinal data to assess the outcomes of school reform initiatives. And DCPS, PCSB and OSSE continue to prominently display percent proficiency scores of each school on their online report cards.
All three agencies should modify their online report cards to prominently display MGP, and explain it in layman’s terms. It’s not complicated. And people make very important decisions based on these scores.
Journalists should also ask advocacy researchers why they use static data when longitudinal data is available. Just like journalists note the margin of error of studies that they report, they should also note when advocacy research relies on non-longitudinal data.
Wanna see where your local school ranks? Here’s the entire list (XLSX) of DC schools, ranked by the average of 2011 and 2012 MGP. How does your child’s school or your neighborhood school rank, and what does this tell you?