Good Stats     Bad Stats

The Atlanta Journal Constitution launched round two in its investigation into cheating on school achievement tests. The best that can be said about the piece is that they found an expected correlation and decided that was evidence of school cheating.

Last month I described how they had studied school test scores and conclude that the evidence indicated widespread cheating in school testing nationwide. They tried to soften that statement, but came out strong that they had no other viable explanation for the numbers they were seeing. At that point I asked several questions about the study and expressed a number of concerns with their analysis. The probabilities they calculated were so small there really could not be any other explanation.

They were wrong, and they are wrong again with their most recent analysis. Last month they flagged the five percent of school districts that were identified as outliers in their statistical methodology. As I said then, five percent would have been identified regardless of any underlying causes for the differences. The methodology was designed to flag five percent of the school districts. They next computed probabilities under what statisticians would call the “null hypothesis” which in essence assumes there are no differences between the school districts. This is wrong. Every statistician, every school administrator and every reporter should know that there are real differences between school systems, between schools, and there are changes over time. None of these were apparently considered in the AJC calculation. Once those differences are recognized the calculation of the probabilities of the outlier events becomes much more complicated and nearly impossible to compute.

I say apparently as I sent an email to the folks at the AJC after their first report asking questions about their methodology and when the data, models, and analysis would be freely distributed. I have yet to receive an answer. It has been a month since I sent that request. As a result I have to make reasonable assumptions about their methods from the incomplete descriptions of they methods the have provided at their website.

In the most recent piece the AJC looked at test scores for “Blue Ribbon Schools.” These are schools singled out by the Department of Education for significant improvement. In today’s language “Duh.” What did you expect AJC? Of course they are the schools that showed up in your analysis. The existence of a correlation between the AJC numbers and the awards from the Department of Education is obviously an expected outcome. The AJC choose to single out Highland Elementary in Silver Spring Maryland with serious questions about the integrity of the testing at that location. The paper seems to have ignored comments by the school distinct that extra funding was a big cause of the improvements at the school. See local media descriptions of the responses by Montgomery County Maryland officials here and here. The extra funding is exactly the kind of thing that invalidates the probability calculations that the AJC relied on in its analysis. The Montgomery County Maryland school superintendent Dr. Starr has provided a response to the the AJC article with some details on the reasons for the improvements at Highland Elementary.

Perhaps telling is Maureen Downey response to comments at her Get Schooled Blog at the AJC where she says:

Last response as this is getting idiotic. Your argument appears to be that while the AJC’s investigation and data analysis got it right with APS, we are getting it wrong with the other school districts — even though we used the same methodology. That somehow our results for those other school districts cannot be accurate even though our results were accurate for Atlanta?…

Perhaps she is getting irritated. More importantly she gets her logic and her statistics wrong. Some of this is Basic Logic 101. She is saying the methodology got the story right once in Atlanta therefore it must be right the second time. This is equivalent to saying I flipped a coin guessed heads and it was head therefore when I flip it a second time and guess heads again it must without a doubt come up heads again. Being right once, no matter what statistical methods were used, is not guarantee that the same method will get it right a second time. I must also point out that they did not use the same methodology. The first time they looked at one school system, the second time they look at over 3,000 school systems and picked out the outliers. Those are not equivalent methodologies.

Maureen Downey goes on to say:

Among all Blue Ribbon schools with suspicious scores, the analysis identified 27, including Highland Elementary, that had the most unlikely gains. In some grades and subjects, the odds against increases occurring without an intervention such as tampering were so high as to be virtually impossible.

The thing is she is wrong and she is right even if her probabilities were correct. The problem is there are a host of other interventions other than than tampering that can provide a good explanation of what is going on. There is no compelling reason to single out “tampering.” But the AJC seems to be unwilling to entertain any other possible explanation.