Do AP Exams Measure School Performance?

The Prince George’s County Maryland county executive, in his frustration over the performance of the school system has proposed to shift many of the responsibility for the school system from the elected school board to the county government. In the mist of the discussions the education reporter for the Washington post has put forth a piece arguing that the success in the Prince George’s school system on advance placement test not be lost in the discussions. He characterizes the AP testing as the “overlooked successes.”

The piece is very short on data demonstrating the successes that the reporter claims exist. He provides numbers for only two high schools in the county. For one he says:

Oxon Hill High School, 45 percent low income, jumped from 490 to 754 AP tests in that same period, and raised its passing rate from 15 to 17 percent.

The time period refers to is never adequately given in the writeup. I was surprised that anyone would be bragging about improving a passing rate form 15 to 17 percent!

Jay Mathews, the author of the piece likes to make the case that exposure to AP level coursework is significant even if the students to not pass the AP tests. He claim there has been a significant increase in the number of students taking the tests and in the number passing the tests in the county.

What do the numbers show? In the end it is clear that this is a case of using the wrong data to make conclusions while ignoring the better data that is readily available.

Let’s start with the number of test takers. Data available from the College Board web site show that nationwide the number of test takers has increased by 102 percent. In Maryland the number has increased by 122 percent. Numbers available from the Prince George’s school system show an increase of 167 percent. Thus it looks like the Prince George’s school system is improving relative to the state and the country. I have not found the key number which is what proportion of the graduating class in Prince George’s county are taking the AP tests.

Performance can not be measured just in terms of the numbers of test takers. If taking more test is an indicator of improvement then the pass rate for the tests needs to reflect that improvement. SAT scores should be increasing. If the improvements are not measurable then they likely did not happen. The county executives frustration is that performance is not where it should be by his measures. Looking at the AP pass rates I have to agree with him. In 2012 results passing grades of three or better was achieved in 2,437 tests out of the 8,951 test taken in the county for a pass rate of 27.2 percent. The comparable figures for the state was 61.3 percent and for public schools nationwide the number was 57.2 percent. The county lags far behind both. Is the situation improving as claimed by Jay Mathews? The comparable numbers for 2007 were 34.7 percent for the county, 62.2 percent for the state and 57.2 percent for the nation. The answer is a resounding no! Even more disconcerting are the scores on the individual AP tests. In the Calculus AB test 85 percent of the scores were a one. This compares to 24.9% at the national level. In fact at the national level 30.6% receive a score of five. In a field near and dear to my heart 70 percent of the scores on the statistics test were a one. The comparable number for the nation was 12.2%. (As a note the nation numbers and some of the state numbers are posted by the College Board at their website. The Prince George’s school system emailed me copes of their 2007 and 2012 reports. These reports show more detail on the scores among the test takers. They are apparently not available online.)

Perhaps AP test scores are not the best measure of improvement. The county also tabulates SAT scores as well in their annual report. The mean critical reading score declined from 446 in 2002 to 433 in 2012. The mean mathematics score has declined from 441 to 422 in the same time period. Certainly if Jay Mathews is correct and students are benefiting from exposure to the AP classes then the results should be showing up in the SAT scores. That is apparently not happening.

The current system has clearly not brought improvements. The proposed changes may not work either. But clearly changes to the system are justified.