Schemo gets pwned

See my summary post in which I try to roll all the updates into a coherent post.

**Update** The scandal grows. It turns out that the Reading First schools in Madison underperformed the non-Reading First schools by almost a 1/4 standard deviation. This is an educationally significant difference. (Clarification: I'm using "Reading First schools" to designate the schools in Madison that were eligible for reading First funding but didn't get it because Madison decided to use their balanced literacy program instead of a Scientifically based reading program.)

In part one of this post, I showed you the embarrassing horror show that is taking place in the Madison school district that has hapless first graders guessing at the word "pea" and thinking the word is "pumpkin." If it weren't embarrassing enough having that displayed in the opening paragraphs of this NYT article, the Madison school district has the audacity to claim that their reading program is actually boosting student performance.

Call it the $2 million reading lesson.

By sticking to its teaching approach, that is the amount Madison passed up under Reading First, the Bush administration’s ambitious effort to turn the nation’s poor children into skilled readers by the third grade.

...

Madison officials say that a year after Wisconsin joined Reading First, in 2004, contractors pressured them to drop their approach, which blends some phonics with whole language in a program called Balanced Literacy. Instead, they gave up the money — about $2 million, according to officials here, who say their program raised reading scores.
...

Under their system, the share of third graders reading at the top two levels, proficient and advanced, had risen to 82 percent by 2004, from 59 percent six years earlier, even as an influx of students in poverty, to 42 percent from 31 percent of Madison’s enrollment, could have driven down test scores. The share of Madison’s black students reading at the top levels had doubled to 64 percent in 2004 from 31 percent six years earlier.

And while 17 percent of African-Americans lacked basic reading skills when Madison started its reading effort in 1998, that number had plunged to 5 percent by 2004. The exams changed after 2004, making it impossible to compare recent results with those of 1998.

No it didn't. It should have taken Diana Jean Schemo, this article's author, about half an hour on the internet to figure out that the "Madison officials" were spinning the "reading scores."

NAEP scores for Wisconsin show that the number of proficient students was 34% percent in 1998 and dropped slightly to 33% in 2003 and stayed there in 2005, the last time fourth graders were tested for reading. Students scoring at the basic level dropped from 69% to 67% during this same period. From the period 1992-2005, the achievement gap between black and white students rose from 28 points to 33 points and the gap between poor and non-poor students dropped slightly from 28 points to 25 points. So, NAEP shows us that the reading proficiency of Wisconsin fourth graders has basically remained flat since about 2000. (Go here and select Wisconsin as the jurisdiction.)

The Wisconsin Reading Comprehension Test (3rd grade) tells a different story. The number of proficient students in Wisconsin rose 22.5 points from 64.9% in 2000 to 87.4% is 2005. Based on our knowledge of NAEP scores for the same population, we know that this gain was imaginary. What Wisconsin did was a combination of making the test easier and lowering the cut score so that 22.5% more students would be able to pass it. The result is that the mean shifted by about +0.77 of a standard deviation.

Gather 'round, kids, and learn how Wisconsin made a fool out of the statistically illiterate Diana Jean Schemo.

Here's what I get from comparing the 1998-99 scores to the 2004-2005 scores:

In 1998 the percent of the state passing the exam was 64.9%. In 2005, it was 87.4%. A shift of 0.77 standard deviations (s.d.). Remember, during this same period NAEP scores for the state remained virtually unchanged.

In 1998 the percent of the Madison school district passing the exam was 58.9%. In 2005, it was 82.7%. A shift of .72 s.d. If anything, Madison underperformed the state during this period.

Update Three: From this source, I found out that Madison's Reading First Schools (the schools eligible for RF funds, but which never go them because Madison made them use a balanced literacy program) are Glendale, Hawthorne, Lincoln, and Orchard Ridge. As it turns out, the average gains made by these schools was only 21.6% or 0.56 s.d. In contrast, the non-Reading First schools gained 25.4% or +0.80 s.d. (Again, the percentile scores are deceptive because the Reading First schools caught the fat part of the curve, whereas the non-RF schools did not.) To put this in perspective, the 0.24 s.d. differential is about the same effect size found in project STAR (class size reduction) which educators rave about. So, the non-Reading First schools in Madison slightly overperformed the state average while the Reading First eligible schools significantly underperformed the state avergae.

I didn't find disaggregated data, so I can't determine if the stats for black students given to Schemo is accurate. But assuming they are, the rise in scores from 31% to 64% represents a shift of +0.86 s.d. That seems highly improbable considering 1) that Madison's overall rise was only +0.72 (this means that the scores for non-black in Madison rose only about +0.58, a significant underperformance compared to the rest of Wisconsin) and 2) black-white achievement gap on NAEP actually increased during this period, meaning that black performance actually declined.

Update: Rory finds the disaggregated data. In 2005, black performance in Madison was 57% which represents a shift of +0.68 s.d. which represents a underperformance of both Madison (+0.72 s.d.) and Wisconsin (+0.77 s.d.).

What actually happened was something like this:

Wisconsin raised the mean passing level from level 1 to level 2. The top curve shows black performance. This shift captured the fat part of the curve for black students (31% to 64%); picture level 1 being even further to the left. The blue shaded portion represents the increase in black performance.

Update two: Just in case it isn't clear, Madison's claim that black student performance "doubled" and exceeded the gains made by other students based on percentile changes is highly mispleading and a misuse of statistics. When you measure gains by the more statistically accurate standard deviation it is readily apparent that blac student performance was no greater than the gains made by other groups.. In fact, it appears to be considerably less. The achievement gap remains unchanged or may have even grew. This is what you would expect to happen under a whole language reading program because those students at the bottom of the curve are the ones most damaged by the non-explicit instruction.

The bottom curve shows white performance. The shaded portion between level one and level two shows the gain. I'd estimate that the white performance rose from about 69% to about 90%. (If anyone finds the disaggregated data let me know.) As you can see, white performance was already past the fat part of the curve--the average white student was passing the exam back in 1998.

Update: White performance in Madison was actually 93%. Ka-ching!

One thing is for certain, Madison's balanced literacy program did not cause the imaginary gains made by Madison school children.

Update: Mark Liberman of Language Log chimes in:

DeRosa's argument seems pretty persuasive to me. If he's right, then I'd consider a different evaluation -- was Schemo bamboozled by the Madison school authorities, or did she pick Madison, and spin the story as she did, in order to make an essentially dishonest point, suggesting that the mean old federal bureaucrats are trying to stop the dedicated local educators from continuing to use the methods that are helping their children so much?

I certainly don't want to discount the possibility that Schemo was a willing dupe. It could very well be that she went looking for a story and Madison told her what she wanted to hear.