March 19, 2007

More on Wisconsin's Reading First Scandal

The Prof at Right Wing Nation has my back on Wisconsin's Reading First Scandal.

From these data, we cannot say that the Reading First districts did or did not increase their proficiencies. There just aren't enough data. But — and this is crucial — neither can the Reading First districts claim that they raised their proficiencies using these data. The only way we can determine whether these school systems did or did not raise their proficiencies is by analyzing the raw data, and not the aggregates by school district. In other words, Ken was right, and the journalist was wrong.

I like that last sentence so much I'm going to repeat it.

In other words, Ken was right, and the journalist was wrong.

I did a very quick initial analysis by assuming both distributions were essentially normal so I wouldn't have to fix the improperly formatted data set. RWP, an expert in data analysis, did the hard word of reformatting the data set and did a complete analysis. I re-analyzed the data and came up with the same numbers as RWP. Time permitting, I'll update my figures with the re-analyzed numbers. The effect sizes will change slightly, but my conclusions will still hold.

The important thing to remember is that RWP's analysis includes the following assumption.

We are assuming here either that the proficiency exam standards did not change between the two years or that the proficiency reports for the two years are comparable (if they are not, then Wisconsin cannot make any statement about their proficiency levels over time).

As the NAEP data clearly shows, the Wisconsin's proficiency exam standards did change between 1998 and 2005. NAEP scores declined slightly, while the Wisconsin scores magically skyrocketed. Suspiciously so.

So, when RWP says that Wisconsin and Madison can validly claim that scores have increased, this conclusion only holds if Wisconsin's proficiency exam standards didn't change between 1998 and 2005. And, as we know from NAEP scores, they did.

RWP's analysis reveals an importnt factoid.

So far, everything looks positive, until we look at the skewness. Go back to that bell curve water balloon. If you grab the tail on the right and pull it further to the right, more of the water will spill into that tail, right? That's what we call a right-skewed curve, and it has a positive skewness factor. If you pull the left tail out, more water spills into that left tail, and we have a left-skewed curve, with a negative skewness factor. When we look at the skewness for the two years, both are left-skewed — that is, in both years, there are more data in the left tails (less than proficient) — but the 04-05 curve is more left-skewed than 98-99 (-1.86 and -0.92, respectively). So even though it does look like Wisconsin may have improved the reading proficiency between the two years, they also slightly increased those who were less than proficient (if this seems like a paradox to you, think of the water balloon again, and all will be made clear).

Even though Wisconsin's scores magically skyrocketed, the number if students who were less than proficient increased slightly. Kids at top learn more, but the kids at the bottom continue to struggle. These would be the kids that Reading First aimed to serve.

I think we can finally put this one to bed, unless Madison officials decide to make more outlandish claims.


TurbineGuy said...

Would it be possible to file a Freedom of Information request to the Wisconsin Dept of Education to get the disaggreated numbers for the District?

Anonymous said...

That would be nearly impossible if you were soliciting the data from a university because of the Privacy Act of 1974 (I haven't read the legislation, so I do not know if it has scope over primary or secondary schools). It took us a year of long meetings with administration and university attorneys before we were allowed to keep a central grade book (for all sections, the problem being multiple instructors), and we could only do so if each instructor was given a bank of code numbers and replaced the student IDs with them (and deleted all personal data). When we wanted to start analyzing student performance over the two-semester course sequence, that took another year and a half of meetings before we were allowed to do so, and only with a lot of restrictions.

When I was an undergrad, faculty would post grades on the wall or office door, with just the student IDs and the grades for privacy. That became illegal after 1974.

I've seen a lot of complaints about university transparency, and I would normally agree except few people know how tightly universities are restricted by law.

When your kids to to college, forget calling the faculty member to ask about their grades or performance. If he tells you, he can be prosecuted.

TurbineGuy said...

I sent off an email to the Wisconsin Dept of Ed asking for it. I figured it couldn't hurt.