December 10, 2007

Winerip Gets Snookered by Smallest School Study

The NY Times needs to hold a seminar on introductory statistics for its reporters, especially its education reporters, so they stop turning in articles like this one one by Michael Winerip which seriously confuses correlation with causation.

It is ETS that deserves the real blame, however, for taking advantage of the statistically gullible, i.e., Winerip. The reason they deserve the blame is because they were peddling their statistical survey "study" as implying causation: From the report Highlights, paragraph one:

The family and the home are both critical education institutions where children begin learning long before they start school, and where they spend much of their time after they start school. So it stands to reason that improving a child’s home environment to make it more conducive to learning is critical if we are to improve the educational achievement of the nation’s students and close the achievement gaps. To do this, we need to develop cooperative partnerships in which families are allies in the efforts of teachers and schools. The kinds of family and home conditions that research has found to make a difference in children’s cognitive development and school achievement include those
highlighted below. (emphasis mine)

The highlighted sentence is the one that lets Winerip plead incompetence, when he gets accused of having a political agenda. Here's what happened.

ETS collects a lot of student data. Once in a while they sift through the data to find the typical unsurprising correlations.

  • students from single parent families on average tend to underperform academically.

  • students from poor (resource deprived) families on average tend to underperform academically.

  • students from language-deprived families on average tend to underperform academically.
To name but a few.

The problem is that ETS is trying to ascribe causation to these various correlations without going through the effort of proving causation. The problem is compounded when ETS peddles their causation theories as fact in, say, a Highlights report designed for statistically-illiterate journalists, i.e., Winerip.

For example, the fact that students from single parent families underperform academically could mean at least three things.

1. the single parent family environment causes lower student achievement;
2. lower student achievement causes more single parent families;

3. some third variable causes both more single parent families and lower student achievement. (One plausible variable is low IQ.)

ETS is suggesting that it knows that the right causation is number one. And, goes on to suggest that improving the familial conditions it has highlighted will lead to increased student performance. (Winerip merely parroted ETS's suggestions and added a shriller tone.)

These are the things bad governmental policy is made of. It all starts with shoddy "research."

Here's a picture of one of the co-authors of the ETS study who's hoping his countenance of fatherly concern will convince you to swallow his dubious "research." Think of the children.


CrypticLife said...

Note that the study also blithely eliminates any causality from the school on all of the factors. While clearly the school has no control over single-parentage of households, the absentee rate could be a factor of past school performance (i.e., a student who becomes discouraged with school in early grades may be skipping more regularly by eighth grade).

But then, it's not science they're trying to do here anyway.

Malcolm Kirkpatrick said...

Years ago, I read somewhere that there is a stronger correlation between student standardized test scores and family income some years later than between student standardized test scores and current family income. The reasonable inference is that children of parents who plan ahead do well in school. That's a "correlation equals causation" I'll believe.