September 15, 2009

SAT Scores and Family Income





Here we go again.




This graph of family income vs. SAT sub-test scores from the College Board (via the NYT's Eeconomix) has been getting a lot of play lately.




Many interpreted the graph as saying that raising family income causes an increase in SAT sub-test scores. (The Economix's sloppy "boost" language didn't help here. Nor did the left-unexplained R2 of 0.95, an artifact of how the graph was constructed, rather than as a measure of the actual variance between family income and test scores which is far lower.)

Faithful readers will already know that the underlying data for the graph is incapable of supporting a causal analysis of the relationship. It only supports descriptive and correlational analyses. Yet that didn't stop many from making the causal leap in logic by opining how obvious it is that all we need to do to produce better students is to provide their family with more money (preferably lots more money) and, thus, solving all our education woes.

Economist Greg Mankiw points out the danger (and ridiculousness) of claiming causality from mere correlational data.

Suppose we were to graph average SAT scores by the number of bathrooms a student has in his or her family home. That curve would also likely slope upward. (After all, people with more money buy larger homes with more bathrooms.) But it would be a mistake to conclude that installing an extra toilet raises yours kids' SAT scores.


It's also a mistake to conclude that winning Powerball or getting a governmental handout is going to raise SAT scores as well. Suggest the former and you'll attract funny glances; suggest the latter and half the edusphere and three quarters of edu-journalists will call you profound. Yet both are un-empiric and equally silly.

This is especially so considering there is ample correlative and experimental evidence indicating that there is a third variable driving the test score - family income/number of bathrooms and similar income-derived correlations. Take a look at the correlation between IQ test scores and biological father's income in this graph.



The graph shows the correlation for both ordinary children and adopted children. The correlations are remarkably similar, yet the contribution of the biological father of adopted children is mostly limited to some sperm. That's some mighty potent sperm. Seems like we'd get a better bang for our buck if we assured that fathers have plenty of money instead of wasting it on kids once they are conceived.

I can waste the better part of the day dredging up silly correlations. Some of them might confirm your biases and opinions on how to improve education outcomes. You'll like those. Others will go against the grain of your biases and opinions. You won't like those and you'll probably think of a way to discount the results. In both cases you're not thinking scientifically. And that's the real problem in education.

2 comments:

Dick Schutz said...

More's the pity that this Table in the College Board report (later converted to a graph for "gee whiz" purposes) attracted media intention. The report includes some very useful tables regarding patterns of high school courses. These data can reasonably be interpreted as causal and are also directly manipulable.

So much for "data-driven
educational decision making." It's a jungle out there.

http://professionals.collegeboard.com/profdownload/cbs-2009-national-TOTAL-GROUP.pdf

Also of interest are the scores by intended college major. "Education ranks in the lower tier" along with
* Family and Consumer Sciences
* Mechanic an Repair Technologies/Technician
* Parks, Recreation, Leisure, and Fitness Studies
* Personal and Culinary Services
* Public Administration and Social Services

The most popular major is "Health Professions and Related Clinical Service" (19%) followed by "Business Management, Marketing, and Related Support Services" (14%)

Students seem to be going where they think the money is. Those interested in the Service Professions don't appear to be the best and the brightest. Not sure what that tells us, but it does give me pause.

Lsquared said...

I think we ought to switch the axes, and conclude that increasing SAT scores (extra tutoring anyone?) will raise the income of the parents. Yeah, that'll work.