October 1, 2007

Unspinning the SAT

Frequent commentor Cal has a good article on the Geiser/Studley study of the SAT which purportedly showed that the SAT is not a good predictor of college grades. Such a claim is, of course, facially ridiculous, but Cal has taken the time to analyze the study and determine its flaws.

Go Read it because the anti-test nutters are already spinning the study wildly.

6 comments:

Cal said...

Hey, thanks for the link. Actually, thanks for reading it! It was a long piece.

rightwingprof said...

Well done, Mr. Lanier.

SATwhiz said...

Well, it's not as ridiculous as you think. Nor is it new information. I can't speak to the details of this particular study (which may in fact be riddled with flaws), but it has been a fairly consistent finding for the last four DECADES that the SAT is only modestly correlated with college grades. And why should that surprise anyone?

The SAT more closely resembles an IQ test than a test of characteristics needed for success in college. Many people with high IQ's underperform in school, for a host of reasons, while those of modest ability do very well. Dr. Richard Feynman, the Nobel-winning physicist, used to boast that his IQ was only 123. Watson, or Crick, I forget which, supposedly had an IQ of 110 or so. Nothing like the Mensa requirement of 140, which is still only the top two percent. SAT results and tested IQ tend to correlate very closely. SAT and *achievement* are much less closely related. Achievement requires more background knowledge, specific values and work habits and personality traits, etc. which are not factors in the SAT.

Now, the SAT Achievement tests (in English, mathematics, etc.) may be more closely related to grades in college -- that would make sense but I don't know if it has been extensively studied. Tests have definite value, but different types of tests are useful for different purposes (something that ought to be obvious but clearly isn't).

The SAT is a test of general ability-- heavily g-loaded, as the psychometrists say -- it measures something very handy to have to do college work, and something of which a certain amount is needed, but not ipso facto a guarantor of performance.

The available data consistently has shown that SAT scores, combined with high school grades, yield a better prediction of who will do well in college. Also, the predictive value varies by gender and ethnic group. Females don't do as well on the SAT, but consistently get better grades than males. Non white, non-mainstream students' performance tends to be underpredicted by SAT scores while grades give a more consistent predictive value. Although grades are a loose variable, due to wide variation in standards, they tend to reflect work ethic, task completion and such matters as showing up for class. These matter in college and are not measured by the SAT.

Anecdotally, those studies were proved accurate in my own case. I had average grades through high school, 1550 on the SAT and several 800's on the achievement tests. My college grades were average, just like my high school grades.

A test alone will never be a good way to predict personal characteristics like self-discipline, independence, follow-through, task persistence and others needed for successful work at the college level. On the other hand, very low scores on the SAT might be reason to question whether a person could succeed in college even with the best work habits. But the test alone is a poor predictor , and ETS itself has come up with recommendations at various times to use a formula combining grades and scores to predict the most likely candidates to succeed.

KDeRosa said...

satwhiz, much of the SAT research which purports to show that the SAT is a bad predictor if grades has the problem of "restricted range." When you take a selective college that only accepts a narrow range of SAT scores, you can expect that the effect of the SAT variance will be greatly diminished. When we look at open admissions schools the SAT variance is more pronounced. The proble is compounded because an A in a rigorous program such as engineering (where the high SAT kids will flock)means a lot more than an in a less rigorous program where the low SAT kids flock.

In short, the SAT research is less than rigorous and mostly worthless.

Actually, IQ tests correlate very well with achievement tests. Some say that what IQ measures is past achievement.

IQ is not the only factor for college achievement, but all the other factors seem to be normally distributed along the IQ curve. SO you have just as many high IQ slackers and low IQ slackers.

And, of course, adding in another measure of work ethic, such as past grads, will tend to be a better predictor of academic success than the SAT alone.

Cal said...

Satwhiz, I don't think you read the article. I'm certainly not disputing that grades aren't decent predictors of grades. I'm saying, so what?

So a kid who gets As in a horrible high school goes to college and spends two years in remediation before he drops out. He will undoubtedly get As in the remedial courses.

Grades do nothing to predict the academic readiness of the student. Zilch. Nada. Only with income, school quality, and test scores can any determinations be made in concert with grades, and all that's then known is whether or not the teachers thought they tried.

Grades are irrelevant. The only reason why the College Board uses that first year grades descriptor is because back in the day, college students all took the same first year courses. That hasn't been true for at least 30 years. Now anyone is accepted into college, and the colleges need to know who will entering with 8th grade knowledge, and who is actually ready for college. That's the SAT and the ACT's job.

Rwprof--thanks.

palisadesk said...

The best predictor of future behavior is past behavior, so good grades in high school should be a reasonable predictor of good grades in colege -- IF they indeed represented superior performance on appropriately challenging academic work. With grade inflation rampant, this is now not something that can be taken for granted, thus an increased need for external and norm-referenced data.

There was an interesting article in Educational Psychologist 2 or 3 years ago by Dr. Robert Sternberg (Yale), who has written a number of interesting books (I recommend "Successful Intelligence.") He attempts to identify and measure other factors that make for what most of us would call "practical" intelligence -- not only having smarts, but the ability to use and apply intelligence in a variety of real-life contexts. Sternberg developed an alternative to the SAT -- it measured the same factors as the SAT but also some others that he identified as related to "successful intelligence." In his study the results of his SAT-alternative (sorry, I don't remember what it was called, or whether it is commercially available) predicted college performance with double the accuracy of the SAT. Just an interesting concept, possibly worth pursuing, since grade inflation is so endemic everywhere that "grades" have practically lost all meaning. For a depressing read, check out "Ivory Tower Blues" by Josept Cote and Anton Allohar.

The findings here were rather interesting: http://www.gseis.ucla.edu/heri/PDFs/PR_ACT04-05.pdf
Does the SAT not reliably identify students who lack the skills to do basic college work? In many public universities faculty appear to perceive that a large number of students (even a majority) are in this position.

Interesting too that many of the students considered themselves to be in the top 10%! They all grew up in Lake Wobegon no doubt.