January 23, 2009

Obama effect for reals according to NYT

Sam Dillon of the NY Times breathlessly reports some real educational magic today:

[The]performance gap between African-Americans and whites on a 20-question test administered before Mr. Obama’s nomination all but disappeared when the exam was administered after his acceptance speech and again after the presidential election.

The inspiring role model that Mr. Obama projected helped blacks overcome anxieties about racial stereotypes that had been shown, in earlier research, to lower the test-taking proficiency of African-Americans, the researchers conclude in a report summarizing their results.

This one was so hot the Times couldn't wait for peer review. No need to waste time with that. The Obama effect is for real and it must be reported right this second.

In fairness, Dillion does mention that the study had not undergone peer review, but only provides an incomplete and misleading explanation of prior research on stereotype threat which might have tipped readers off as to the dubiousness of this latest study.

Here are the money grafs from the Wikipedia entry on stereotype threat:

Furthermore, while Sackett et al. do not dispute the fact that stereotype threat has a real, measurable effect on test scores, they posit that in the part of the experiment where Steele and Aronson removed the stereotype threat, the achievement gap which did remain correlated closely with the existing African American - White achievement gap on large-scale standardized testing such as the SAT. In their own words:

Thus, rather than showing that eliminating threat eliminates the large score gap on standardized tests, the research actually shows something very different. Specifically, absent stereotype threat, the African American-White difference is just what one would expect based on the African American-White difference in SAT scores, whereas in the presence of stereotype threat, the difference is larger than would be expected based on the difference in SAT scores.

In subsequent correspondence between Sackett et al. and Steele and Aronson, Sackett et al. wrote that "They [Steele and Aronson] agree that it is a misinterpretation of the Steele and Aronson (1995) results to conclude that eliminating stereotype threat eliminates the African American-White test-score gap."

In the past researchers have been able to depress scores by introducing a stereotype threat (basically, the researchers told the test subjects that they were part of a group that were dummies). Removing the threat only brought scores back up to historic averages. They have not been able, however, to actually increase scores above historic averages.

This new research, in contrast, supposedly shows that test scores can be increased by such a large amount (somewhere between 0.5 to 1.0 standard deviations) to wipe out the achievement gap that exists between blacks and whites. Educationally speaking, that's a giant effect size and a truly unprecedented result (if true). Indeed, the Obama effect must be extraordinary to achieve such a result.

It's basically the educational equivalent of cold fusion. The Times apparently forgot about that lesson in journalistic humility. And so much for acknowledging legitimate opposing viewpoints that might cast some doubts on these extraordinary, unprecedented findings. Get a load of the expert the Times dredged up, replete with some nice spin supplied by the Times.

“It’s a nice piece of work,” said G. Gage Kingsbury, a testing expert who is a director at the Northwest Evaluation Association, who read the study on Thursday.

But Dr. Kingsbury wondered whether the Obama effect would extend beyond the election, or prove transitory. “I’d want to see another study replicating their results before I get too excited about it,” he said.

Kingsbury's wants to see replication -- a prefectly reasonable response. And the Times spins the failure to achieve replication as possibly being cause by "transitory" effects. The expert provides no indication that the results might be in doubt (in fact, he praises the study as "a nice piece of work") and the Times provides no indication that any doubt exists.

The impression I get from the Times article is that the Obama effect is for real, pending peer review, but might fade due to its transitory nature. The Obama legacy is already being written.

Keep your eyes on test results this summer. If we are to believe the Times the achievement gap should be eradicated due to the Obama effect. NCLB will turn out to be a smashing success. And, the world will be a happier place. Unless those nasty transitory effects dash our hopes once again.


Anonymous said...

The Obama Effect.....what a bunch of baloney!

Anonymous said...

Sam Dillon is not a credit to his trade on this one. He generally is "at or above proficiency" at journalistic grade level.

Obviously, it's easy to get caught up in the fervor.

A 20-item test. "In total, 472 Americans — 84 blacks and 388 whites — took the exam. Both white and black test-takers ranged in age from 18 to 63, and their educational attainment ranged from high school dropout to Ph.D."

That's some sample, isn't it? Any student passing Stat 101 should be able to see the clock is striking 13.

My Oh My. I'd like to peek over a shoulder to read the rejection letter. In all fairness to Sam, he did check with reputable sources. Harvard, Vandy, San Diego State, Northwestern, and Northwest Eval Ass aren't bottom drawer outfits. Ed research capability is in a long-term meltdown.

Anonymous said...

As a practical matter, could one use positive messages to one group to eliminate stereotype threat, and negative messages on another group to cause it as a means to reduce a gap? It sounds like one could.

Any idea what sized gap could be closed by working both ends? 1 SD? 0.5 SD?

-Mark Roulo

Anonymous said...

Have you heard of the Hawthorne Effect? Have you heard of the Placebo Effect? The Obama Effect is same ol same ol. Yes we can't!

Anonymous said...

Have you heard of the Hawthorne Effect? Have you heard of the Placebo Effect? The Obama Effect is same ol same ol. Yes we can't!

Anonymous said...

And yet, one nice thing about this is that it is very easy and cheap to test.

The speech on YouTube runs about 20 minutes (it might be shorter if we edited the clip down to just the speech).

So, we can burn 20 minutes before each SAT test for the next few years showing the speech. If black test scores move up the 1 standard deviation or so that they are supposed to, it will be very obvious (and we can be very happy).

If black test scores don't budge much (or at all), then we can stop doing this.

-Mark Roulo

Anonymous said...

what about the Colin Powell effect or the Condi Rice effect?