Education's Scam du Jour

This is the stirring conclusion of a multi-part post. Although I'd like the think all my posts make sense in and of themselves, you might be better off starting at Part I. We'll wait for you to catch up. Promise.)

In this post, we were discussing the research base (or lack thereof) for Kid Writing and I indicated that:

The first paper is a position paper by an advocacy group co-authored by the International Reading Association, a whole language shill group. Research is cited in the paper, but none of it is on Kid Writing -- it's all on other people's reading programs. This is the current scam du jour in education research and I'll cover it in more detail in my next post.

Such is the sad state of education research today. When a program claims to be "research based" what the program's developers mean is that the program contains bits and pieces of amorphous concepts (i.e., phonics, phonemic awareness, fluency, and the like) that were present in real researched based programs.

Let me explain by using the National Reading Panel as an example.

The National Reading Panel was convened to search through the existing research on early reading and identify specific programs that worked. The result was Report of the National Reading Panel "Teaching Children to Read."

The first thing the NRP did was to throw out all the non-scientific research. This eliminated about 90% of the research available for review. This alone speaks volumes about the quality of education research.

Having whittled down the research to exclude the crap, the NRP next tried to identify the components of the more effective beginning-reading programs. As I'll describe shortly, this was a fool's errand. One conclusion of the NRP was that the effective programs have common features: phonics, phonemic awareness, decodable text, oral practice formats, and a few others.
This is bad enough, but then the NRP made an even worse error, they formulated recommendations based on these common elements. The NRP's recommended that if a program included phonics, phonemic awareness, decodable text, oral practice formats, and so forth, it will also be highly effective. This is a logical fallacy.

Engelmann described the illogic well in his Education Week article "The Dalmatian and Its Spots" He also has a more in-depth treatment of it here (pdf) for you geeky science-types. I'll try to summarize it briefly.

Here's a non-education example of the NRP's faulty reasoning:

1. If a dog is a Dalmatian, it has spots.
2. Therefore, if a dog has spots, it is a Dalmatian.

Hopefully it is clear that the second statement does not follow from the first. Lots of dogs have spots -- English setters, some terriers, sheepdogs, and many mutts, not just Dalmations. Here is the NRP's parallel logic:

1. If a beginning-reading program is highly effective, it has various features: phonics, phonemic awareness, and so on.
2. Therefore, if a program has these features, it will be highly effective.

The NRP went looking for dalmations, identified spots as a common element (why not paws or snouts) and concluded that any dog with spots must be a Dalmation. Again, Engelman explains why the conclusion is faulty:

There is a lot more to a Dalmatian than having spots, and a lot more to programs that generate superior outcomes than having the features that are specified in recommendations. The additional features would include the amount of new material introduced on each lesson, the nature of the reviews that children receive, the ways in which the program tests mastery, the number of times something is presented in a structured context before it occurs in other contexts, and many more technical details about how the material is sequenced and field-tested.

Engelmann is saying that not only was the NRP's recommendation based on a logical fallacy, they didn't even identify the right "spots." You might not like Engelmann or his (highly effective) instructional programs, but I trust his identification of the "spots" in his own program over those identified by the NRP, none of whose members ever developed a highly effective beginning-reading program.

More pernicious is the fact that spots have been removed from the Dalmation; they're not Dalmation spots anymore, they're just spots. "So the analysis moves from a more careful articulation of each Dalmatian (effective program) to an elaboration of spots, now freed from the constraints of the effective program."

By removing the spots you start down a road to disaster. What if you didn't identify the right spots or missed a few? What if the arrangement and interconnection of the spots was important? Engelmann analogizes effective reading programs to complicated inventions, like the airplane, to drive the point home:

The Wright brothers had to orchestrate hundreds of specific details to make a flying machine. Yet, if any one of those pieces had been out of place or misconstructed, the machine would have failed. So it is with educational programs. You can't provide teachers with simplistic slogans and expect them to create programs that are highly successful because the criteria expressed through the slogans are far too general to result in effective programs. The field has to recognize that because highly effective educational programs are inventions, there is no intellectually honest way to describe their structure or why they are highly successful without presenting a myriad of criteria...

If you think about it, you see that the program has to be an orchestration of detail. If it weren't the moment-to-moment performance of the students would not be smooth and successful but lumpy, with no control of tiny details that could make it smooth.

As usual the education industry has ignored Engelmann and we are left with just spots. If you can say you have the right spots and pass the laugh test, you can claim that you're program is research based. Once you have the monicker "research based,' they'll let you pass through the flimsy Reading First gateway.

This is why a program like Kid Writing can claim it is a research-based program even though no research has ever been conducted on it. This is why Kid Writing can imply that it will be highly effective, when it has no evidence of success.

Thank the NRP.