Reading Recovery gets the WWC Treatment

The What Works Clearinghouse issued a report yesterday on the whole-language uber-expensive tutoring program Reading Recovery.

Let me quickly summarize the good news for Reading Recovery: if you allow the developers of Reading Recovery to research their own program enough times, allow them to collect their own data, and allow them to use their own non-standard measure to gauge efficacy,then you might be able to show positive results if you're willing to live with serious methodological flaws in the research.

But, the bad news is pretty devastating: The only study conducted by independent researchers found that if you add an explicit systematic phonics component to Reading Recovery, you get results that are 37% better.

As usual, the WWC report shows that the state of education research is execrable. The WWC reviewed 78 studies. Only four met the WWC's standards and one met with reservations. That means that 73 didn't meet the WWC's standards. That's a 6.4% success rate. This doesn't necessarily reflect badly on Reading Recovery, but a fair amount of that bad research was conducted by Reading Recovery affiliated researchers.

Many of the positive findings were the result of using non-standard assessments, such as the Reading Recovery created Observation Survey of early Literacy Achievement, which are biased in favor of Reading Recovery and use "predictable text, rather than text that uses authentic, natural language patterns. Children who have learned the prediction strategies of Reading Recovery will score better reading predictable text than they will reading authentic text."

Stanovich and Stanovich (1995) report that many studies have found that authentic text is not very predictable:

It is often incorrectly assumed that predicting upcoming words in sentences is a relatively easy and highly accurate activity. Actually, many different empirical studies have indicated that naturalistic text is not that predictable. Alford (1980) found that for a set of moderately long expository passages of text, subjects needed an average of more than four guesses to correctly anticipate upcoming words in the passage (the method of scoring actually makes this a considerable underestimate). Across a variety of subject populations and texts, a reader's probability of predicting the next word in a passage is usually between .20 and .35 (Aborn, Rubenstein, & Sterling, 1959; Gough, 1983; Miller & Coleman, 1967; Perfetti, Goldman, & Hogaboam, 1979; Rubenstein & Aborn, 1958). Indeed, as Gough (1983) has shown, the figure is highest for function words, and is often quite low for the very words in the passage that carry the most information content." (p. 90)

If authentic text is not very predictable, then children who read well in predictable text may not necessarily read well in authentic text. The strategies they have learned for reading may not generalize to real reading. So, much of the positive findings for Reading Recovery do not pertain to what is considered to be real reading.

Then we have the inconvenient problem that three of the studies meeting WWC studies were conducted by researchers affiliated with Reading Recovery. Note the researcher names in the following studies:

• Pinnell, G. S., Lyons, C. A., DeFord, D. E., Bryk, A. S., & Seltzer, M. (1994). Comparing instructional models for the literacy education of high-risk first graders. Reading Research Quarterly, 29(1), 9-38.
• Pinnell, G. S., Lyons, C. A., & DeFord, D. E. (1988). Reading Recovery: Early intervention for at-risk first graders. Arlington, VA: Educational Research Service
• Schwartz, R. M. (2005). Literacy learning of at-risk first-grade students in the Reading Recovery early intervention. Journal of Educational Psychology, 97(2), 257–267.
  

Gay Sue Pinnell, Diane Deford, and Carol A. Lyons are directors of the National Reading Recovery Center at Ohio State in the U.S. Bob Schwartz is a Reading Recovery Trainer from Oakland University.

Having your own researchers conduct experiments isn't necessarily fatal. It is cause for concern due to the issue of apparent bias and grounds for scrutiny. That scrutiny was forthcoming and the research was found wanting.

Shanahan and Barr reviewed the pre-1995 Reading Recovery studies and noted that all the studies contained serious methodological problems: "We found no studies of Reading Recovery that did not suffer from serious methodological or reporting flaws-published or not." (1995, p. 961) Shanahan and Barr identified three types of problems in the Reading Recovery pre-post design, which would lead to exaggerated success rates:

[The reported learning gain] most certainly is an overestimate of typical amounts of learning from Reading Recovery for several reasons: (a) test score improvements not linked to learning are likely to occur when students with extreme scores are selected for participation; (b) normal development and learning gains typical of young children can be due to other sources of growth and education; and (c) there is systematic omission of children who are not having success in Reading Recovery. (p. 969)

Worse still, is the systematic omission of data in the Reading Recovery affiliated research because among those omitted are children the Reading Recovery teachers identify as ones who are not progressing well. Children who are not successful are intentionally dropped before completing the entire program. The reports then do not reflect how well Reading Recovery serves the entire population it claims to serve, nor do they provide information regarding overall class effects or school effects. Consequently, the success rates cannot be used to evaluate the effectiveness of Reading Recovery.

Probably the most serious flaw in Reading Recovery research has to do with who is included in the experimental sample. In some analyses, only discontinued students were examined, making the program appear more effective than it really is. In most of the studies, students were omitted from analysis because of serious learning problems, poor school attendance, or other similar difficulties. These omissions were often made without mention. It is impossible to provide a valid estimate of the effects of Reading Recovery unless all children who start the program are included in the eventual analysis….Unfortunately, even two of the more sophisticated studies (Center, Wheldall, Freeman, Outhred, & McNaught, 1995; Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994) that we analysed have lost as much as half of their data, without any empirical estimate or control of the effects of these missing data. (p. 991-2)

The Ohio State programs have routinely collected information on those who are
dropped for various reasons, but these data have not been taken account of in their studies or technical reports, nor have they been available to the public. Depending on the proportion of participants omitted in this fashion, this creates a substantial bias in favor of Reading Recovery gains, and there is no sound way to adjust the scores that are reported on this basis." (Shanahan & Barr, 1995, p. 966)

See READING RECOVERY: AN EVALUATION OF BENEFITS AND COSTS for a much more thorough discussion of these methodological flaws in the Reading recovery research conducted by Reading Recovery affiliated researchers.

So what are we left with? Three of the studies were conducted by Reading Recovery affiliated researchers. At least two of these studies contained serious methodological flaws. One of the independent studies showed no effects using the Reading Recovery intervention. And, the final study, Iversen, S., & Tunmer, W. (1993). Phonological Processing Skills and the Reading Recovery Program. Journal of Educational Psychology, 85(1), 112-126, showed that Reading Recovery was 37% more successful when a systematic explicit phonics portion was added. And, let's not forget that most of the measurement devices were non-standard devices created by the Reading Recovery people and don't generalize to the reading of non-predictable text.

Doesn't seem like there's much left at all. Keep that in mind when you read the inevitable whoring of this report by tricky Dick Allington, the Dick Van Patten of bad education research.