March 20, 2007

Reading Recovery gets the WWC Treatment

The What Works Clearinghouse issued a report yesterday on the whole-language uber-expensive tutoring program Reading Recovery.

Let me quickly summarize the good news for Reading Recovery: if you allow the developers of Reading Recovery to research their own program enough times, allow them to collect their own data, and allow them to use their own non-standard measure to gauge efficacy,then you might be able to show positive results if you're willing to live with serious methodological flaws in the research.

But, the bad news is pretty devastating: The only study conducted by independent researchers found that if you add an explicit systematic phonics component to Reading Recovery, you get results that are 37% better.

As usual, the WWC report shows that the state of education research is execrable. The WWC reviewed 78 studies. Only four met the WWC's standards and one met with reservations. That means that 73 didn't meet the WWC's standards. That's a 6.4% success rate. This doesn't necessarily reflect badly on Reading Recovery, but a fair amount of that bad research was conducted by Reading Recovery affiliated researchers.

Many of the positive findings were the result of using non-standard assessments, such as the Reading Recovery created Observation Survey of early Literacy Achievement, which are biased in favor of Reading Recovery and use "predictable text, rather than text that uses authentic, natural language patterns. Children who have learned the prediction strategies of Reading Recovery will score better reading predictable text than they will reading authentic text."

Stanovich and Stanovich (1995) report that many studies have found that authentic text is not very predictable:

It is often incorrectly assumed that predicting upcoming words in sentences is a relatively easy and highly accurate activity. Actually, many different empirical studies have indicated that naturalistic text is not that predictable. Alford (1980) found that for a set of moderately long expository passages of text, subjects needed an average of more than four guesses to correctly anticipate upcoming words in the passage (the method of scoring actually makes this a considerable underestimate). Across a variety of subject populations and texts, a reader's probability of predicting the next word in a passage is usually between .20 and .35 (Aborn, Rubenstein, & Sterling, 1959; Gough, 1983; Miller & Coleman, 1967; Perfetti, Goldman, & Hogaboam, 1979; Rubenstein & Aborn, 1958). Indeed, as Gough (1983) has shown, the figure is highest for function words, and is often quite low for the very words in the passage that carry the most information content." (p. 90)

If authentic text is not very predictable, then children who read well in predictable text may not necessarily read well in authentic text. The strategies they have learned for reading may not generalize to real reading. So, much of the positive findings for Reading Recovery do not pertain to what is considered to be real reading.

Then we have the inconvenient problem that three of the studies meeting WWC studies were conducted by researchers affiliated with Reading Recovery. Note the researcher names in the following studies:

  • Pinnell, G. S., Lyons, C. A., DeFord, D. E., Bryk, A. S., & Seltzer, M. (1994). Comparing instructional models for the literacy education of high-risk first graders. Reading Research Quarterly, 29(1), 9-38.
  • Pinnell, G. S., Lyons, C. A., & DeFord, D. E. (1988). Reading Recovery: Early intervention for at-risk first graders. Arlington, VA: Educational Research Service
  • Schwartz, R. M. (2005). Literacy learning of at-risk first-grade students in the Reading Recovery early intervention. Journal of Educational Psychology, 97(2), 257–267.

Gay Sue Pinnell, Diane Deford, and Carol A. Lyons are directors of the National Reading Recovery Center at Ohio State in the U.S. Bob Schwartz is a Reading Recovery Trainer from Oakland University.

Having your own researchers conduct experiments isn't necessarily fatal. It is cause for concern due to the issue of apparent bias and grounds for scrutiny. That scrutiny was forthcoming and the research was found wanting.

Shanahan and Barr reviewed the pre-1995 Reading Recovery studies and noted that all the studies contained serious methodological problems: "We found no studies of Reading Recovery that did not suffer from serious methodological or reporting flaws-published or not." (1995, p. 961) Shanahan and Barr identified three types of problems in the Reading Recovery pre-post design, which would lead to exaggerated success rates:

[The reported learning gain] most certainly is an overestimate of typical amounts of learning from Reading Recovery for several reasons: (a) test score improvements not linked to learning are likely to occur when students with extreme scores are selected for participation; (b) normal development and learning gains typical of young children can be due to other sources of growth and education; and (c) there is systematic omission of children who are not having success in Reading Recovery. (p. 969)

Worse still, is the systematic omission of data in the Reading Recovery affiliated research because among those omitted are children the Reading Recovery teachers identify as ones who are not progressing well. Children who are not successful are intentionally dropped before completing the entire program. The reports then do not reflect how well Reading Recovery serves the entire population it claims to serve, nor do they provide information regarding overall class effects or school effects. Consequently, the success rates cannot be used to evaluate the effectiveness of Reading Recovery.

Probably the most serious flaw in Reading Recovery research has to do with who is included in the experimental sample. In some analyses, only discontinued students were examined, making the program appear more effective than it really is. In most of the studies, students were omitted from analysis because of serious learning problems, poor school attendance, or other similar difficulties. These omissions were often made without mention. It is impossible to provide a valid estimate of the effects of Reading Recovery unless all children who start the program are included in the eventual analysis….Unfortunately, even two of the more sophisticated studies (Center, Wheldall, Freeman, Outhred, & McNaught, 1995; Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994) that we analysed have lost as much as half of their data, without any empirical estimate or control of the effects of these missing data. (p. 991-2)

The Ohio State programs have routinely collected information on those who are
dropped for various reasons, but these data have not been taken account of in their studies or technical reports, nor have they been available to the public. Depending on the proportion of participants omitted in this fashion, this creates a substantial bias in favor of Reading Recovery gains, and there is no sound way to adjust the scores that are reported on this basis." (Shanahan & Barr, 1995, p. 966)

See READING RECOVERY: AN EVALUATION OF BENEFITS AND COSTS for a much more thorough discussion of these methodological flaws in the Reading recovery research conducted by Reading Recovery affiliated researchers.

So what are we left with? Three of the studies were conducted by Reading Recovery affiliated researchers. At least two of these studies contained serious methodological flaws. One of the independent studies showed no effects using the Reading Recovery intervention. And, the final study, Iversen, S., & Tunmer, W. (1993). Phonological Processing Skills and the Reading Recovery Program. Journal of Educational Psychology, 85(1), 112-126, showed that Reading Recovery was 37% more successful when a systematic explicit phonics portion was added. And, let's not forget that most of the measurement devices were non-standard devices created by the Reading Recovery people and don't generalize to the reading of non-predictable text.

Doesn't seem like there's much left at all. Keep that in mind when you read the inevitable whoring of this report by tricky Dick Allington, the Dick Van Patten of bad education research.


TurbineGuy said...

Have I mentioned how much I hate whole language.

CrypticLife said...

Oh Rory, that's horrible.

I go over my son's spelling words at least three times a week with him, having him write them and mixing them up each time.

I taught him a "trick" for spelling that I used when I was a kid: instead of trying to memorize the spelling, remember a different pronunciation for the word that corresponds to the way it's actually spelled, exaggerating if necessary.

Think of the poor parents who have no idea how their children are learning.

Liz Ditz said...

It's Liz from I Speak of Dreams.

Wrightslaw also has a page on the lameness of reading recovery:

Reading Recovery: What do School Districts Get for Their Money? A Review of the Research by Melissa Farrall, Ph.D.

Short version: LD kids, who don't respond, are dropped from the program, but it is so expensive that more effective programs are underfunded.

Liz Ditz said...

Another one:

'R' Stands for Reading Rat Race
By Nancy Salvato
Oct 22, 2006

In the Summer of 2001 Dame Marie Clay, creator of the New Zealand based Reading Recovery program, and her entourage came to the Rayburn House Office Building in Washington, DC, to speak with House Education Committee Staffer Bob Sweet. Her purpose was to ascertain whether Reading Recovery would be eligible for Reading First funding once the bill was passed. Bob explained to Ms. Clay that explicit, systematic phonics instruction had to be included in any program eligible for RF funding because it was one of the necessary key components of reading instruction that had been established through decades of carefully conducted quantitative research.

These findings had been validated in the Report of the National Reading Panel in 2000 and were now going to become an essential part of the Reading First Law. He pleaded with Ms. Clay to use her extensive network of teacher training programs all over the US to help in the implementation of the RF program. He encouraged her to provide the leadership within the RR family to make the modifications necessary, and thus make RR eligible for RF funding consideration.

With a stare as cold as ice, Marie Clay replied that RR would not be making any changes to their program; however, Mr. Sweet could be certain a new description of its components would be written in such a way as to bring it into compliance with the RF law. Momentarily dumbfounded, he maintained that Reading Recovery could not be eligible for RF funding without modification, and his initial estimation then still stands today.
read the rest.

1citizen said...

Hey Ken,

You create a diversion and I'll make a run for the Reading Recovery Council's accounting department. I'm dying to go through their checkbook.

r schneider

Instructivist said...

"I taught him a "trick" for spelling that I used when I was a kid: instead of trying to memorize the spelling, remember a different pronunciation for the word that corresponds to the way it's actually spelled, exaggerating if necessary."

So I am not the only one who uses that method. I came up with that method because I am multilingual (including Spanish) and I playfully pronounce English phonetically. As a result I don't misspell words like definitely, laboratory, unfortunately, relevant, fraudulent...

Anonymous said...

Every year I get the job of re-teaching the Reading Recovery "successes" when they move on to second grade. When we do our September assessments, these kids are at the bottom of the heap, most test out as complete non-readers. They guess wildly, they make up stories to go with the pictures, they are at DRA level A or 1. On norm-referenced tests they score at a kindergarten level. About 12 children per year in first grade are served through Reading Recovery, and of that number 1 or sometimes 2 need no further assistance amd the rest have to be "remediated."

It's harder to get the Reading Recovery students to learn to read well, and to overcome their bad habits, than it is to teach a beginner or "failed" reader in the first place. I find it takes almost a year to get them over thinking that the pictures or other "cues" are going to magically provide them with the word, and to look at the actual letters and sounds in front of them.

Of course when the district does its follow-up on these students in THIRD grade, the ones I have "fixed" are doing well (often in top third of class) but Reading Recovery gets the credit;-)


Unknown said...

Dear Urbanteach,
I can really relate to your experiences. I am a teacher working in the UK, having to teach children who have been through RR. They come into my class every year as the poorest readers. This shows how effective the programme is! I have a really difficult time trying to get them to sound out words rather than guess at them. I was teaching some ex-RR children in my class with a synthetic phonics programme. One day the RR teacher screamed at me I shouldn't be doing that and complained to the Headteacher, a former RR teacher herself. Now I am banned from teaching synthetic phonics in my classroom.
Has the Reading First programme in the US made a difference since you posted this comment? I believe that RR no longer gets funding in the US, is that right?
Teacher K

Anonymous said...

Reading Recovery is alive and well in the US. With 22 universities turning out new Wreckers annually, it just keeps rolling along.

Moreover, RWreck tenets now permeate “Balanced Literacy” instruction, which is the mask worn by “Whole Language.”

Go figure.

Anonymous said...

Moreover, RWreck tenets now permeate “Balanced Literacy” instruction

Of course they do. Reading Recovery was brought to the US by Gay Su Pinnell, at Ohio State, who is also -- along with Irene Fountas -- the guru of Guided Reading and Balanced Literacy.

The philosophy of Guided Reading, the Literacy Collaborative and other balanced literacy initiatives openly traces its roots to Marie Clay. Pinnell is the developer of the "leveled books" system.
These levels are based on various text features but not on how readily words can be decoded.

Pinnell book on learning from Reading Recovery

Deities of the Balanced Literacy Pantheon

more than you want to know about Balanced Literacy

How to Teach Guessing, I mean, Reading

Anonymous said...

Secretary of Education Duncan and Beltway "policy makers" have no idea that Whole Language was just re-branded as Balanced Literacy and the WL still dominates reading instruction, not only in the US but in every English-speaking country in the world. There are exceptions here and there, with programs such as DI in the US, and Synthetic Phonics in the UK,but they are exceptions rather than the rule.

Secretary Duncan apparently thinks that "re-branding" is all that is required to "fix" NCLB. If so, he could hardly be more misguided.

Soma said...

Come on, Karina, we're in the post Rose Report era now. If your head teacher has really banned the teaching of synthetic phonics, why not ask the LA what they think? I'm sure the literacy leader would be interested to hear her rationale, not to mention Ofsted.

I do teach RR with an emphasis on phonic decoding. I am also currently privately teaching a seven year old child with a 'reading age' of ten who barely understands a sentence he reads: a true case of 'barking at the text'. It's now MY job to 'undo' the poor teaching he has received under a regime that has only valued his ability to phonically pronounce words with no consideration given to comprehension. All the phonics in the world won't provide a child with the meanings of words, phrases and sentences if they do not yet have good language and vocabulary, and that has to be nurtured alongside decoding skills.