Reading Recovery gets the WWC Treatment

The What Works Clearinghouse issued a report yesterday on the whole-language uber-expensive tutoring program Reading Recovery.

Let me quickly summarize the good news for Reading Recovery: if you allow the developers of Reading Recovery to research their own program enough times, allow them to collect their own data, and allow them to use their own non-standard measure to gauge efficacy,then you might be able to show positive results if you're willing to live with serious methodological flaws in the research.

But, the bad news is pretty devastating: The only study conducted by independent researchers found that if you add an explicit systematic phonics component to Reading Recovery, you get results that are 37% better.

As usual, the WWC report shows that the state of education research is execrable. The WWC reviewed 78 studies. Only four met the WWC's standards and one met with reservations. That means that 73 didn't meet the WWC's standards. That's a 6.4% success rate. This doesn't necessarily reflect badly on Reading Recovery, but a fair amount of that bad research was conducted by Reading Recovery affiliated researchers.

Many of the positive findings were the result of using non-standard assessments, such as the Reading Recovery created Observation Survey of early Literacy Achievement, which are biased in favor of Reading Recovery and use "predictable text, rather than text that uses authentic, natural language patterns. Children who have learned the prediction strategies of Reading Recovery will score better reading predictable text than they will reading authentic text."

Stanovich and Stanovich (1995) report that many studies have found that authentic text is not very predictable:

It is often incorrectly assumed that predicting upcoming words in sentences is a relatively easy and highly accurate activity. Actually, many different empirical studies have indicated that naturalistic text is not that predictable. Alford (1980) found that for a set of moderately long expository passages of text, subjects needed an average of more than four guesses to correctly anticipate upcoming words in the passage (the method of scoring actually makes this a considerable underestimate). Across a variety of subject populations and texts, a reader's probability of predicting the next word in a passage is usually between .20 and .35 (Aborn, Rubenstein, & Sterling, 1959; Gough, 1983; Miller & Coleman, 1967; Perfetti, Goldman, & Hogaboam, 1979; Rubenstein & Aborn, 1958). Indeed, as Gough (1983) has shown, the figure is highest for function words, and is often quite low for the very words in the passage that carry the most information content." (p. 90)

If authentic text is not very predictable, then children who read well in predictable text may not necessarily read well in authentic text. The strategies they have learned for reading may not generalize to real reading. So, much of the positive findings for Reading Recovery do not pertain to what is considered to be real reading.

Then we have the inconvenient problem that three of the studies meeting WWC studies were conducted by researchers affiliated with Reading Recovery. Note the researcher names in the following studies:

• Pinnell, G. S., Lyons, C. A., DeFord, D. E., Bryk, A. S., & Seltzer, M. (1994). Comparing instructional models for the literacy education of high-risk first graders. Reading Research Quarterly, 29(1), 9-38.
• Pinnell, G. S., Lyons, C. A., & DeFord, D. E. (1988). Reading Recovery: Early intervention for at-risk first graders. Arlington, VA: Educational Research Service
• Schwartz, R. M. (2005). Literacy learning of at-risk first-grade students in the Reading Recovery early intervention. Journal of Educational Psychology, 97(2), 257–267.
  

Gay Sue Pinnell, Diane Deford, and Carol A. Lyons are directors of the National Reading Recovery Center at Ohio State in the U.S. Bob Schwartz is a Reading Recovery Trainer from Oakland University.

Having your own researchers conduct experiments isn't necessarily fatal. It is cause for concern due to the issue of apparent bias and grounds for scrutiny. That scrutiny was forthcoming and the research was found wanting.

Shanahan and Barr reviewed the pre-1995 Reading Recovery studies and noted that all the studies contained serious methodological problems: "We found no studies of Reading Recovery that did not suffer from serious methodological or reporting flaws-published or not." (1995, p. 961) Shanahan and Barr identified three types of problems in the Reading Recovery pre-post design, which would lead to exaggerated success rates:

[The reported learning gain] most certainly is an overestimate of typical amounts of learning from Reading Recovery for several reasons: (a) test score improvements not linked to learning are likely to occur when students with extreme scores are selected for participation; (b) normal development and learning gains typical of young children can be due to other sources of growth and education; and (c) there is systematic omission of children who are not having success in Reading Recovery. (p. 969)

Worse still, is the systematic omission of data in the Reading Recovery affiliated research because among those omitted are children the Reading Recovery teachers identify as ones who are not progressing well. Children who are not successful are intentionally dropped before completing the entire program. The reports then do not reflect how well Reading Recovery serves the entire population it claims to serve, nor do they provide information regarding overall class effects or school effects. Consequently, the success rates cannot be used to evaluate the effectiveness of Reading Recovery.

Probably the most serious flaw in Reading Recovery research has to do with who is included in the experimental sample. In some analyses, only discontinued students were examined, making the program appear more effective than it really is. In most of the studies, students were omitted from analysis because of serious learning problems, poor school attendance, or other similar difficulties. These omissions were often made without mention. It is impossible to provide a valid estimate of the effects of Reading Recovery unless all children who start the program are included in the eventual analysis….Unfortunately, even two of the more sophisticated studies (Center, Wheldall, Freeman, Outhred, & McNaught, 1995; Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994) that we analysed have lost as much as half of their data, without any empirical estimate or control of the effects of these missing data. (p. 991-2)

The Ohio State programs have routinely collected information on those who are
dropped for various reasons, but these data have not been taken account of in their studies or technical reports, nor have they been available to the public. Depending on the proportion of participants omitted in this fashion, this creates a substantial bias in favor of Reading Recovery gains, and there is no sound way to adjust the scores that are reported on this basis." (Shanahan & Barr, 1995, p. 966)

See READING RECOVERY: AN EVALUATION OF BENEFITS AND COSTS for a much more thorough discussion of these methodological flaws in the Reading recovery research conducted by Reading Recovery affiliated researchers.

So what are we left with? Three of the studies were conducted by Reading Recovery affiliated researchers. At least two of these studies contained serious methodological flaws. One of the independent studies showed no effects using the Reading Recovery intervention. And, the final study, Iversen, S., & Tunmer, W. (1993). Phonological Processing Skills and the Reading Recovery Program. Journal of Educational Psychology, 85(1), 112-126, showed that Reading Recovery was 37% more successful when a systematic explicit phonics portion was added. And, let's not forget that most of the measurement devices were non-standard devices created by the Reading Recovery people and don't generalize to the reading of non-predictable text.

Doesn't seem like there's much left at all. Keep that in mind when you read the inevitable whoring of this report by tricky Dick Allington, the Dick Van Patten of bad education research.

More on Wisconsin's Reading First Scandal

The Prof at Right Wing Nation has my back on Wisconsin's Reading First Scandal.

From these data, we cannot say that the Reading First districts did or did not increase their proficiencies. There just aren't enough data. But — and this is crucial — neither can the Reading First districts claim that they raised their proficiencies using these data. The only way we can determine whether these school systems did or did not raise their proficiencies is by analyzing the raw data, and not the aggregates by school district. In other words, Ken was right, and the journalist was wrong.

I like that last sentence so much I'm going to repeat it.

In other words, Ken was right, and the journalist was wrong.

I did a very quick initial analysis by assuming both distributions were essentially normal so I wouldn't have to fix the improperly formatted data set. RWP, an expert in data analysis, did the hard word of reformatting the data set and did a complete analysis. I re-analyzed the data and came up with the same numbers as RWP. Time permitting, I'll update my figures with the re-analyzed numbers. The effect sizes will change slightly, but my conclusions will still hold.

The important thing to remember is that RWP's analysis includes the following assumption.

We are assuming here either that the proficiency exam standards did not change between the two years or that the proficiency reports for the two years are comparable (if they are not, then Wisconsin cannot make any statement about their proficiency levels over time).

As the NAEP data clearly shows, the Wisconsin's proficiency exam standards did change between 1998 and 2005. NAEP scores declined slightly, while the Wisconsin scores magically skyrocketed. Suspiciously so.

So, when RWP says that Wisconsin and Madison can validly claim that scores have increased, this conclusion only holds if Wisconsin's proficiency exam standards didn't change between 1998 and 2005. And, as we know from NAEP scores, they did.

RWP's analysis reveals an importnt factoid.

So far, everything looks positive, until we look at the skewness. Go back to that bell curve water balloon. If you grab the tail on the right and pull it further to the right, more of the water will spill into that tail, right? That's what we call a right-skewed curve, and it has a positive skewness factor. If you pull the left tail out, more water spills into that left tail, and we have a left-skewed curve, with a negative skewness factor. When we look at the skewness for the two years, both are left-skewed — that is, in both years, there are more data in the left tails (less than proficient) — but the 04-05 curve is more left-skewed than 98-99 (-1.86 and -0.92, respectively). So even though it does look like Wisconsin may have improved the reading proficiency between the two years, they also slightly increased those who were less than proficient (if this seems like a paradox to you, think of the water balloon again, and all will be made clear).

Even though Wisconsin's scores magically skyrocketed, the number if students who were less than proficient increased slightly. Kids at top learn more, but the kids at the bottom continue to struggle. These would be the kids that Reading First aimed to serve.

I think we can finally put this one to bed, unless Madison officials decide to make more outlandish claims.

medieval barbers

Teacher TMAO turns in another great post on teacher autonomy:

As a teacher I should get to teach whatever I want because I know best and no one is allowed to determine if what I taught or how I taught it was important, valid, or successful because I earned tenure in the name of academic freedom.

That's the argument our profession makes. It's weak sauce.

Check it out.

The Big Guns Pound Schemo

The story so far ...

Diane Jean Schemo writes a highly biased, misinformed, and irresponsible article insinuating that the Madison Wisconsin Metropolitan school district was bullied into dropping their "balanced literacy" program and adopting a phonics-based reading program, refused, and allegedly have experienced great success with their own program.

Mark Liberman, of Language Log, discussed the article here and included a letter from Mark Seidenberg,Professor, Psychology and Cognitive Neuroscience, University of Wisconsin-Madison, also critical of the Schemo article.

I pointed out some of the flaws in this argument here. Basically, Wisconsin made its third grade reading test easier during the period tested and this accounted for all of Madison's gains rather than real improvement in reading ability.

Schemo responded to my criticisms in an email and I deconstructed those arguments here.

Now the big guns in early reading instruction are responding to Schemo's article.

• Reid Lyon, former Chief of the Child Development and Behavior Branch within the National Institute of Child Health and Human Development (NICHD) at the National Institute of Health (NIH),
• Robert Sweet, Former Professional Staff Member Committee on Education and the Workforce U.S. House of Representatives,
• Louisa Moats, Ed.D. Formerly co-investigator of the NICHD Early Interventions Project,
• Linnea Ehri, Ph.D., Distinguished Professor Program in Educational Psychology CUNY Graduate Center and Joanna Williams, Ph.D. Professor, Program in Cognitive Studies in Education Teachers College, Columbia University, and
• Timothy Shanahan, Professor of Urban Education, University of Illinois at Chicago President, International Reading Association

A common theme emerges:

The at-risk students that Reading First aims to serve are underperforming compared to more advantaged students by Wisconsin's own measures

"However, the poor, disadvantaged children for whom RF was specifically targeted lost out. The State of Wisconsin’s own statistics tell the story. In 2005, forty five percent of African Americans in Madison schools are in the lowest two categories of reading ability. In another State assessment corroborating this fact forty six percent of third grade African American students scored below grade level compared to nine percent of white students."

Richard Allington, Schemo's "expert," is a buffoon

"Richard Allington, who was quoted in opposition to Reading First, has no credentials as a researcher or scientist. He and the "reading community" to which he refers have perpetuated myths and ineffective practices associated with Whole Language for decades – and look at what those have brought us."

There is no scientific basis for Madison's reading program

"Like the issue of global warming, there is no scientific debate about whether children benefit from direct instruction in how the alphabetic code of English represents speech. There is, in contrast, plenty of evidence that teaching children to guess at words through context and pictures is, indeed, malpractice, and that most poor readers fall by the wayside early because no one is teaching them how to read."

The National Reading Panel's meta-study has not been rejected by the reading community, only by the nutters

"The Panel's report became the basis of Reading First, and has been cited hundreds of times in scholarly research journals, endorsed by the International Reading Association, and identified as one of the most influential educational reports by the Editorial Projects in Education Research Center."

Schemo Responds

Courtesy of gadfly Rory, Diane Schemo responds (via email) to my criticism of her reporting of the Madison Wisconsin school district's turning down of $2 million dollars in Reading First grant money.

A. Wisconsin's average NAEP score remained the same, but that fact used alone is misleading.

Enrollment changes during that period were severe: from 1998 to 2005, Wisconsin's low-income population rose from 1 in 4 students to 1 in 3 students, and blacks were 13 percent of all students, up from 10 percent in 1998. Latinos rose from 4 to 6 percent. White student enrollment by 2005 had declined to 77 percent, from 82 percent in 1998. According to the state Department of Public Instruction, Wisconsin has the fastest growing poverty rate of any state in the nation.

If you look at the scores for black 4th graders, they rose to 194 on the 2005 NAEP, up from 187 in 1998. Scores for Latinos rose to 208 from 201 during the same period.

Though White scores remained steady at 227-228, the shrinking pool of whites meant that the increasing numbers of black and Latino students brought down the overall average, even though these two groups made significant gains..

To put the case in its most extreme terms, a severe enough exodus of white students would inevitably mean that NAEP scores would nose dive as the population shifts, until Wisconsin had succeeded in completely erasing any achievement gap, and black and Latino scores were on par with white scores--something no state, anywhere, has done, using phonics, whole language, steroids, tea leaves . . . you name it.

I can sum up this argument in one word: irrelevant.

This argument is irrelevant to the question of whether performance in Madison, and in particular the Reading First eligible schools, actually increased.

Even if black and Hispanic performance rose with respect to white performance it occurred on a state-wide basis. This tells us nothing about black and Hispanic performance in Madison. As far as we know, performance in Madison could have plummeted during this time period and black and Hispanic performance outside of Madison could have skyrocketed. The data is silent.

Now, let's take a closer look at this stunning increase in black and Hispanic performance in Wisconsin that's got Schemo giddy as a schoolgirl.

NAEP 4th Grade Reading Test -- Scale Scores

Year	Black	Hispanic White
1992	198	209	227
1994	196	203	227
1998	187	201	228
2003	200	209	225
2005	194	208	227

These are not the miraculous gains Schemo would have us believe. A more accurate characterization is partial recovery of lost ground lost since 1994. In any event, according to NAEP's statistical significance calculator the difference between black performance between 1998 and 2005 was not statistically significant. Oops. Moreover, the "gain" in black perfromance from the 1998 nadir to 2005 was only about 0.20 s.d. which is far less than the alleged gain of 0.77 s.d. on the Wisconsin test. Even with that small "gain" the performance level for black students still remains less than it did in 1994.

My initial allegation remains unrebutted by Schemo. NAEP 4th grade reading data for all students in the state of Wisconsin shows flat performance between 1998 and 2005. This NAEP data casts serious doubt on the miraculous state-wide student gains made in Wisconsin's own 3rd grade reading test.

Schemo seems to forget that a little thing called NCLB was passed during this period and that in response thereto many states responded by making their assessments easier in order to make it easier to comply with NCLB. There's even a name for it: the race to the bottom.

Schemo's demographic shift argument doesn't fly either. According to NAEP, the change in scores for whites, blacks and Hispanics was not statistically significant, as I indicated above. The only realistic inference we can draw is that scores remained about the same for all groups, as did the overall performance level.

B. To say that a city, with its higher concentration of poverty and minority students, performed on a rough par with the state average is not an indictment. Most cities would be quite proud to keep pace with state averages. Indeed, the reason you have a special NAEP urban assessment is because the profiles of cities are so different from states.

Madison's performance level in 2005 was close to the average performance of the rest of the state. But then again, this is almost exactly where it was back in 1998. It certainly didn't gain any ground. Perhaps Madison doesn't have the Dickens-like poverty levels that Schemo is trying to portray.

Then again, compared to the rest of the state, whites in Madison seem to slightly overperform whites state-wide and blacks and Hispanics seem to slightly underperform blacks and Hispanics state-wide. Maybe Madison has a bunch of affluent whites making up for the lagging poor blacks and Hispanics. It's no secret that balanced literacy reading programs favor the affluent who have the superior background knowledge needed to succeed in these programs.

But again, this argument by Schemo is also irrelevant because we are concerned with the performance of the Reading First eligible schools, not all of Madison's schools. And these schools, as I pointed out, significantly underperformed other schools in Madison and the rest of Wisconsin.

Let me try to explain it a different way. Something happened in the state of Wisconsin between the years 1998 and 2005 such that 3rd grade students need to perform about 0.77 s.d. better on Wisconsin's WRCT in order that they may score exactly the same on the NAEP as 4th graders performed back in 1998. This 1998 cohort scored 0.77 s.d. less on the 3rd grade test and still managed to score the same on the NAEP as the 2005 cohort with those significantly better 3rd grade scores. Apparently, something magical is going on in Wisconsin such that reading comprehension rose dramatically in 3rd grade only to completely wash away by 4th grade. As it turns out, the kids in Madison's eligible Reading First schools didn't get as much magic as the rest of the kids in Madison or Wisconsin because their third grade scores didn't magically rise by as much as the kids in these other schools.

So if the object of Madison's fancy home-brewed reading program was to raise the scores of the kids in the eligible Reading First school relative to the better performing schools, we know that didn't happen because these kids now perform worse relative to these other kids in the third grade tests and the 4th graders in the better performing schools still perform exactly the same as they did in 1998.

No matter how you slice it, the performance of Madison's eligible Reading First schools did not improve relative to other schools in Wisconsin.

C. Also, Madison's efforts were part of a statewide drive to improve reading scores, so keeping pace with the rest of the state is, again, not an indictment of balanced literacy or any other single approach. To conclude that Madison's approach was unsuccessful, you'd have to compare districts across the state by their method of instruction, enrollment features and test score gains.

Another losing argument.

According to NAEP, this state-wide initiative failed miserably. Performance for all groups is not statistically different than it was back in 1998. And based on the 3rd grade test, the relative performance of Madison's eligible reading first schools declined. So, yes, it is clear: Madison's approach has been unsuccessful because it has failed to increase student performance.

And just so we're clear exactly just how miserable reading scores are in Wisconsin: 2/3 of black kids, 1/2 of Hispanic kids, and 1/4 of white kids performed below the basic level on the NAEP in 2005 just like they did back in 1998. These are the kind of scores that has the public calling for reform. And here we have Schemo trying to spin these scores as some kind of success story.

D. Finally, the blogger uses statistical sleight of hand when he wants to discuss the achievement gap, switching the time frame back to 1992. But the ground zero we were counting from was 1998, when Wisconsin apparently reacted to its eroding performance and started a statewide drive to improve early reading. And if you look at those figures (cited above), the gap between African-Americans, Latinos and whites shrank.

Let's measure the achievement gap from the table above

Year	Achievement Gap
1992	29%
1994	31%
1998	41%
2003	25%
2005	33%

The mean achievement gap from 1992-2005 is 31.8% and the 2005 level is still above the mean. About the best we can say is that Wisconsin has almost gotten back to 1994 levels.

But here's the important thing, reading scores during this period have remained flat. If the object is to improve reading scores for all kids, and especially minorities, Wisconsin has failed at this task for 13 years.

My arguments remain unrebutted.

Schemo has committed journalistic malpractice. The article was supposed to be honest reporting of facts. It is one thing to report that Madison's reading program for its eligible Reading First schools has been successful based on Wisconsin's 3rd grade test. But it is malpractice not to also have reported that:

1. Wisconsin's scores on the same test actually improved by a greater amount state-wide than did Madison's eligible Reading First schools, and
   
2. Wisconsin's scores have remained flat based on NAEP.
   

Both of these facts cast serious doubt on the Madison school district's contention that their reading program has been successful. A serious reporter would have asked for better data or would have challenged the district to test a random sample of students on a generally accepted measure of reading ability. Schemo is not this reporter.

Update: For those of you with weak statistics skills, the fact that there is not NAEP data for Madison is not dispositive. We have state level data for both assessments. The state level data shows that the mean shifted 0.77 standard deviations for Wisconsin's test and didn't shift at all for the NAEP for the 1998-2005 period in question. Madison's shift in mean scores on the Wisconsin test parallels that of the state-wide shift. As such, Madison's expected performance gain on the NAEP will be close to zero, the same as the state-wide performance gains.

The other suspicious factor is the magnitude of the shift in the Wisconsin assessment. A 0.77 sd shift is considered to be a large effect size. Such large shifts are all but unheard of in education. We have found Lake Wobegone, kids, it is the entire state of Wisconsin.

We'd expect some statistical noise due to the differences in cohorts and the like, but we don't expect to see systematic error between tests like these. Like I said at the outset, this back of the envelope analysis raises serious doubts as to the accuracy of Madison's claims of performance gains sufficient that the yoke is on Madison to back up their contentions with data from an assessment that is known not to have shifted during this period, such as the CTBS, ITBS, SAT-9, and the like. Why isn't that data forthcoming?

When TAWLers Attack

looks like my recent post mocking the balanced literacy shenanigans being perpetrated in Madison has attracted the attention and ire of the Teachers Applying Whole Language listserv.

Go read my initial post first to get some context and I'll try to deal with the TAWLers arguments, such that they are, as they come up.

I don't understand how people cannot see that WL is offering MORE strategies, not fewer.

More strategies: yes. More productive strategies: no. Not only have they inserted a bunch of unproductive strategies to confuse naive readers, they've downplayed the most productive strategy, letter-sound correspondence, as the strategy of last resort.

It isn't like taking MOST OF THE LETTERS OUT of the words, it is not requiring every single letter to be agonized over every single time.

But, unfortunately, that is how skilled readers process words. That's what eye-movement research has shown us. Skilled readers do so at a very rapid pace. I would think any teacher of reading would be familiar with this research.

I watched Reading Mastery in Roseville one time in CA, and they were making the kids sound out SAID! Kids who could read better already than that lesson was calling for were penalized for being able to say the words automatically because they weren't slowing down and sounding out the parts.

"Said" is one of the first irregular words taught in the Reading Mastery sequence. We're talking like the second month of kindergarten when kids are just learning how to read. First the word is taught in isolation and then read in connected text. It is a difficult word for children because it is one of the first words they are taught that breaks the phonetic rules.

The kids are taught to sound it out /s/ /a/ /i/ /d/ phonetically and then taught that it is pronounced /s/ /e/ /d/. In this way, the child has a mental hook that the letter combination s-a-i-d equals /s/ /e/ /d/. They wanted to get to the meaning, and were told to stick with the single letters.

A child who can read the passages with no errors is most likely a child that is misplaced in the sequence. This would be a teacher placement problem, not a curriculum problem.

And, despite the effort to make it impossible to read that Call of the Wild passage, it wasn't so impossible, either.

It wasn't meant to be impossible, it should have been at an instructionally comprehensible level, but at a word identification level that was meant to be at the frustration level (80%). It also demonstrates how difficult it is to guess at the omitted words even when read in context and with some phonetic clues and word structure clues provided. It demonstrates that skilled readers aren't able to read a passage with fluency once the phonetic markers are removed. Skilled readers do not rely on context clues to identify unknown words. They do rely on context clues to ascertain meaning, as this TAWLer demonstrated. But, let's not kid ourselves, that's not reading. And, any kid whose decoding skills are so bad that he can only identify 80% of the words is a kid who will have no love for reading even if he can get the gist of the passage through context clues. This is the main fallacy underlying whole language pedagogy.

The example in the Times article, showed the boy "guessing" "pumpkin" and being told to "look at the word." How is that telling him to randomly guess? One clue was how long the word was, but if that did not cause him to actually look more closely at the letters in that word, another clue was surely to follow.

So, here's a few guesses that might have worked just as well as "pea" -- pop, pup, puppy, petunia, and the like.

The boy is guessing because he clearly hasn't been taught that the p stands for /p/ and ea stands for /ee/ or /aa/. All the kid needed to read the word was to know two phonics rules to read the word. Some kids will figure this out. But others won't unless it is explicitly shown to them and ample practice provided.

It is impossible to get what a real supportive reading session would be given one small example like that. The author of the article in the other publication made unfounded assumptions that the boy did not look at the word but at the picture. Well, if the picture was being used to give him the words, he wouldn't have said "pumpkin" then, would he.

Or maybe there was a picture of a pumpkin and a pea and the kid just happened to pick the wrong one. That wouldn't happen if he'd been taught to read correctly now would it. Guessing at pictures isn't reading. And guessing at pictures or predictable text does not translate into reading regular text without pictures.

They didn't point out that he was using background knowledge of what would fit in a sentence that must have been about food, vegetables, or farming...and that he didn't just throw out random words beginning with "p".

Another problem is that many at-risk kids lack the background knowledge that they'll be asked to call on to engage in these whole language guessing games. Again, this argument clearly shows that this TAWLer doesn't know the difference between using background knowledge for word identification as opposed to ascertaining word meaning. Most readers know all the vocabulary and background knowledge to fluently read the Call of the Wild passage, but once the letter identification clues are removed, skilled readers are unable to activate that vocabulary and background knowledge to identify the missing words from the context provided.

This makes me so mad. How can they gloss over the fact that the district is successful by the very measures that are being used to push other programs that they don't want to use?

Except that it's not successful. As I showed here. Madison's Reading First eligible schools are underperforming in the balanced literacy program.

As for the Call of the Wild passage... If a student had that many gaps in their reading, then I would say it is too difficult and they need to read at a lower level. They will spend their whole time trying to figure out the words and make no sense of the story.

Welcome to what reading is like for a kid who has weak decoding skills. Skills they won't be taught explicitly or in a systematic manner in balanced literacy.

And the Jack London example is beyond nonsense, because as someone ... has already said, a text this full of holes is far above the reading level of reader. Guess book choice and teacher guidance count for nothing!

It's an easy test. If the child struggles to identify at least 2 out of every 10 words read, then the level of the book is at the same level of modified Jack London passage I provided. Most books my first grade son comes home with in his balanced literacy class fall into this frustration level category because the books are not carefully vetted for decodability. These teachers really have no clue what they are asking of their students. Apparently, when it comes to whole language instruction, teacher guidance doesn't count for anything.

This guy makes it up as he goes along. Nothing is referenced to any citation. Obviously, he is not planning to debate ... using any recognized, peer reviewed research. He is only out to smash whole language. Makes me wonder why he is so passionate about this when you consider that there are probably fewer than 5% of teachers in the world who even know what whole language is, let along claim to be whole language proponents!

This is a pot, kettle, black moment. This is the crowd that cites their own opinions as research and doesn't know how to design a scientific study if it bit them. Please.

This blog isn't a research paper. I try to cite when I can, but I don't cite every well-known point. As always, if someone doubts any point I make, provide a reason why you doubt it, and request a cite. One will be provided.

Plus, I have nothing against Whole Language per se. I just dislike bad instruction. And, whole language is bad instruction based on bad science practiced by ideologues on unsuspecting children.

There's also a No True Scotsman fallacy thrown in there too. See if you can spot it.

Keep those arguments coming TAWLers. It's like shooting fish in a barrel.

Madison Cooks Books

(This post summarizes, in a hopefully more coherent manner, the analysis I did in this post.)

The NYT reports that the Madison Wisconsin school district turned down $2 million dollars in Reading First grant money. The reason: they didn't want to use a phonics-based reading program as required by Reading First. Instead, Madison wanted to continue using the district-created "balanced literacy" reading program. Nothing wrong with that per se; Reading First is a voluntary program entailing federal oversight.

But with NCLB requirements for increased student proficiency looming on the horizon, the pressure is on for low-performing school districts, like Madison, to increase student performance. It would be quite an embarrassment if Madison's reading program failed to produce results, considering their very visible refusal to adopt a phonics-based reading program--especially since there was grant money attached.

So what do you suppose Madison did when the agenda-driven NYT came looking for a poster child of a school district bullied by the Feds in the (phony) scandal-plagued Reading First program? They cooked the books.

Under their system, the share of third graders reading at the top two levels, proficient and advanced, had risen to 82 percent by 2004, from 59 percent six years earlier, even as an influx of students in poverty, to 42 percent from 31 percent of Madison’s enrollment, could have driven down test scores. The share of Madison’s black students reading at the top levels had doubled to 64 percent in 2004 from 31 percent six years earlier.

And while 17 percent of African-Americans lacked basic reading skills when Madison started its reading effort in 1998, that number had plunged to 5 percent by 2004. The exams changed after 2004, making it impossible to compare recent results with those of 1998.

Madison's scores did rise from 58.9% in 1998 to 82.7% in 2004, an apparent rise of +0.72 standard deviations (s.d.) which is a large effect size. But the question remains: did student performance actually improve?

Let's find out.

Madison's scores rose less than Wisconsin's

According to the Wisconsin Reading Comprehension Test (3rd grade), the number of proficient students in Wisconsin rose 22.5 points from 64.9% in 1998 to 87.4% is 2005, an increase of +0.77 s.d. So it wasn't just Madison's scores that rose, scores rose across the board in Wisconsin. If anything, Madison's scores rose slightly less than the average score in Wisconsin.

But, is such an across-the-board gain in achievement realistic in the first place?


Wisconsin's NAEP scores remained flat

NAEP scores for Wisconsin show that the number of proficient (and above) students was 34% percent in 1998 and dropped slightly to 33% in 2003 and stayed there in 2005, the last time fourth graders were tested for reading. Students scoring at the basic level (and above) dropped from 69% to 67% during this same period. From the period 1992-2005, the achievement gap between black and white students rose from 28 points to 33 points and the gap between poor and non-poor students dropped slightly from 28 points to 25 points.

So, NAEP shows us that the reading performance of Wisconsin fourth graders has basically remained flat since about 2000. (Go here and select Wisconsin as the jurisdiction.)

The NAEP scores tell us that Wisconsin's miraculous gains in reading achievement from 1998-2005 are non-existent. Wisconsin did what most states did in response to NCLB, they goosed the tests to artificially increase test scores to give the appearance of increased student achievement.

Let's recalibrate Wisconsin's gain of +0.77 s.d. to 0.0 s.d. to account for the non-gains made in NAEP. This means that Madison's real performance during 1998-2005 actually declined by -0.05 s.d.

Madison's schools eligible for Reading First funding performed significantly below Madison's other schools

According to this source, there were four Madison schools eligible for Reading First funding: Glendale, Hawthorne, Lincoln, and Orchard Ridge. These schools never received this funding because Madison choose to stick with its Balanced Literacy reading program. Let's see how those schools performed.

The average gain made by these four schools was only 21.6% or +0.56 s.d. This is a significant under-performance compared to the statewide gain of +0.77 s.d. Using our NAEP recalibration, the performance of these schools actually declined by -0.19 s.d. That's big.

In contrast, the average gain made by the remaining schools in Madison was 25.4% or +0.80 s.d. which is about the same gain made statewide.

The Disaggregated Data

In the NYT article, Madison officials made some wild suggestions about the relative performance of black students compared to other students (mostly white) in Madison. I showed in my previous post how these percentile gains are misleading due to how scores are distributed (a normal distribution), so I won't repeat it here.

In fact, I believe the scores that Madison is using are inaccurate. Madison's numbers have black performance rising by +0.86 s.d., which is high. I can't find disaggregated data for the WRCT, but the disaggregated data from the 3rd grade performance on the new WSAS test shows that black performance is significantly less than this and less than the average gain made by white students. It's not an apples to apples comparison, but it is consistent with the rest of the data.

Conclusion

Madison is cooking the books.

Its schools slightly underperformed Wisconsin schools and Madison's other schools.

In fact, NAEP data shows that the gains made by Wisconsin are illusory. It's doubtful that scores rose at all in Wisconsin.

If we look at only the schools in Madison that were eligible for Reading First funding, we see that these schools performed significantly worse than other schools in Wisconsin.

So it appears that Madison's Balanced Literacy reading program, which cost the district $2 million, failed to increase student performance in Madison and actually caused a relative decline in the schools that were supposed to get Reading First funding.

This is exactly what we expect to see in your typical balanced literacy program, at-risk children failing to achieve. These are the children most damaged by "balanced literacy" programs, kids with low language skills and background knowledge. These were the kids that Reading First intended to serve.

Schemo gets pwned

See my summary post in which I try to roll all the updates into a coherent post.

**Update** The scandal grows. It turns out that the Reading First schools in Madison underperformed the non-Reading First schools by almost a 1/4 standard deviation. This is an educationally significant difference. (Clarification: I'm using "Reading First schools" to designate the schools in Madison that were eligible for reading First funding but didn't get it because Madison decided to use their balanced literacy program instead of a Scientifically based reading program.)

In part one of this post, I showed you the embarrassing horror show that is taking place in the Madison school district that has hapless first graders guessing at the word "pea" and thinking the word is "pumpkin." If it weren't embarrassing enough having that displayed in the opening paragraphs of this NYT article, the Madison school district has the audacity to claim that their reading program is actually boosting student performance.

Call it the $2 million reading lesson.

By sticking to its teaching approach, that is the amount Madison passed up under Reading First, the Bush administration’s ambitious effort to turn the nation’s poor children into skilled readers by the third grade.

...

Madison officials say that a year after Wisconsin joined Reading First, in 2004, contractors pressured them to drop their approach, which blends some phonics with whole language in a program called Balanced Literacy. Instead, they gave up the money — about $2 million, according to officials here, who say their program raised reading scores.
...

Under their system, the share of third graders reading at the top two levels, proficient and advanced, had risen to 82 percent by 2004, from 59 percent six years earlier, even as an influx of students in poverty, to 42 percent from 31 percent of Madison’s enrollment, could have driven down test scores. The share of Madison’s black students reading at the top levels had doubled to 64 percent in 2004 from 31 percent six years earlier.

And while 17 percent of African-Americans lacked basic reading skills when Madison started its reading effort in 1998, that number had plunged to 5 percent by 2004. The exams changed after 2004, making it impossible to compare recent results with those of 1998.

No it didn't. It should have taken Diana Jean Schemo, this article's author, about half an hour on the internet to figure out that the "Madison officials" were spinning the "reading scores."

NAEP scores for Wisconsin show that the number of proficient students was 34% percent in 1998 and dropped slightly to 33% in 2003 and stayed there in 2005, the last time fourth graders were tested for reading. Students scoring at the basic level dropped from 69% to 67% during this same period. From the period 1992-2005, the achievement gap between black and white students rose from 28 points to 33 points and the gap between poor and non-poor students dropped slightly from 28 points to 25 points. So, NAEP shows us that the reading proficiency of Wisconsin fourth graders has basically remained flat since about 2000. (Go here and select Wisconsin as the jurisdiction.)

The Wisconsin Reading Comprehension Test (3rd grade) tells a different story. The number of proficient students in Wisconsin rose 22.5 points from 64.9% in 2000 to 87.4% is 2005. Based on our knowledge of NAEP scores for the same population, we know that this gain was imaginary. What Wisconsin did was a combination of making the test easier and lowering the cut score so that 22.5% more students would be able to pass it. The result is that the mean shifted by about +0.77 of a standard deviation.

Gather 'round, kids, and learn how Wisconsin made a fool out of the statistically illiterate Diana Jean Schemo.

Here's what I get from comparing the 1998-99 scores to the 2004-2005 scores:

In 1998 the percent of the state passing the exam was 64.9%. In 2005, it was 87.4%. A shift of 0.77 standard deviations (s.d.). Remember, during this same period NAEP scores for the state remained virtually unchanged.

In 1998 the percent of the Madison school district passing the exam was 58.9%. In 2005, it was 82.7%. A shift of .72 s.d. If anything, Madison underperformed the state during this period.

Update Three: From this source, I found out that Madison's Reading First Schools (the schools eligible for RF funds, but which never go them because Madison made them use a balanced literacy program) are Glendale, Hawthorne, Lincoln, and Orchard Ridge. As it turns out, the average gains made by these schools was only 21.6% or 0.56 s.d. In contrast, the non-Reading First schools gained 25.4% or +0.80 s.d. (Again, the percentile scores are deceptive because the Reading First schools caught the fat part of the curve, whereas the non-RF schools did not.) To put this in perspective, the 0.24 s.d. differential is about the same effect size found in project STAR (class size reduction) which educators rave about. So, the non-Reading First schools in Madison slightly overperformed the state average while the Reading First eligible schools significantly underperformed the state avergae.

I didn't find disaggregated data, so I can't determine if the stats for black students given to Schemo is accurate. But assuming they are, the rise in scores from 31% to 64% represents a shift of +0.86 s.d. That seems highly improbable considering 1) that Madison's overall rise was only +0.72 (this means that the scores for non-black in Madison rose only about +0.58, a significant underperformance compared to the rest of Wisconsin) and 2) black-white achievement gap on NAEP actually increased during this period, meaning that black performance actually declined.

Update: Rory finds the disaggregated data. In 2005, black performance in Madison was 57% which represents a shift of +0.68 s.d. which represents a underperformance of both Madison (+0.72 s.d.) and Wisconsin (+0.77 s.d.).

What actually happened was something like this:

Wisconsin raised the mean passing level from level 1 to level 2. The top curve shows black performance. This shift captured the fat part of the curve for black students (31% to 64%); picture level 1 being even further to the left. The blue shaded portion represents the increase in black performance.

Update two: Just in case it isn't clear, Madison's claim that black student performance "doubled" and exceeded the gains made by other students based on percentile changes is highly mispleading and a misuse of statistics. When you measure gains by the more statistically accurate standard deviation it is readily apparent that blac student performance was no greater than the gains made by other groups.. In fact, it appears to be considerably less. The achievement gap remains unchanged or may have even grew. This is what you would expect to happen under a whole language reading program because those students at the bottom of the curve are the ones most damaged by the non-explicit instruction.

The bottom curve shows white performance. The shaded portion between level one and level two shows the gain. I'd estimate that the white performance rose from about 69% to about 90%. (If anyone finds the disaggregated data let me know.) As you can see, white performance was already past the fat part of the curve--the average white student was passing the exam back in 1998.

Update: White performance in Madison was actually 93%. Ka-ching!

One thing is for certain, Madison's balanced literacy program did not cause the imaginary gains made by Madison school children.

Update: Mark Liberman of Language Log chimes in:

DeRosa's argument seems pretty persuasive to me. If he's right, then I'd consider a different evaluation -- was Schemo bamboozled by the Madison school authorities, or did she pick Madison, and spin the story as she did, in order to make an essentially dishonest point, suggesting that the mean old federal bureaucrats are trying to stop the dedicated local educators from continuing to use the methods that are helping their children so much?

I certainly don't want to discount the possibility that Schemo was a willing dupe. It could very well be that she went looking for a story and Madison told her what she wanted to hear.

Madison school district passes up free money

Instructivist tipped me off to this NYT article about the Madison Wisconson school district which passed up $2 million in Reading First grant money so it could continue to misteach children to read using their ideologically preferred, but research discredited, reading programs.

The article starts out with a real horror show:

Surrounded by five first graders learning to read at Hawthorne Elementary here, Stacey Hodiewicz listened as one boy struggled over a word.

“Pumpkin,” ventured the boy, Parker Kuehni.“Look at the word,” the teacher suggested.

Using a method known as whole language, she prompted him to consider the word’s size. “Is it long enough to be pumpkin?”Parker looked again. “Pea,” he said, correctly.

Young Parker isn't reading, he's guessing. I bet there was a picture of pea in the illustration that no doubt accompanied the passage that Parker was attempting to read. This is whole language--kid's aren't taught that the p stands for the sound /p/ and that ea stands for the sound /eee/. The student is supposed to identify words based on context clues, such as the shape of the word and the meaning of what he is reading, i.e., he is supposed to use his understanding of the surrounding words, sentences, or even paragraphs to help them read an unfamiliar word.

In balanced literacy when the student isn't able to guess the word correctly based on the context clues, the teacher throws him a bone and tells him that p stands for /p/ and ask him to guess again with this new tidbit of information.

In a real phonics class, the teacher first instructs the student that p stands for the sound /p/ and that ea stands for the sound /eee/ and that when the student reads the word "pea" he should blend the sounds /p/ /eeee/ to identify the word "pea." If the student knows the meaning of the word pea, i.e., it is in the student's oral vocabulary, he will comprehend what he has just read assuming that his fluency rate was sufficiently high.

What the whole language people get wrong is that it is not a productive reading strategy to identify words based on context clues. Using context clues for deriving the meaning of identified (decoded) words is perfectly ok and is what skilled readers do.

It is difficult or a skilled reader to appreciate just how wacky reading the whole language way really is. Skilled readers identify written words very rapidly and use context clues so seamlessly to ascertain the meaning of unknown decoded words that it is difficult to separate the identification (decoding) part of reading with the meaning deriving part. Skilled readers don't remember how difficult reading was when they were just learning how to read and their decoding skills were not as fast or accurate as they are today.

So let's simulate a reading passage in which you can only identify (decode) about 80% of the words. This passage reduces your ability to use phonics to identify words. You are stuck using whole word reading strategies to identify words and comprehend the passage. See how well you use those those strategies and context clues to read the following passage:

He had never seen dogs fight as these w__ish c___ f____t, and his firs ex__________ t____t him an unf________able l_____n. It is true, it was a vi_______ ex_________, else he would not have lived to pr_____it by it. Curly was the v_________. They were camped near the log store, where she, in her friend__ way, made ad_________ to a husky dog the size of a full-______ wolf, th____ not half so large as _he. __ere was no w_____ing, only a leap in like a flash, a met_____ clip of teeth, a leap out equal__ swift, and Curly's face was ripped open from eye to jaw.

It was the wolf manner of fight__, to st___ and leap away; but there was more to it than this. Th__ or forty huskies ran _o the spot and not com______d that s_____t circle. Buck did not com______d that s_____t in_______, not the e___ way with which they were licking their chops. Curly rushed her ant________, who struck again and leaped aside. He met her next rush with his chest, in a p________ fash___ that tum___ed her off her feet. She never re_____ed them. This was __at the on______ing huskies had w______ for.

Weeeeeeeee! Wasn't that fun? I'm sure you enjoyed your reading experience immensely. Imagine reading an entire book that way. And by the way, the passage was from Jack London, Call of the Wild, and is generally recognized as being great literature. According to the whole language people that should have instilled a love for learning in you.

Skilled readers eventually figure out the code and learn how to identify words with a great deal of accuracy. However, for many kids, text continues to look like this because they fail to learn the code in the absence of explicit phonics instruction.

But this isn't the worst part of the article. I'm saving that for part two.

How not to teach comprehension

It's been a while since I deconstructed an inane education article, but I think this Seattle Times article, Revving up reading in Marysville schools, is worthy of that treatment:

On a recent Monday morning at Cascade Elementary School in the Marysville School District, veteran teacher Lauri Hagglund engaged in a timeless activity — reading aloud to her second-grade students.

The children sit cross-legged or up on their knees on brightly colored carpet squares. The teacher stops at the end of each page to display the book's illustrations.

But pull the lens back a few feet and the classroom becomes a laboratory for practicing the latest approaches to literacy. A reading coach sits a few feet away from Hagglund, charting plot details about cause and effect from a story involving a very tidy cat and a very messy one.

The problem with read-alouds is pretty clear. During the read-aloud the student isn't actually reading! And what second grade students need to be doing more than anything else is practicing reading.

A lot.

But, let's get beyond that.

What is most string about this "laboratory for practicing the latest approaches to literacy," as you will soon see is how primitive the laboratory actually is. I would have thought that the level of "experimentation" would have been a tad more sophisticated in 2007.

This isn't even the beta version of reading comprehension. In this school, reading comprehension is still in the alpha stage and this is the first trial run through to catch bugs.

At the edge of the reading carpet, several observers, including the district's superintendent, assistant superintendent and school principal, take notes.

When Larry Nyland took over the district in 2004, the new superintendent launched an initiative focused on strengthening students' reading skills. Standardized test scores in the district were among the county's lowest, and classroom assessments showed that even students who read stories at grade level sometimes struggled to understand their science and history books.

It's always worse than you think. They've been in alpha testing for three years now.

Let's go to the videotape and see what the reading scores are:

Fourth grade: percent meeting standard (state average)

2003-04: 75.6% (74.4%)
2004-05: 82.8% (79.5%)
2005-06: 84.9% (81.2%)

Slightly better than average. But look at the third grade scores (only tested in 05-06):

2005-06: 60.9% (68.3%)

Below average.

And, finally let's use the reality check to see how inflates those scores really are: Fourth grade NAEP reading scores for Washington: 33% proficient or better in 2003 and 36% in 2005. That's a nice little 45 point discrepancy.

Perhaps, they'd be getting better results if they wore white lab jackets, thick horn-rimmed glasses, and had shiny silver clipboards because that's what I hear all the real scientists wear when they conduct science-like experiments.

The district developed focused training for teachers on ways they could help students more readily grasp meaning and deepen their thinking about books. Principals learned how to observe and give feedback to their teachers. Reading coaches at every elementary school worked with struggling students.

Seems like they forgot the most important step. Step One: determine if the intervention works. Step Two: disseminate and train.

In January, they began presenting lessons in classrooms while teachers observed both the techniques and their effectiveness with the children.

The elementary classrooms were also re-imagined so that each had a communal-reading area where the children could sit comfortably and closely to the spoken words.

Maybe instead of re-imagining the classroom they should have re-imagined their concept of reading. Last I checked reading involved getting close to the text of books, not getting close to the "spoken word" of a teacher reading the book for you. No wonder the kids have a hard time reading.

District administrators began making regular visits to classrooms to gauge how the training might be furthered or refined. Last year, Nyland and his staff made 500 such visits.

For many teachers, it was the first time they'd been regularly observed and given feedback since they were students, Nyland said. And some were wary of the approach.

"Up until now, administrators were only in the classroom to evaluate teachers," Nyland said. "We had to show them our focus was on improving student learning. It's like athletes watching game film to see what they're doing and what they might want to change."

What is it with teachers being so reluctant to be observed by management? They need to get over that real quick.

Back in the classroom, the school's reading coach, Leanne Rivas, asks the children how the two cats in "The Tale of Two Kitties" feel about each other. In the illustration, they sit at opposite ends of a long fence, their tails toward each other.

Ostensibly, this is a lesson on reading comprehension. And here we are discussing an illustration in the book. I think there's some cognitive dissonance in play here.

Let me suggest a better exercise to teach drawing inferences from text. The students read the following passage: "Linda and Cathy were alone on an island. Linda said, 'Stop crying, Kathy. We are both very smart, and if we use our heads, we will get out of here.'" Afterwards, the teacher asks: "Why was Kathy crying?"

That micro- exercise comes from lesson 68 of Reading Mastery III, a second grade curriculum. The instructional content of that lesson is light years ahead of what's going on in this classroom. Go take a look.

"Turn and talk," Rivas instructs, and on cue, the students turn to a partner to discuss the cats' mutual disdain. The approach seems more typical of a grown-up reading group than an elementary-school classroom, but Assistant Superintendent Gail Miller said that by recalling details, summarizing their thoughts and finding evidence for their views, the students go more deeply into the text and practice skills they'll use all of their reading lives.

No, they're talking about a picture of two cats. One kid is talking to another kid about a picture. Then the other kid takes a turn. Missing is the feedback and assessment by a teacher to determine who understands and who doesn't.

Some shortcomings of the approach are quickly apparent. Confident and chatty children give their opinions first. When Rivas counts backward — three, two, one, time for talk is done — several quieter students haven't said a word.

Another challenge emerges. Rivas is drawing boxes around the plot elements to help illustrate cause-and-effect. Writing the students' own words down also helps them summarize their ideas and gives kids just learning English a chance to see their spoken words in print, but her examples of cause-and-effect get a little lost in all the plot elements.

In short, this is a poorly thought out and even more poorly executed lesson. It is confused. The point of the instruction is unclear. Assessment of individual students is impossible. It is a waste of time from beginning to end. So why was it being presented to the students in the first place. This one should have been strangled in the cradle.

Still, the pedagogy doesn't interrupt the students' enjoyment of the book. Most laugh and point as the cats, Fluffy and Scruffy, team up to rout some impudent mice.

I think they call that being off-task.

I wonder how many of these students know what the words "rout" and "impudent" mean? Not knowing the meaning of words like this is what causes comprehension difficulty in the later grades. Yet, the teaching of vocabulary will be downplayed in classrooms like this in favor of these hokey reading strategies.

About a half-hour after it began, what the district calls a reading "walk-through" ends.

You don't want to know what I call it.

The school administrators regroup with the reading coach in the office of Cascade's principal, Chris Sampley. Miller, who previously worked with student teachers at Seattle Pacific University, asks Rivas to recall the purpose of the lesson and how it went.

Rivas says she realizes she strayed from the initial goal and isn't sure what the students learned.

That pretty much sums up what's wrong with education today. No one knows what the students actually learned.

And here's problem number two:

The observers are less hard on her than she is on herself. Miller reminds her how at different points she guided the students back to the text, the same strategy a more experienced reader would use to search for meaning or relationships.

Everything is all right, sweetie. You just keep working at it. You'll get better. Someday. hopefully soon. Or not.

Rivas, in hindsight, thinks "The Tale of Two Kitties" was better suited to demonstrate compare-and-contrast than cause-and-effect. The observers agree that one challenge of their literacy effort is that each book has to be dissected for its best lessons in advance.

It took three years to come to that realization? This is why we have NCLB. This should have been done thirty years ago.

At lunch time, she'll sit down with the classroom teacher and review any thoughts both of them have.

A week later, Rivas and Hagglund are teamed up again with another book. Some adjustments are immediately evident. The students are instructed to take turns talking about the book so no one is left out. Rivas is also more deliberate about what she chooses to chart, emphasizing cause-and-effect, which she notes is both a life lesson and a question likely to appear on the Washington Assessment of Student Learning.

See I was wrong. I had initially assumed that the students were instructed to take turns discussing. They weren't. And, it took these two trained professionals a week to figure that out.

The main problems with the lesson are still present: the teacher is reading the book, not the students, the teacher is making the chart, not the students, and the teacher still has no idea if each student learned anything.

After class, Hagglund talks about the literacy initiative and how, after 27 years in the classroom, it's changing her teaching. More than anything, she says, the literacy initiative has reminded her what it's like to be a student again, to be asked to do something new and hard.

"It puts us all in the place of being learners," she said.

I thought that's what ed school was supposed to be for?

Our School out in paper back

Joanne Jacob's book, Our School: The Inspiring Story of Two Teachers, One Big Idea and the Charter School That Beat the Odds, is now out in paperback for you thrifty folks.

Things have changed since the hardcover edition. Charters are now in vogue:

When the hardcover came out, the publisher insisted on taking “charter” out of the subtitle, saying that it would depress sales. For the paperback, the publisher decided to put it back. Apparently, they think charter schools are hot.

Just to show you my street cred, my bookmark to Joanne's site is so old it still says Read Jacobs, the name of Joanne's blog when she was pretty much the only edu-blogger on the net. My how times have changed.

Bubble-gate

Under NCLB states were supposed to develop challenging academic standards and create assessment devices aligned to those standards. The intention was that schools would then teach the material set forth in the academic standards in whatever way they saw fit. Afterwards, the assessments would then be employed to measure whether the students who had been taught had actually learned the material covered by the standards. In short, the assessments determined whether the teaching was effective.

In order to comply with NCLB, most schools would have to improve their teaching outcomes or they would eventually fail to make adequate yearly progress (AYP).

Many schools have responded to NCLB by engaging in activities that appear to be aimed at raising student outcomes in the short term without regard to their long term consequences. For example, the WaPo worked itself up into a lather recently when it lamented the phenomenon of schools concentrating resources on the "bubble kids," kids on the cusp of passing the assessments, by giving them intense test prep sessions.

"We're not talking about instruction," said Bonnie Cullison, president of the Montgomery County Education Association, who is investigating teacher concerns at the school. "We're talking about a narrow set of skills that is really about passing a test."

Few doubt that the test prep is largely a waste a time. But who's kidding who. The reason why all the test prep is going on is because the schools aren't confident that the "instruction" was working in the first place.

But none of that matters anyway. Under NCLB, the chickens of bad instructions will eventually come home to roost sooner or later. A school only has so many hours of instructional time to teach. It either makes productive use of that time or it doesn't.

Extensive test prep seems like a waste of time to me, but lots of what most schools do is a waste of time.

Teaching kids that reading is a psycholinguistic guessing game--that's a waste of time. A big waste of time.

Teaching kids by playing hide the instructional ball--that's a waste of time too.

So is most everything that gets peddled as instruction on the elementary level.

I find it difficult to get worked up over the misuse of test prep when the only thing being lost is time spent misteaching other material.

In the real world, bad practices get kicked to the curb all the time. This is a feature, not a bug, of any accountability system. The only question is whether NCLB is robust enough to work as a decent accountability system. I don't believe that it is.

The Counter Example

More people should visit inner city schools to see these educational horror stories first-hand. These are the places that demoralize teachers. Where teachers develop their pernicious beliefs that children like this can't or won't learn. These schools are badly broken. In these schools, traditional education techniques barely stand a chance. You can forget about the faddish child-centered techniques. Education apologists would have you believe that schools like this don't exist or that they can be turned around with lots of money and a generous application of the latest ed-school fad.

The following is the condensed story of how the National Institute for Direct Instruction turned around the worst school in the inner city of Baltimore, a seemingly impossible task. The percentile scores given in the story are from the CTBS, a legitimate test similar to the Iowa Basic Skills test. The entire story appears on pages 17-24 of Chapter 7 of The Outrage of Project Follow Through: 5 Million Failed Kids Later. Lots of gory details are provided. Almost all of the excuses given by educators were present at City Springs and then some:

Possibly the most notorious school in the district at the time was City Springs. It was in a high-crime black neighborhood and had the lowest achievement scores of the 119 elementary schools in the district. The average performance of the students in City Springs when we started was below the 10th percentile in all grades and all subjects. 98 percent of the students were on free-and-reduced lunch schedule, which is used as an indicator of the poverty level of schools.

Two of our trainers visited City Springs in the spring before the implementation began. Their summary statement was, “It’s a zoo; totally out of control.” Children were running through the halls when they were supposed to be in classrooms. In many classrooms, kids were fighting and throwing things. Most teachers had no control over the children. During recess, the behavior of children on the playground could be conservatively described as rowdy and dangerous for the
younger children.

City Springs proved to be as challenging as any school we’ve ever implemented. The students had been reinforced for their behavior. They knew they could get away with it, so when we robbed them of their “freedom,” they would try harder to reinstate it.

Substitutes always refused to go to City Springs, which made the job of turning the school around even more difficult during the winter, when several teachers were absent, and the logistics made it problematic to put two classrooms of children in a single room.

Progress in turning the school around was slow, and if the school had weak leadership, it never would have happened. When I visited City Springs during the second year of the implementation, the principal first took me to three classrooms that looked okay, but certainly not outstanding. Then she said, “Those are my best teachers. They give us something to build on, but I don’t know what to do about some of the others.” She took me on a tour of possibly six teachers she judged to be her worst. I had suggestions for half of them. After we observed the rest, she asked, “What do I do about those teachers?” I told her that I didn’t know, short of getting rid of them. She responded, “That won’t happen soon. What do I do in the meantime?” I didn’t have any answers except for us to do the best we could by using aides or other school personnel to work in those classrooms.

Replacing teachers is not a routine assignment for a school as notorious as City Springs. The only teacher the district sends to replace a failed teacher is either a first-year rookie, a misfit, or a reject from other schools. Teachers not in these groups typically have other options, and they won’t go to schools like City Springs. After the first year, 19 new teachers were installed. The new population of teachers was not much better than the teachers they replaced. So the second year was like starting over—rolling a rock up a hill one more time.

The misfits often had good potential and became high performers. Some of the rejects were as bad as the teachers they replaced. For first-year teachers, the City Springs experience went beyond culture shock—more like undiluted horror.

Nothing in their college work had prepared rookies for the students. Most of them came in with expectations of teaching and interacting with students that were shattered within the first hour in the classroom. Even those who had studied “behavioral principles” were quickly traumatized by the rude realities of City Springs.

The county presented another barrier to improving City Springs and our other Baltimore schools. Salaries for teachers in Baltimore were significantly lower than in the three counties that surround the city of Baltimore. These counties wanted good teachers because the population of the counties was growing while that of Baltimore had been shrinking. So there was a market for our better teachers. As their skills increased, the probability that they would defect to one of the counties increased. A rough guess is that we lost 15 percent of our teachers every year to the counties. A pattern for the younger teachers was to work two years in a DI school and then defect to a county school. I couldn’t blame these teachers. Some actually lived a lot closer to their new county school than they did to their former city school.

Of the second–year teachers, at least one couldn’t even read the script and another seemed to be insane; however, some of the teachers and aides got a lot better. The coordinator worked patiently with one first-grade teacher who could scarcely read and who could not produce the sounds for the letters and combinations that she was trying to teach to the children. She wanted to learn; she worked hard; and she became much better. After one practice session with the coordinator, she gently put her hand on the coordinator's forearm and said tearfully, “Thank you so much. Nobody has ever worked with me before.” She was a certified teacher, however.

A serious problem in the upper grades was that some teachers did not know the content of what they were trying to teach. They needed to go through the program as students before trying to teach it.

By the spring of 2000, City Springs’ neighborhood hadn’t changed. Nor had the middle school across the street, which was still loud, disorderly, and had regular visits from the police. City Springs, however, had changed. It was far from the lowest school in the district. First-grade performance had improved from below the 10th percentile to the 75th percentile in reading. The school had order. Children had bought into the role of being good students. There was a lot of positive reinforcement and greatly improved teaching.

With K and 1 in place and getting better, the scene was set for the subsequent grades to inherit higher performers and achieve better results. If the performance of children remained at around the 75th percentile, the improvement would be spectacular. Over the next three years, however, the teachers in K and 1 continued to improve and the performance of their students reflected this improvement.

In 2001, first graders achieved at the 80th percentile in reading.
In 2002, first graders achieved at the 91st percentile in reading.
In 2003, first graders achieved at the 99th percentile.

City Springs first graders had gone from the lowest in the district to the highest. The fifth graders did not achieve as high as the first graders, largely because of more than 20 percent annual turnover of students; however, by 2003, the fifth grade performance was one of the highest in the district, well above average: the 87th percentile in reading and the 79th percentile in math. One classroom had finished the 6th level of the DI programs and completed a middle-school textbook on U.S. history. When these students visited Monticello, the tour
guide indicated that their group was the most intelligent the guide had conducted.

In 2004, NIFDI had a falling-out with City Springs. The school wanted to go in a different direction than the one that brought it to where it was. The school believed that it should place more emphasis on test preparation. We indicated that we would not work with the school if it chose that direction. It chose that direction, and NIFDI dropped it in the spring of 2004.

----------
The improvement in the academic performance of the fifth graders in City Springs represents an effect size of about 2.5 standard deviations in reading and about 2.0 in math. It took seven years to turn City Springs around. That's a little over half the time we've given schools to turn around under NCLB. Don't let it be said that it's impossible for most schools to meet the mandates of NCLB.

The Better Mousetrap

The seventh and final chapter of Zig's book is now available for download.

Zig gives us a new twist on Emerson's mousetrap analogy as it applies to education

I’ve made several references to building a better mousetrap. The original saying came from the famous American essayist, Ralph Waldo Emerson. He wrote, “Build a better mousetrap and the world will beat a path to your door.” If Emerson’s quote addressed the actual strategies of U.S. education, it would go something like this: “Build a better mousetrap and the educational community will lie about either your mousetrap, where your door is, or what constitutes beating a path to a door.” (p. 54)

This last chapter provides much information on the DI implementations since 2000. The Baltimore project is particularly interesting. I'll devote a post to it later.

Zig concludes:

If a mind is a terrible thing to waste, the educational establishment is guilty of incomprehensible waste.

Indeed.

Teacher Autonomy

Teachers will often tell you that they need the freedom to be creative. They need flexibility to modify and expand on the textbooks and other curricular material the school makes them use. They know how to improve the deficiencies of bad curricula. In short, teachers would have you think that they know best in all things instructional.

But do they?

There is quite a bit of evidence that shows that they do not. The evidence shows that teachers do not know how to improve a deficient curriculum and that teacher performance is limited by the instructional program they're using. The highest level of student performance is dictated by the curriculum, not by the ability of the teacher.

In 1981, Zig Engelmann and Don Steely performed a study that directly addressed the issue of how much teachers rely on instructional material, and how effectively they interpret, expand, and ask questions to teach skills and concepts that are presented in the students’ textbook. It is important to note that the teachers in this study worked with average students, not at-risk students.

The researchers first analyzed the more popular reading programs in grades 4–6. According to Englemann all the programs had the same format:

They presented a series of topics over the course of the year—main idea, fact versus opinion, context clues for word meaning, cause and effect, key words, and a few others. Each lesson would take a few days. The lessons were in cycles so that a lesson on main idea would occur now and would be followed by lessons on all the other topics. After this round of lessons, the cycle would then be repeated to further develop the topic. On the average, the lessons for any given topic occurred about every 50-60 school days.

The researcher s interviewed 17 teachers on questions about critical beliefs and facts. The teachers believed that 86% of children should master any skill that is taught and that only 58% could master master main idea.

After the interview, the teachers were taped teaching two topics from the program they used. The teachers were taped teaching main idea and another topic selected by each teacher. The teachers were instructed to teach the way they normally would.

The researchers analyzed the teaching and compared it to the directions and specifications provided by the program. They also designed tests that evaluated how much of what was taught was learned by students. The testing occurred immediately after the teaching of each topic. Each test was custom designed, so that it used the same language specified in the book and items very similar to those the teacher had presented.

The reasearhers had analyzed the programs the teachers were to use before the experiment:

The analysis of the programs (which occurred before the teaching) implied that the teaching would be unsuccessful... The problem was created by the programs and by the examples it presented. For the first example, the main idea was the first sentence. For the next two examples, the main idea was either the first sentence or the last sentence. The final example contained no sentence that expressed the main idea. This sequence is analytically crazy. You don’t give naïve students three examples of a sentence expressing the main idea followed by an example that breaks this rule, without warning or preparation.

The study noted many inconsistencies between what the teachers said they did and what they actually did. For example, the tapes of each teacher teaching showed that the teachers followed the specifications of the program very closely. However, their verbal reports indicated that they deviated from the directions more than they actually did. They asked a fair number of additional questions and added some explanations, but they presented all the examples the material indicated and followed what wording the teacher’s material suggested.

Here's Engelmann's conclusions from the study:

The comparison of their teaching with the directions and specifications of the programs they used showed that no teacher taught better than the program. In other words, they tried to improve on what the program presented, but they didn’t know how. The fact that no teacher taught better than the program specifications casts doubt on the assertions that teachers are able to adapt and respond to the students in a way that is superior to a canned program. The program may be viewed as a limiter. Except for an occasional exception, a careful analysis of the program reveals how the best teachers will perform. Where the program has logical problems, the observed teaching has predictable performance problems.

...

The data showed that only ten percent of the students mastered main idea and only a little more than half of them got a score above 50 percent correct. The topic that had the lowest scores was context clues. Only 15 percent of the students got half or more of the items correct. No student achieved mastery. Overall performance across all topics showed that 12 percent of the students achieved mastery at the level of 90 percent correct. Interestingly, the topics that educators suggested were most important—higher-order skills like identifying main idea and using context clues—were the areas of poorest student performance.

The questionnaire we designed revealed that the teachers were aware of the mastery problems. Teachers indicated that over half of their students needed one more week of instruction on just main idea. We sent the same questionnaire we used with the teachers we taped to 3,000 other teachers. About 500responded. Their responses were quite consistent with those of our teachers.I wrote an article on this research for the Association for Direct Instruction News (Spring 1982). I ended with this observation:

This study made me feel very sad, not so much because the results surprised me, but because the tapes of the teachers revealed both concern and a lot of raw talent. Most of the teachers who volunteered forth is study were clearly intelligent people who were trying very hard to do an important job. Their verbal responses and the questionnaire responses suggest that these teachers are quite aware of the … learning problems that their students experience. They know, for example, that students tend to confuse the title or the first sentence with the main idea. Teachers simply don’t know how to avoid this problem, how to teach in a way that will help solve it, and how to provide explanations and examples that correct the problem. As it is, their talent, their potential to be super teachers, is unfulfilled, in the same way the potential of their students is.

(This passage is adapted from pp. 24-27 of The Outrage of Project Follow Through, chapter 6.)

Corroborating this evidence is the flat performance of student achievement on the NAEP for the past 30 years. Lost of fad have come and gone during those 30 years, all the while student performance has remained essentially flat. All the curricular tinkering was for naught. In fact, we put this theory to the test on a very large scale basis in the second half of Project Follow Through. The results were predictable:

There were 15 self-sponsored districts, and many implemented in more than one school. If Follow Through had been limited to the 15 self-sponsored communities, it still would have been the largest educational experiment ever conducted.

The performance of these sites would be important to those in the educational community who believed that teacher autonomy and collaboration would transform failed schools into effective ones. Nearly all the self-sponsored sites promoted teacher autonomy and school autonomy. They permitted enormous latitude in how teachers developed material and practices to meet the needs of children, and nearly all had provisions for teachers working collaboratively to meet the children's need. If judged on the appeal of goals and practices, these sites should have been 10s. In fact, all failed. Incredibly, the self-sponsored sites performed below the average of the sponsored sites, which was considerably below the level of children in Title 1 schools.

Instructional innovations are not coming from our teacher core, from schools, or from our education schools. And, there's no reason to believe that any of these institutions are capable of anything more than the continuation of the status quo. Teacher's do not seem to know best, they seem only to know the status quo. Improvements in education won't be coming from inside the system, at least not voluntarily.

Teachers Resist Change

Peter Senge, a systems-theory expert and advocate of "learning organizations" in business, relates an anecdote that reveals teachers' perspectives on change. He asked a group of educators if systems change occurs only in response to crisis. Usually in business groups, three-quarters will agree that's true, but the response in the education group was different. When he asked the question, very few said yes. So he asked, "Does that mean you believe significant innovation can occur without crises?" but again no one raised a hand. He went on, "Well, if change doesn't occur in response to a crisis, and if it doesn't occur in the absence of a crisis, what other possibilities are there?" Finally somone from the audience responded, "I guess we don't believe significant change can occur under any circumstances" (cited in Smith, 2001, p. 30).

From Myths and Misconceptions about Teaching

I believe that this is the majority view among teachers. I also believe that is is a rational response by teachers who see a large number of failing student first hand day in and day out. Inherent in this point of view is the realization that teachers are powerless to improve student outcomes. It is beyond their ability to improve student performance and they know it. These teachers have seen a parade of ineffective reforms pass through their classrooms, most of which claimed to have a research base, albeit an unscientific research base. These teachers have settled into a status quo position.

There is a minority, yet vocal, teacher position that refuses to accept the fact that teachers are unable to educate a large segment of the student population. There is a large body of objective evidence which shows that teachers holding this position are wrong. Nonetheless, these teachers have woven a web of rationalizations that permits them to discount this data and maintain the untenable position that teachers know best. They don't. And I'll show why in my next post.