The NYT reports that the Madison Wisconsin school district turned down $2 million dollars in Reading First grant money. The reason: they didn't want to use a phonics-based reading program as required by Reading First. Instead, Madison wanted to continue using the district-created "balanced literacy" reading program. Nothing wrong with that per se; Reading First is a voluntary program entailing federal oversight.
But with NCLB requirements for increased student proficiency looming on the horizon, the pressure is on for low-performing school districts, like Madison, to increase student performance. It would be quite an embarrassment if Madison's reading program failed to produce results, considering their very visible refusal to adopt a phonics-based reading program--especially since there was grant money attached.
So what do you suppose Madison did when the agenda-driven NYT came looking for a poster child of a school district bullied by the Feds in the (phony) scandal-plagued Reading First program? They cooked the books.
Under their system, the share of third graders reading at the top two levels, proficient and advanced, had risen to 82 percent by 2004, from 59 percent six years earlier, even as an influx of students in poverty, to 42 percent from 31 percent of Madison’s enrollment, could have driven down test scores. The share of Madison’s black students reading at the top levels had doubled to 64 percent in 2004 from 31 percent six years earlier.
And while 17 percent of African-Americans lacked basic reading skills when Madison started its reading effort in 1998, that number had plunged to 5 percent by 2004. The exams changed after 2004, making it impossible to compare recent results with those of 1998.
Madison's scores did rise from 58.9% in 1998 to 82.7% in 2004, an apparent rise of +0.72 standard deviations (s.d.) which is a large effect size. But the question remains: did student performance actually improve?
Let's find out.
Madison's scores rose less than Wisconsin's
According to the Wisconsin Reading Comprehension Test (3rd grade), the number of proficient students in Wisconsin rose 22.5 points from 64.9% in 1998 to 87.4% is 2005, an increase of +0.77 s.d. So it wasn't just Madison's scores that rose, scores rose across the board in Wisconsin. If anything, Madison's scores rose slightly less than the average score in Wisconsin.
But, is such an across-the-board gain in achievement realistic in the first place?
Wisconsin's NAEP scores remained flat
NAEP scores for Wisconsin show that the number of proficient (and above) students was 34% percent in 1998 and dropped slightly to 33% in 2003 and stayed there in 2005, the last time fourth graders were tested for reading. Students scoring at the basic level (and above) dropped from 69% to 67% during this same period. From the period 1992-2005, the achievement gap between black and white students rose from 28 points to 33 points and the gap between poor and non-poor students dropped slightly from 28 points to 25 points.
So, NAEP shows us that the reading performance of Wisconsin fourth graders has basically remained flat since about 2000. (Go here and select Wisconsin as the jurisdiction.)
The NAEP scores tell us that Wisconsin's miraculous gains in reading achievement from 1998-2005 are non-existent. Wisconsin did what most states did in response to NCLB, they goosed the tests to artificially increase test scores to give the appearance of increased student achievement.
Let's recalibrate Wisconsin's gain of +0.77 s.d. to 0.0 s.d. to account for the non-gains made in NAEP. This means that Madison's real performance during 1998-2005 actually declined by -0.05 s.d.
Madison's schools eligible for Reading First funding performed significantly below Madison's other schools
According to this source, there were four Madison schools eligible for Reading First funding: Glendale, Hawthorne, Lincoln, and Orchard Ridge. These schools never received this funding because Madison choose to stick with its Balanced Literacy reading program. Let's see how those schools performed.
The average gain made by these four schools was only 21.6% or +0.56 s.d. This is a significant under-performance compared to the statewide gain of +0.77 s.d. Using our NAEP recalibration, the performance of these schools actually declined by -0.19 s.d. That's big.
In contrast, the average gain made by the remaining schools in Madison was 25.4% or +0.80 s.d. which is about the same gain made statewide.
The Disaggregated Data
In the NYT article, Madison officials made some wild suggestions about the relative performance of black students compared to other students (mostly white) in Madison. I showed in my previous post how these percentile gains are misleading due to how scores are distributed (a normal distribution), so I won't repeat it here.
In fact, I believe the scores that Madison is using are inaccurate. Madison's numbers have black performance rising by +0.86 s.d., which is high. I can't find disaggregated data for the WRCT, but the disaggregated data from the 3rd grade performance on the new WSAS test shows that black performance is significantly less than this and less than the average gain made by white students. It's not an apples to apples comparison, but it is consistent with the rest of the data.
Conclusion
Madison is cooking the books.
Its schools slightly underperformed Wisconsin schools and Madison's other schools.
In fact, NAEP data shows that the gains made by Wisconsin are illusory. It's doubtful that scores rose at all in Wisconsin.
If we look at only the schools in Madison that were eligible for Reading First funding, we see that these schools performed significantly worse than other schools in Wisconsin.
So it appears that Madison's Balanced Literacy reading program, which cost the district $2 million, failed to increase student performance in Madison and actually caused a relative decline in the schools that were supposed to get Reading First funding.
This is exactly what we expect to see in your typical balanced literacy program, at-risk children failing to achieve. These are the children most damaged by "balanced literacy" programs, kids with low language skills and background knowledge. These were the kids that Reading First intended to serve.
16 comments:
Thanks -- this does the trick.
I do have one question: are kids with low background knowledge necessarily more harmed by a balanced literacy program?
Offhand I would think that low background knowledge kicks in as a major obstacle to reading comprehension scores a bit down the line.
I'm thinking I've read (perhaps in Moats?) that at-risk kids are equally distributed across all SES groups.
Sorry -- I mean kids at risk for problems decoding are (or may be) equally distributed across SES groups.
Reading comprehension problems are directly linked with SES.
Here it is:
[Reading scientists] established that most students will learn to read adequately (though not necessarily well)
regardless of the instructional methods they’re subjected to in school. But they’ve also found that fully 40 percent of children are less fortunate. For them, explicit instruction (including phonics) is
necessary if they are to ever become capable readers. These findings are true across race, socioeconomic status, and family background.
A parent here, whose child is a struggling reader, tells me that our K-3 school is "crawling" with reading specialists.
The middle school has, I believe, 12% disadvantaged kids; I don't know whether the percentage is the same in K-3 though I assume it is.
In any case, we have a LOT of high-SES white children who are having reading problems.
iirc, we've also hired an assitant principal K-3 who is an expert in balanced literacy.
The quote comes from "Whole-Language High Jinks: How to Tell When "Scientifically-Based Reading Instruction" Isn't by Louisa Moats
availabe at the Fordham Foundation website
Anyone out there from Wisconsin's 2nd congressional district? Any additions, corrections, improvement in tone (esp the last sentence) for the following letter:
The Hon. Tammy Baldwin
House of Representatives
Washington, D.C. 20515
Dear Madam Baldwin,
A school in your district was recently featured by the New York Times. The Times reporter neglected to mention that this school is underachieving relative to other Wisconsin schools. We hope you will take the opportunity to identify and familiarize yourself with an exemplary school system in your district to ensure your role in setting national education policy reflects the needs of dedicated educators rather than PR mongers.
Wisconsin uses criteria from the National Institute of Standards and Technology to identify performance excellence in education. However, (unlike Texas, New York, Maryland and others) no Wisconsin school system has earned their state's award (i.e. the Governor's Forward Award for Excellence).
In lieu of excellence relative to nationally recognized criteria, Wisconsin school boards, teachers' unions, and schools of education might be able to suggest an exemplary school system in your district. The National School Board Association's Federal Relations Network, Wisconsin Education Association, and Wisconsin education deans can provide points of contact to assist your office.
For example, local district school board members are oath-bound to ensure "the opportunity for students to be proficient in ... reading and writing ..." (Vincent v. Voight, 2000). Similarly, the National Education Association committed itself to improving student achievement through "new unionism," which would, if faithfully supported, eliminate the lure of providing private-school vouchers to children in faltering public schools. Finally, schools of education prepare teachers as ethical practitioners who meet their legal obligations. Perhaps a school of education could suggest a school system in your district where students benefit from ethical practitioners trained in cost-effective educational practice.
Of course, you might discover that local school board members are elected without assessment of the training required or their capacity to fulfill their oaths of office; that teachers have abandoned the challenge of new unionism; and that schools of education provide mere lip service to ethics while advocating "reforms" that constitute unwarranted experimentation on involuntary human subjects. Should this be the case, please do the nation a favor and refrain from representing these constituents in matters of national education policy.
Sincerely,
The Ed Blogosphere
I know this is a repost from last comment, but I think it is important to highlight that at least one person on the Madison Board of Education, realized there was a disconnect between the Wisconsin tests and the NAEP as far back as 2004.
Here is a letter that Ruth Robarts of the Madison Board of Educatin wrote into their local paper.
Thanks to Jason Shepard for highlighting comments of UW Psychology Professor Mark Seidenberg at the Dec. 13 Madison School Board meeting in his article, �Not all good news on reading�. Dr. Seidenberg asked important questions following the administration�s presentation on the reading program. One question was whether the district should measure the effectiveness of its reading program by the percentages of third-graders scoring at �proficient� or �advanced� on the Wisconsin Reading Comprehension Test (WRCT). He suggested that the scores may be improving because the tests �aren�t that rigorous�.
I have reflected on his comment and decided that he is correct.
Using success on the WRCT as our measurement of student achievement likely overstates the reading skills of our students. The WRCT---like the Wisconsin Knowledge and Concepts Examination (WKCE) given in major subject areas in fourth, eighth and tenth grades--- measures student performance against standards developed in Wisconsin. The more teaching in Wisconsin schools aims at success on the WRCT or WKCE, the more likely it is that student scores will improve. If the tests provide an accurate, objective assessment of reading skills, then rising percentages of students who score at the �proficient� and �advanced� levels would mean that more children are reaching desirable reading competence.
However, there are reasons to doubt that high percentages of students scoring at these levels on the WRCT mean that high percentages of students are very proficient readers. High scores on Wisconsin tests do not correlate with high scores on the more rigorous National Assessment of Educational Progress (NAEP) tests.
In 2003, 80% of Wisconsin fourth graders scored �proficient� or �advanced� on the WCKE in reading. However, in the same year only 33% of Wisconsin fourth graders reached the �proficient� or �advanced� level in reading on the NAEP. Because the performance of Madison students on the WCKE reading tests mirrors the performance of students statewide, it is reasonable to conclude that many of Madison�s �proficient� and �advanced� readers would also score much lower on the NAEP. For more information about the gap between scores on the WKCE and the NAEP in reading and math, see EdWatch Online 2004 State Summary Reports at www.edtrust.org.
Next year the federal No Child Left Behind Act replaces the Wisconsin subject area tests with national tests. In view of this change and questions about the value of WRCT scores, it�s time for the Board of Education to review its benchmarks for progress on its goal of all third-graders reading at grade level by the end of third grade.
Ruth Robarts
Member, Madison Board of Education
End Quoted Letter
This seems like important information for the papers in Madison.
Madison Supt claims they were told we had to use one of the preferred reading packages authorized by USDOE
Sens Kohl, Feingold and Rep Baldwin write USDOE, November 3, 2006: [O]ur constituent, Mr. Art Rainwater, Superintendent for the Madison Metropolitan School District ... [seeks] reinstatement of lost federal resources to the Madison Schools from the Reading First program
Madison Supt latter claims "We turned down the Reading First Grant because the Technical Assistance Center at the University of Oregon required us to make changes ..."
Surely any ed school graduate, fully trained in the duties of citizenship, ought to be able to assess whether Supt Rainwater has ensured sufficient plausible deniability lest his Representative and Senators aid and abet fraud, waste, and abuse...
This is really nice work you're doing. The fur is flying in response to the NY Times article, with lots of letters being submitted from national figures. Whether the NY Times decides to run any of them--or which of them--is of course unknown.
I have a question. It's clear that the NAEP scores are flat and discrepant with the WCRT data. But, a skeptic could ask: what were the NAEP data like for Madison alone? This information is not downloadable from the NAEP site. Does anyone know if it is accessible?
thanks.
Mark Seidenberg
mark, I don't think that data is available at the sub-state level with the exception of the few large cities being tracked. Madison unfortunately doesn't fall into that list.
And, I don't suppose that Madison uses any of the popular commercially available reading testing instruments. It only takes a few minutes to administer the DIBELS test which would show exactly which students had the necessary 3rd grade decoding skills.
From an email from the author of the article:
The argument seems to boil down to:
A. Wisconsin's performance on NAEP from 1998 to 2005 is flat.
B. Therefore any gains that showed up on the state test are illusory, i.e., manipulation of test scores and such;
C. Madison, where test scores fell short of state gains (0.77 standard deviations vs. 0.72 sd for Madison) actually did worse than the state, and there was no improvement in student performance.
D. And the black white achievement gap actually grew, it didn't shrink.
Here are the problems as I see with the reasoning:
A. Wisconsin's average NAEP score remained the same, but that fact used alone is misleading.
Enrollment changes during that period were severe: from 1998 to 2005, Wisconsin's low-income population rose from 1 in 4 students to 1 in 3 students, and blacks were 13 percent of all students, up from 10 percent in 1998. Latinos rose from 4 to 6 percent. White student enrollment by 2005 had declined to 77 percent, from 82 percent in 1998. According to the state Department of Public Instruction, Wisconsin has the fastest growing poverty rate of any state in the nation.
If you look at the scores for black 4th graders, they rose to 194 on the 2005 NAEP, up from 187 in 1998. Scores for Latinos rose to 208 from 201 during the same period.
Though White scores remained steady at 227-228, the shrinking pool of whites meant that the increasing numbers of black and Latino students brought down the overall average, even though these two groups made significant gains..
To put the case in its most extreme terms, a severe enough exodus of white students would inevitably mean that NAEP scores would nose dive as the population shifts, until Wisconsin had succeeded in completely erasing any achievement gap, and black and Latino scores were on par with white scores--something no state, anywhere, has done, using phonics, whole language, steroids, tea leaves . . . you name it.
B. To say that a city, with its higher concentration of poverty and minority students, performed on a rough par with the state average is not an indictment. Most cities would be quite proud to keep pace with state averages. Indeed, the reason you have a special NAEP urban assessment is because the profiles of cities are so different from states.
C. Also, Madison's efforts were part of a statewide drive to improve reading scores, so keeping pace with the rest of the state is, again, not an indictment of balanced literacy or any other single approach. To conclude that Madison's approach was unsuccessful, you'd have to compare districts across the state by their method of instruction, enrollment features and test score gains.
D. Finally, the blogger uses statistical sleight of hand when he wants to discuss the achievement gap, switching the time frame back to 1992. But the ground zero we were counting from was 1998, when Wisconsin apparently reacted to its eroding performance and started a statewide drive to improve early reading. And if you look at those figures (cited above), the gap between African-Americans, Latinos and whites shrank.
Again, thanks for your interest.
Diana Jean Schemo.
First rule of holes: stop digging.
I'll deal with this in another post.
Since I posted the email, I figured I would try and respond with what I see are its weaknesses.
Point A.
I was under the impression that NAEP to Wisconsin scores were used as a point of reference. Wouldn't both scores reflect the same demographic changes? Sine would negate the shift that she uses as an argument.
Point B.
True, but Madison doesn't keep up with the state when you seperate everything out by traditionally underperforming demographic groups. Madison blacks perform worse than Wisconsin blacks. Madison free and reduced lunch kids perform worse than Wisconsin free and reduced lunches. White kids do outperform the state as a whole.
C.
The failure of Madison to keep pace with Wisconsin may not be an indictment of Balanced Literary, but it is an indictment of whatever plan Madison chose.
D.
I suspect even with using 1998 as ground zero, that K Ed's original argument would still hold.
p.s. I forwarded you the email from her.
roro
Mark Seidenberg?
Mark Seidenberg, Ph.D.
Columbia, Ph.D.
Professor, Psychology
Publications here: Way over my head...
Hi. I'm Mark Seidenberg. I'm lurking here! I'm a reading researcher at UW Madison. The NY Times article was atrocious; web sites like this one are doing a great job at unpacking the highly misleading information that it contained.
I think the Times runs these articles when they think they reflect broader, national trends. So, the issues extend beyond Madison and they're worth going to the mat about.
Yes my articles are pretty impenetrable; sorry. They're aimed at an audience of scientists. there is one article on my web site that was a piece on reading instruction coauthored by 5 of us that might be sort of accessible.
I would like to write a little book or manual that would provide information that parents and teachers can use. Because there's a lot of misinformation out there.
I don't know why the MMSD and the New York Times decided to use the Reading First fiasco as a stick with which to beat phonics. Madison is committed to its "balanced literacy" program, but one requirement for RF funds was that they be used for ..... balanced programs that included phonics as only one element.
For some reason, some people cannot distinguish between "phonics as an essential element in early reading education' and "phonics all the way down," i.e., don't teach anything else.
I would like to be able to help out and can be contacted by email, mark.seidenberg@gmail.com. I can't guarantee I'll be able to answer all email but I'll try.
Mark S.
Post a Comment