- Poor kids often come to school far behind their middle class peers in reading ability, can this initial achievement gap be eliminated in a timely manner?
- Can this initial achievement gap in reading ability be eliminated by a school-based instructional intervention?
The study is Academic Acceleration in First Grade Reading Using the Direct Instruction Model by Michael Rebar answers both questions in the affirmative.
I alluded to this study in an earlier post on Douglass High. As I indicated in that post, in the mid 90s Baltimore decided it would mandate the use of SRA's Open Court reading program in all its public elementary schools. At the same time, The Baltimore Curriculum Project contracted with the National Institute for Direct Instruction (NIFDI) to provide training, coaching, and support for reading instruction in eleven Baltimore schools.
This set up the conditions for a nice experiment. Eleven Baltimore City schools were found with similar demographics and achievement levels to the 11 NIFDI schools to serve as controls. Thus, the study involved three groups: the 11 NIFDI schools, the 11 matched control schools, and the remaining Baltimore schools. In total, between 1998 and 2003, 41,223 kindergartners and first graders in Baltimore participated in the study.
Here is the demographic break-down for all three groups:
The above table shows the average percentage of poor students (based on free and reduced lunch participation) in the NIFDI schools, the control schools, and the remaining Baltimore schools for each year of the experiment (1998-2003). As you can see, the poverty rate in Baltimore is high (about 72%) but the poverty rate at the NIFDI and control schools was even higher -- these were some of the poorest schools in the district. These schools were also the likely feeder schools for Douglass High.
The experiment was phased in over three years -- 1997-1999. Five additional schools initially chose to implement the NIFDI model, but dropped out; they are included in the Other Baltimore group. The number of schools in the Other Baltimore group varied between 103-122 schools.
In the spring of 1997, the Peabody Picture Vocabulary Test (PPVT) was administered to kindergartners in the NIFDI and Matched Control conditions. The PPVT is a widely used norm-referenced picture identification test that is highly predictive of future reading ability. The results serve as the best estimate of initial achievement reading level (i.e., prior to intervention).
The NIFDI schools received two 30 minute periods of reading instruction per day in K and 1. The curriculum was Reading Mastery Classic. The NIFDI schools also received an additional 30 minute period of language instruction using the DI program Language for Learning and Language for Thinking which focus on oral language development.
Prior to 1998, the Matched Control and Baltimore condition schools were free to use any curriculum program desired. There was no district-wide structured reading program and schools used a variety of instructional programs. In the fall of 1998, the district adopted Open Court Reading in kindergarten through second grade.
Let me translate that into English. Prior to the experiment, Baltimore schools taught Reading in whatever manner they desired with predictably bad results. In 1997 Baltimore schools were performing at the 27th percentile on the PPVT. In 1998, all but 11 of Baltimore's schools switched to the research-validated Open Court Program. The remaining 11 schools switched to DI under the guidance of NIFDI. Thus, the experiment allows us to make three comparisons: 1. between DI and Open Court, 2. between DI and the pre-experiment reading curricula, and 3. between Open Court and the pre-experiment reading curricula.
In the spring of 1998 and 1999 the Comprehensive Test of Basic Skills, Fourth Edition (CTBS/4) (CTB,1991) was administered to all Baltimore first graders. The Comprehensive Test of Basic Skills, Fifth Edition (CTBS/5–TerraNova) was administered to all first graders in 2000 through 2003. Here are the results., including the PPVT baseline results.
The NIFDI schools went from the 15th percentile in 1998 to the 75th percentile in 2003, a significant increase in student achievement. In contrast, the Matched Controlled schools (using Open Court) went from the 14th percentile in 1998 to the 40th percentile in 2003, also a significant increase, but not as much as the NIFDI schools. The Other Baltimore schools (using Open Court) went from the 24th percentile in 1998 to the 59th percentile in 2003, also a significant gain. Here are the effect sizes for each group.
As you can see, the NFDI intervention had a very large effect size of 1.87 standard deviations. Let's put this in context. The NFDI intervention was 750% more effective than lowering class sizes to 13-17 students. In fact, since most non-instructional interventions perform about as well or worse than lowering class size, the NIFDI DI intervention was at least 750% better than those as well.
Even the Matched Control Open Court schools had a large effect size of 0.90 standard deviations or 360% more effective than lowering class size. That's not too shabby at all.
Below is a graph of the NIFDI schools showing performance gains in normal curve equivalents.
Here's the analysis from the study:
The average student performance level in the NIFDI schools moved from the lowest quintile to well above the national mean. All NIFDI schools showed substantial achievement gains and several became beacon schools for the district. The three highest achieving schools in the district in 2003 were NIFDI schools. The Matched Control schools never achieved a mean score above the national average.
In all, 49 of the 103 Baltimore condition schools (48%) achieved a mean Total Reading NCE score of 55 or greater in the 2003 school year. In the NIFDI condition, ten of eleven schools (or 91% of these schools) reached this level. Only one of the eleven Matched Control condition schools reached an NCE score of 55; this school ranked 49th in the Baltimore condition.
The Fisher’s Exact Test indicates that schools in the NIFDI condition were significantly more likely to exceed the national mean than either the Baltimore or Matched Condition schools. The three highest achieving elementary schools in the district were NIFDI schools, achieving mean Total Reading NCE scores of 92, 90, and 83. The highest three schools in the Baltimore condition achieved scores of 78, 76, and 75. It is perhaps ironic that the highest achieving Baltimore school (with a mean NCE score of 78) is one of the DI schools that did not continue as a NIFDI school but continued to use the Direct Instruction reading program.
There you go. The top four schools in Baltimore were all schools using DI. Of course, the Baltimore schools, despite a long history of academic failure, still though they knew the best way to teach:
Despite the gains demonstrated with the NIFDI model in Baltimore in 1997 through 2003, the district did not systematically expand the model to other schools. In fact, during the course of this study the district mandated procedures that were at odds with model provisions. Teachers were required to attend district inservice trainings that advised them to do things differently than what the NIFDI model specifies. The central administration required schools to provide daily test-preparation periods for the entire school year. NIFDI advised the principals in their model not to do this because it believed greater gains would be possible by implementing the model rather than providing test preparation (which would also tend to artificially inflate achievement scores). In fact, the six principals who implemented the model most faithfully and ignored some of the district mandates achieved better performance than any of those who complied with district regulations.
So what do you think Baltimore did when faced with these improved scores?
That's right, they stopped the program and went back to what they were doing before the intervention. Unfortunately, I've been unable to find CTBS scores from 2004-2008.
So the answer to our initial questions is: Yes, we can eliminate the achievement gap between low-SES students and their middle-class peers in reading by the end of first grade by means of a school-based instructional reform. In fact, we can get low-SES kids to significantly outperform their middle-class peers. These achievement gains can be accomplished without any SES-based reforms or by reducing any of the symptoms associated with growing up in a low-SES environment.
So, how do all you Broader, Bolder types explain these results?