August 7, 2008

Real Education: A Call for an Educational Experiment

I'm reading Charles Murray's latest book, Real Education: Four Simple Truths for Bringing America's Schools Back to Reality.

I agree with some of the points Murray makes and I disagree with others.

In any event Murray proposes a very good idea in Chapter 5:

Hence my second proposal, for a study that would be the most expensive educational demonstration project in history and would take as much as fifteen or twenty years from beginning to end. I state in the form of a challenge to everyone who is convinced that we can tach low-ability children far more than we are currently teaching them: Put up or shut up... Here is the proposal:

select children who test low in accademic ability but are not clinically retarded--say, children with measured IQs from 80 to 95, which demarcate the 10th to 37th percentiles. Make the number of the children in the study large enough that the results cannot be explained away as an accidents of small samples. Then provide these children with the best elementary education that anyone knows how to provide. Build new facilities or renovate existing ones. Hire the best teachers and create model curriculum. Measure how well the children are doing at the end of elementary school, and compare their progress with that of other children matched for IQ, family background, and whatever other variables are considered important.


The people who conduct the experiment should be free to use any teaching techniques, any class sizes, any amount of one-on-one tutoring, and type of technological aid. They shouldn't worry about making the program financially affordable for wider application, but instead bring to bear every resource that anyone can think of, at whatever cost that will maximize the education that these children acquire. Or to put it another way, their mission is to conduct the experiment in such a way, if it fails to produce success, there will be no excuses. Only three ground rules are nonnegotiable:

  • The organization that selects the experimental and control samples and tests the children must be completely independent of and isolated from the organization that conducts the experiment.
  • The design must protect against teaching to the test and test-practice effects.
  • The design must include a test for fadeout, conducted three years after the experimental education ends.

Great idea. Sound familiar?

That's what I thought too. So I dashed off an email to Murray informing him that we'd already done something very similar thirty years ago: Project Follow Through.

Murray wrote back that he thought something was out there (even though people kept telling him there wasn't) and hoped that Real Education would surface it. Sure enough it had and I gave him a crash course on PFT.

In the post I'll tell you what Murray predicted would be the results of this grand experiment and we'll see how well his predictions matched the results of PFT.


Anonymous said...

When I started teaching I learned about Project Follow Through via blogs and web sites critical of the current state of primary education. In other words, I found it because I sought out voices that I wasn't hearing in the ed schools and teacher's lounge.

I am quite astonished that Mr. Murray was unaware of PFT. Is it that far underground?

ZZMike said...

We've been doing "educational experiments" ever since the 40s - and maybe before. Read Diane Ravitch's Left Back

The problem is that those experiments weren't as controlled as Murray's seem to be.

I'd really like to hear someone from the NEA talk about this book. It would be "most amusing".

Brett Pawlowski said...

former nyc math teacher,

in my limited experience, it is pretty far underground. I just held a conference where one of our keynotes talked about PFT; one of the attendees - a former superintendent in Iowa - told me he had never heard of it until that day, and was amazed at the results. Ironically, he thought our speaker was "ahead of his time" by quoting research that happened decades ago.

Anonymous said...

Good one, Ken. I think I can dig up the figures from Wes Becker's analysis of low-IQ kids who got DI in PFT. As I recall, those figures show that in some academic areas kids with IQs ~80 were making one year of progress for each year of instruction.

Anonymous said...

Bonnie Grossen gets kudos for informing the world of the web about Glass's PFT critique, but now ERIC has the abstract and full text. This is explains quite a bit... Gene Glass and Gregory Camilli "Follow Through" Evaluation:
"NIE should conduct evaluation emphasizing an ethnographic, principally descriptive case- study approach to enable informed choice by those involved in the program. ... (2) The deficiencies of quantitative, experimental evaluation approaches are so thorough and irreparable as to disqualify their use; ... (4) The audience for FT evaluations is an audience of teachers to whom appeals to the need for accountability for public funds or the rationality of science are largely irrelevant"

Homework: enumerate all the ways Glass's critiques strikes at the core of American democracy. My favorite: if "deficiencies ... so thorough and irreparable as to disqualify their use" exist, what evidence do we need to strike from 20+ years of litigation in Abbott v Burke? Knowing what evidence will "stand up in court" ought to concern anyone who values the rule of law--even ed school faculty.

Redkudu said...

NYC math-

I found out about Project Follow Through much like you did - but not until I'd been teaching for 8 years. It was a happy coincidence of education blog digging that finally led me to it. I can also tell you that not one colleague I've ever spoken to has ever heard of it. That's English teachers mostly (you'd think they'd need to be aware of it) and a few special education teachers as well.

KDeRosa said...

John (and anyone else interested), ERIC now has available Becker and Engelmann's analysis of the PFT:

Analysis of Achievement Data on Six Cohorts of Low-Income Children From 20 School Districts in the University of Oregon Direct Instruction Follow Through Model

palisadesk said...

A must-read (wonder if Ken has this one?) on Project Follow Through is this slim but potent 1997 volume by Cathy Watkins, Project Follow Through: A case study of contingencies influencing instructional practices of the educational establishment (see here):

Watkins book

It contains an outstanding summary together with a cogent analysis of the interlocking factors that prevented PFT from influencing mainstream practices -- the most thorough analysis I have ever seen, and as true today as ten years ago. She has further suggestions for how to bring about change but these are more tentative. So far nothing has been very effective;-(

A short article by Watkins is available here (but the book is a sine qua non):
Project Follow Through: Why Didn't We?

In fact there are a number of interesting articles on PFT here:
Effective School Practices

Murray should read the book. He is a scientist and does appreciate data.

KDeRosa said...

I have not read the Watkins book, but I did forward the Effective School Practices articles to Murray.

I'll put the Watkins book on my To Buy list.

Brett Pawlowski said...

Also along these lines, I'd suggest looking at "Annual Growth, Catch-up Growth", detailing (successful!) efforts in Kennewick WA to bring 90% of third graders up to grade level.

Can't recommend it highly enough.

Anonymous said...

Like others mention, PFT is pretty new to me, and I have been in and out of education since the 1960’s.

I looked in the index of “Left Back” and find neither “follow through”, nor “project follow through” listed. I looked in “The Academic Achievement Challenge” by Jeanne Chall, and she devotes two pages to it, page 80-81.

Murray also wrote “Losing Ground”, which I read some years ago. As I recall there was a lot of good food for thought there. I see from Amazon that he has several other books that I suspect we would all profit from reading.

I will give first priority in the weeks and months ahead to finding everything that’s written on PFT, starting with links from this blog. If anyone has a list of available sources of information on PFT, and advice on prioritizing those sources, that would be most valuable.

TangoMan said...

Wasn't one of the conclusions of PFT:

-"No type of model was notably more successful than the others in raising scores on cognitive conceptual skills”

It would seem to me that CM and KDR should first come to an understanding of exactly what study design they are considering. The CM condition that caught my eye was "The design must include a test for fadeout, conducted three years after the experimental education ends."

I'm not as intimately familiar with DI as many here, but IIRC, this condition wasn't a feature of PFT. Let me be more specific. Studies which look at effects 3 years after a program ends, such as Englemann and Beckers, where they look at the kids as they enter 6th grade, will not capture the same dynamic as studies that try to measure the educational effect received in 9th grade and whether the effects still hold in 12th grade, or effects received in 12th grade and measured 3 years out of school. The factor that changes between the 3rd-6th, 9th-12th, and 12th - graduation +3 years, is the age of the student and their physiological/neurological

What I think CM is alluding to is the fact that heritability of intelligence increases as one grows older, so the academic gains one earns through good curriculum and beneficial learning environment erode over time as it becomes more difficult to equalize environmental factors for all children as the children grow older.

So, if CM and KDR agree that a basic educational curriculum through the 12th grade can be mastered by anyone with an IQ greater than, let's say, 90, then curriculum design will matter a great deal. The one unresolved point of difference likely hinges on how we define a basic curriculum.

So, here's a direct question: Did PFT follow students right from 1st grade to 12th grade and then evaluate students 3 years later. This is what I think CM might have in mind.

Count me as skeptical that DI, as effective as it is, can take a child with an IQ of 85 or 90 and with 12 years of influence get that child to pass a calculus or physics class. I would argue that such classes go beyond a basic curriculum in that they require conceptual and analytic skills that aren't present in those with an 85 IQ. Similarly there is a reason why we don't teach topics which rely on abstract thought to 6 year olds. I'm wholly on board with the notion that DI can graduate a citizen with the basic skills to function in society, such as an ability to read, to conduct math that is necessary for everyday living, etc but I put some pretty tight boundaries on what I define as basic skills. That said, it's a travesty that the present educational zeitgeist can't even deliver such basic skills to many of the students who warm the seats in the classrooms of the nation.

Anonymous said...

It is disconcerting that a first-rate public intellectual like Charles Murray could write and publish an education reform book without ever hearing of Project Follow Through. That's like writing a history of amphibious warfare and first hearing after publication that there was this underground military organization (even though people kept telling him there wasn't) called the US Marine Corps.

I first learned about PFT (and DI) last year in Ian Ayers's book Super Crunchers.

KDeRosa said...

Tangoman, in PFT DI had a higher average in significant outcomes in cognitive skills (354) than in basic skills (297). No other model in PFT had a positive outcome for cognitive skills.

The official statement issued by NIE was that PFT in the aggregate had failed to significantly raise the achievement of students above extant title I programs. That's because only DI was the only model to achieve consistent positive results. Most of the other models produced no or negative results.

I'll deal with the fadeout issue in another post since it is a significant issue. Some of the PFT kids from DI were later evaluated in sixth and eighth grades and there was fadeout.

In fact, you bring up a lot of good points, but right that I'll deal with next week. But right now I'm off to the beach.

Anonymous said...

... voices that I wasn't hearing in the ed schools and teacher's lounge ...

My initial reaction (when this comment was fresh) was "Oh, you too."

Upon reflection, why haven't your state's civil rights activists asked your state attorney general, auditor, governor, or chacellor to address possible negligent credentialing of ed schools?