October 11, 2010

Replication, but no Verification

Geoffrey Canada's Harlem Children's Zone is going to be replicated in 21 communities with the help of  $200 million in federal dollars.

The philosophy of the HCZ is simple:  it takes more than a good school to educate poor black kids.  To this end the HCZ steps in loco parenti to offer a panoply of the kind of social services many people believe to have an effect on education outcomes,such as:

  • early childhood programs with parenting classes;
  • academic advisors and afterschool programs for students attending regular public schools;
  • a support system for former HCZ students who have enrolled in college;
  • a fitness program;
  • asthma management;
  • a nutrition program;
  • organizing tenant associations;
  • one-on-one counseling to families;
  • foster care prevention programs;
  • community centers;
  • an employment and technology center that teaches job-related skills to teens and adults.

In a nutshell, the HCZ is an implementation of the Broader Bolder Approach to Education.  Basically, they believe that the link between poverty and student achievement is more than merely correlation; they believe it's causal and fundamental:

More than a half century of research has documented a powerful association between social and economic disadvantage and low student achievement. Weakening that association is the fundamental challenge facing America's education policymakers.
Evidence demonstrates, however, that achievement gaps based on socioeconomic status are present before children even begin formal schooling. Despite impressive academic gains registered by some schools serving disadvantaged students, there is no evidence that school improvement strategies by themselves can substantially, consistently, and sustainably close these gaps.

Bold words from a bold movement.  Let's see if those words survive some scrutiny.

Roland Fryer first looked at the HCA and found:

  • Students attending the HCZ charter school outperformed students who lost the lottery to attend the HCZ charter. That's promising, at least for the school part of the reform.
  • HCZ students who only attended the charter school performed as well as the children who attended the charter school who received the full panoply of social services. 

One could readily conclude from the Fryer study that the HCZ charter school does a good job in raising student achievement, but that the social component of the HCZ isn't having much effect.

Oddly, but not surprisingly, Fryer's conclusion was more ambiguous:

We conclude . . . that high-quality schools or high-quality schools coupled with community investments generate the achievement gains. Community investments alone cannot explain the results.

And a $200 million federal funding stream was born.

Brookings also looked at the HCZ data and found that the HCZ charter school performs about as well as the average charter school in New York City.

None of the charters that perform better than HCZ provide "or depend on community and social services to achieve their academic mission." Brookings concludes:

These findings create a large question mark for the theory of action of the HCZ.  If other charter schools generate outcomes that are superior to those of the HCZ and those charter schools are not embedded in broad neighborhood improvement programs, why should we think that a neighborhood approach is superior to a schools-only approach?  
There is no compelling evidence that investments in parenting classes, health services, nutritional programs, and community improvement in general have appreciable effects on student achievement in schools in the U.S.  Indeed there is considerable evidence in addition to the results from the present study that questions the return on such investments for academic achievement.  For example, the Moving to Opportunity study, a large scale randomized trial that compared the school outcomes of students from poor families who did or did not receive a voucher to move to a better neighborhood, found no impact of better neighborhoods on student academic achievement.[x]  The Nurse-Family Partnership, a highly regarded program in which experienced nurses visit low-income expectant mothers during their first pregnancy and the first two years of their children’s lives to teach parenting and life skills, does not have an impact on children’s reading and mathematics test scores.[xi] Head Start, the federal early childhood program, differs from other preschool programs in its inclusion of health, nutrition, and family supports.  Children from families enrolled in Head Start do no better academically in early elementary school than similar children whose parents enroll them in preschool programs that do not include these broader services.[xii] Even Start, a federal program that combines early childhood education with educational services for parents on the theory that better educated parents produce better educated kids, generates no measureable impact on the academic achievement of children.[xiii] 

The HCZ, and programs like it, are based on the alluring but mistaken hypothesis that you can get middle-class academic performance from "poor" kids by giving them the things that middle-class families have that the Broader Bolder people think are the cause of middle-class student achievement -- things like what the HCZ is providing: health care, prenatal care, better nutrition, better parenting skills, community centers and the like.  But these things are merely the status markers of the middle class.  They are merely markers for the traits the middle class possess, like self-discipline, the ability to defer gratification, cognitive ability that let you enter and stay in the middle class.  Subsidizing the markers doesn't increase student achievement because the markers don't produce (or cause) student achievement; the underlying traits do.  If anything, subsidizing the the markers serves to undermine the traits.

We see this most clearly with respect to the generous benefits provided to the poor in our welfare state.  we've alleviated many of the hardships associated with poverty, but the alleviation of those hardships isn't turning the poor into the middle-class and the poor still don't act like the middle class even though they are far more wealthy than the poor of 100 years ago.


Stephen Downes said...

Replication *IS* verification.

If there's nothing to Canada's approach, and if the BBB approach is all smoke and mirrors, this trial in 21 communities will tell us. And it will cost only $200 million (which is a small fraction of the overall education budget).

Far better to run a trial than to simply listen to the relentlessly negative aspersions cast by a patently unqualified education blogger.

KDeRosa said...

Replication *IS* verification.

Only when a properly constructed experimental design is in place and there is no indication that that will be done with the grants.

I agree that that's what should be done, but it isn't.

Far better to run a trial than to simply listen to the relentlessly negative aspersions cast by a patently unqualified education blogger.

I just report the data; it's not my fault the data doesn't support your worldview..

Jared Stein said...

I've read Dobbie and Fryer's HCZ report, and, despite its authors' several conjectural qualifiers, found the same information and arrived at the same conclusions. So I too was confused that the whole package is being replicated when the school treatment--independent from the community treatment--seems to be the trigger for improved student achievement. I certainly understand the reasoning for the inclusive language used in the analysis--the researchers would clearly like to see the whole thing implemented across the board because it agrees with their politics. But, politics and well-meaning aside, the data doesn't support it, and we do best to replicate the parts of treatments that are shown to be effective, not the parts that confirm the null hypothesis.

Having said that, I'll admit I was tempted to simply not read or misunderstand the point of this post in order to be the first to insult you in the comments. Maybe next time!

Dick Schutz said...

Follow the money. If you look at the abstracts of the grants, there is no "replication" other than in the rhetoric. HCZ is highly subsidized and highly publicized.

The grants are puny and partial--most are foreplay with the expectation that much larger federal funding will be forthcoming. The chances of that happening are slim to none.

The HCZ logic is analogous to the belief that not only can "education" meet children's "instructional needs, it can and must meet their health needs and their "social needs" The net result is overlapping bureaucracies with no personal responsibility--as Ken explains in some detail. Instruction remains a black box, which for some kids is a black hole.

Sell short on this one.

Roger Sweeny said...


The fact that this will only cost $200 million dollars means that if it doesn't succeed, its proponents will say, "The trials prove nothing. We didn't spend nearly enough to make a difference."

It's Model Cities all over again.