Today's Chart

Update: more visual aids here.

Arnold Kling suggests another way to present education data to determine if funding matters in education:

On the X-axis, plot the percentage of students in a county who are above the FARMS line (that is the "free and reduced meals" indicator of poverty). On the Y-axis, plot the percentage of students that pass the math exam. For each county in Maryland, put a data point on the chart. Next to each data point, put the County's ranking in terms of expenditure per pupil.

Next, draw the line of best fit through the data points. Counties that fall above the line are adding relatively more value than counties below the line. If education spending matters, then Montgomery County and other high-spending schools should be above the line. It would be interesting to see whether this is in fact the case.

Here is the chart for 497 school districts in Pennsylvania for 2005. On the X axis I have the percentage of students in the district that do not receive free and reduced meals. On the Y-axis I have the pass rate for PA's PSSA 11th grade exam (math and reading). There were too many school districts to add the funding data but I did do the calculations.



For school districts falling below the regression line the average total expenditures was $11,417.

For school districts falling above the regression line the average total expenditures was $11,214.

The overperforming schools actually spent less on average. Go figure.

Back in March I ran few different regressions on expenditures and FARM perfromance, household incomes, teacher salaries and parental education. The results are not always what you'd expect.

Update: Brett from DeHavilland blog has the numbers from Tennessee. Brett writes, "After looking at the correlation between TCAP and poverty rates, we looked at correlation between free/reduced lunch rates and value-added performance of the schools. Virtually no correlation to be found: in other words, some schools with 100% free/reduced lunch rates are contributing tremendously to student learning, and some with almost no free/reduced lunch participants are dropping the ball." Notice the variance (R²) is virtually the same as what I calculated for PA. (Note: Brett 's graph shows the percentage of FARM students not non-FARM students.)



Update II: Unbroken window runs the data for New York and finds the same relationship. Although, it appears that the some schools are maxing out the test and distorting the data.

Inquiry Physics

I was looking for something in my storeroom earlier today and stumbled upon my old College Physics textbook. Seven hundred pages of pure applied brutality that every science and engineering student had to complete.

Physics I was the course that sent a large portion of my freshman college class for greener pastures over at the business school. This happened despite the fact that almost every student had taken a high school physics course, so this was the second time through this material.

If my recollection serves me correctly, Physics instruction was supposed to go something like this. The student was supposed to read one or more sections of the textbook every week and attend a lecture given by the professor elucidating the sections we were to have read. A few dozen problems from those sections were assigned to us to work out. Then we attended three hours of recitation classes given by graduate students who worked through some of the problems we had been assigned to make sure we understood what was going on.

This brutal pace kept up for fourteen weeks and we covered nearly the entire textbook. During that time, we worked through hundreds and hundreds of problems. We were permitted to take into each exam one sheet of paper with whatever we could fit thereon. Otherwise, the exams were closed book. Nonetheless, the average grade for each exam was almost always less than 50%.

Here's my question.

How could one possibly teach an inquiry-based (problem-based learning) Physics course and possibly hope to get through more than say a quarter of the syllabus of a lecture-based course? I don't even see how this might work for a high school level course.

Update: Based on Stephen Downes' comment, I sense some confusion. I consider this to be a direct instruction/lecture based course not an inquiry based course. The pace is brutal for a lecture based course. I can't imagine covering this amount of content in a true inquiry based course.

Snake Oil is still Snake Oil even when its Broader and Bolder

I've been perusing the various Background Papers for the Broader, Bolder Initiative looking for some valid research pertaining to an actual implementation of one of the Broader, Bolder ideas. What I didn't find was:

Nevertheless, there is solid evidence that policies aimed directly at education-related social and economic disadvantages can improve school performance and student achievement. The persistent failure of policy makers to act on that evidence—in tandem with a school-improvement agenda—is a major reason why the association between social and economic disadvantage and low student achievement remains so strong.

What I find is a lot of observational studies that find various correlations between traits associated with at-risk children and their families and the fact that these children tend to perform worse than their mainstream peers. The causal jump is then assumed.

For example, studies have shown that at-risk kids report in questionnaires that they experience more hunger, which might be broadly defined to include everything from extreme malnutrition to missing a snack once a week, than their middle-class peers. Since the performance of at-risk kids is less than their mainstream peers, Broader, Bolder reasons that hunger causes distraction and distraction causes lower performance. And, therefore, we should provide more nutrition to at-risk kids.

But since there's no such thing as the nutrition fairy, this broader, bolder plan has to implemented somehow. For example, we might fund the public schools so that they might provide free and reduced lunches to qualifying at-risk students. Actually, we do that already. Then, how about if we fund breakfast programs for qualifying at-risk students. We do that too. I'm confused.

You see, the question isn't whether we should be providing more nutrition to at-risk students. That low-hanging fruit has already been picked. The question now is whether we should expand or supplement these existing programs and are educationally significant gains in student achievement to be forthcoming.

This is the question that should have been answered before Broader, Bolder issued their manifesto. But it wasn't. At best, we have some small scale research, usually rife with methodological flaws, that was conducted so that the researcher could provide evidence that their belief was correct. You can always find evidence that some kids learned something when you changed some condition. Round up enough experimental subjects and conduct enough experiments and you are bound to find some statistically significant, though not necessarily educationally significant, increase in performance whether by chance, good fortune, or hoax.

What we want is research in which the researcher started withe assumption that their idea was wrong. Then the researcher collects evidence that shows either that the researcher's beliefs are false or not false. This is called testing the null hypothesis. Kozloff gives us an example of this type of research:

I believe program X works, but I'm going to assume that it doesn't work and I'm going to collect data to try to show that it doesn't work. If the data do not show that X does not work, I will conclude that maybe it does work. Maybe.

We don't see this kind of research cited by Broader, Bolder. What we see is "research" that is attempting to persuade us that we should join the researcher in accepting his beliefs and that the researcher is not very interested in the possibility that he is wrong.

Broader, Bolder needs a healthy dosage of humility, especially since so many of its bromides remain untested.

(Picture adapted from Telling the Difference Between Baloney and Serious Claims About What Works, Kozloff and Madigan, DI News, Summer 2007)

Today's Best Education Paragraph

From Jay Greene:

Besides neither being unfunded nor a mandate, the argument that NCLB is an unfunded mandate is especially odd because it makes one wonder what all of the funding that schools received before NCLB was for. It’s as if the unfunded mandate crowd is saying: “The $10,000 per pupil we already get just pays for warehousing. If you actually want us to educate kids, that’ll cost ya extra.” Remember, that NCLB just asks states to establish and meet their own goals. Didn’t they have goals before NCLB?

Oh, we were supposed to educate them as well with that money?

Bogus Bowl V

Go take Teach Effectively's latest Bogus Bowl poll.

Which of the following do you consider to be the most bogus reason for failing to teach prospective teachers how to employ teaching procedures that have been documented to be effective?

• Professors want future teachers to find their own teaching styles.
• Professors don't want to stifle future teachers' creativity.
• Professors say that using research-based practice is only one small part of what future teachers need to know.
• Professors believe that there is not one best way to teach.

Hoffman and the Rule of Holes

Tom Hoffman is having a grand ol' time digging himself deeper into a rhetorical hole.

Let's start with the argument Hoffman apparently considers to be the knock-out blow.

DeRosa's critique of the assignment is based on how he imagines the Dred Scott decision ought to be taught in a US History class. Had he asked before writing his missive, or bothered to read the History and Social Studies section of the Curriculum Guide, he would have known that his entire frame for critiquing the assignment was incorrect, because this was not an assignment for a US History class (taken in 11th grade at SLA), but an African-American History class.

I'm not sure why Tom thinks I didn't know that the project was for an African-American history class considering this statement from my initial post.

The project comes from page 10 of the Family Handbook and pertains to African-American History.

I've taken the liberty of highlighting the relevant portion for Tom's benefit. Apparently, my "entire frame of critiquing" wasn't correct after all.

In this context, what is important is the decision's impact on African-Americans and the abolitionist movement, not the balance of power in the great game between the North and the South in which the African-Americans are seen as mere pawns. Perhaps in 11th grade US History, the pre-war balance of power dynamic will be emphasized.

I agree. That's why I gave the following as an example of analysis showing deep understanding for a high school student.

[T]he Dred Scott decision is important because it upset the political compromise at the federal level (the Missouri Compromise and the Kansas-Nebraska Act) which served to limit the spread of slavery.

Again I've emphasized the relevant portion for Tom's benefit. The balance of power issue remains an important issue for African-Americans history since it affected the growth of slavery. I'm thinking the growth of slavery might have "impacted" African-Americans.

I provided yet another reason for why the balance of power issue was important to the slavery issue:

As long as the Senate was gridlocked, the North would not be able to pass a constitutional amendment banning slavery in all the states.

Again, I've scaffolded the passage for Tom's benefit. I'm thinking the balance of power issue was a little more important than merely a "great game between the North and the South in which the African-Americans are seen as mere pawns."

Tom continues:

I would note that Chris Lehmann told me that he left a comment on Ken's blog explaining this oversight on Ken's part, but for whatever reason, that comment has not been published as of this date.

Unlike Tom, I don't moderate comments and I only delete spam comments. If Chris' comment didn't post it's either because Chris did something wrong or blogger ate the comment. Most likely it was the latter.

I'm thinking at this point Chris is glad the comment didn't make it through.

The knock-out punch missed its mark. Let's see if Tom's remaining argument lands.

Beyond Ken's unhappiness of the framing of the decision and the assignment, his criticism of the student work itself is not based on any knowledge of the kind of work 14 year olds typically do. As a piece of writing, the letter in question would stand up admirably against the anchor papers used in any 9th grade writing assessment in the country, if not the world. DeRosa never questions the accuracy of the student's historical information.

Actually, I did question the efficacy of petitioning Southern Democrats for redress. That seemed to be on the wrong side of a few historical facts. However. the primary deficiency in the student project was that the student didn't give us much to work with, hence my characterization of the project as "superficial understanding." This contradicted the claims made by SLA in the family handbook, and I quote:

Teachers in each course ask the question – “What are the enduring understandings students should have when they leave this class?” Teachers then create projects that can only be completed by showing both the skills and knowledge that are deemed to be critical to master the subject and demonstrate that deep level of understanding.

...

At SLA, there may be multiple assessments – including quizzes and tests – along the way, but the primary assessment of student learning is through their projects.

This was supposed to be an exemplary project, yet it contradicted SLA's assertion that it showed deep understanding of the subject matter and mastery of skills. I argued that it showed superficial understanding because it "fail[ed] to cover any of the important issues presented by the decision, the historical context of the case, and why the case is historically important."

Tom's point with respect to what a 14 year old should be expected to know is relevant. Tom cites an AP prep book for U.S. History that provides far more detail on the Dred Scott decision (p. 132)and the relevant history leading up to that decision that my analysis. I also cited a middle school history text which gave about the same level of analysis that I provided. What is clear is that the student example project provided far less analysis than either the middle school analysis, my analysis, or the AP prep book analysis.

That's two misses.

My advice to Tom: stop digging.

Still Waiting on Broader, Bolder

I'm still waiting on someone fom Broader, Bolder to offer some evidence supporting the effectiveness of their call to expand public education to cover a myriad of social services.

This week both Diane Ravitch and Randi Weingarten offer tepid defenses over at The Education Gadfly.

Let's take Ravitch's defense first:

I care as much about academic achievement as Checker or anyone else in the world, but I don't see any contradiction between caring about academic achievement and caring about children's health and well-being.

The issue isn't about who cares about children's health and well-being. The issue is whether public schools, who are by and large failing at their primary task of education, should take on the additional responsibilities of caring about children's health and well-being. You could care very much about the health and well-being of children and NOT think it's a good idea to hand these services over to our public schools.

The argument seems to be that since children attend school every day (cough, cough) that social services could be easily provided at school. Then why not hand over these responsibilities to the post office. After all, they make house calls six days a week regardless of the rain, snow, heat, or gloom of night. They could give the kids a quick vision screen and drop off any drug prescriptions.

Will it help or harm children's academic achievement--most especially children who are living in poverty--if they have access to good pre-K programs?

The extant evidence suggests that pre-k programs will have no or a negligible effect on academic performance. The yoke is on you to show that there will not only be an educationally significant effect on academic achievement but also that the benefit will persist when the public schools do the provisioning.

Will it help or harm children's academic achievement--most especially the neediest children--if they have access to good medical care, with dental treatment, vision screening, and the like? Will it help or harm children's academic achievement--the children whose lives are blighted by the burdens of poverty--to have access to high-quality after-school programs?

The evidence here is even more scant. I know these community service schools exist, so where is the evidence that they are raising academic achievement. Show me the money.

So, I explain my dissent briefly: One, what we are doing now--the standards & assessments & accountability strategy alone--bears little or no resemblance to genuine academic excellence.

But this doesn't mean that the Broader, Bolder way will be any improvement.

And two, children who come to school hungry and ill cannot learn no matter how often they are tested.

Last I checked, schools offer free and reduced price lunches to practically the entire left side of the socio-economic curve. If these kids are still hungry what makes you think that expanding these current programs will solve the problem, to the extent that there even is one. Again, where is the data?

And three, a good education must include attention not only to academics but to children's character, civic development, physical education, and physical health.

Schools are attempting to do most of this stuff already. Where is the evidence that it is working? Where is the evidence that providing more will bring about improvement?

All we seem to have is rhetoric. Show me the data.

Let's move on to Weingarten.

I'm stating the obvious when I say that No Child Left Behind's testing regime has left little time for these kinds of in-class activities.

By "the obvious," Weingarten appears to mean "with little evidentiary support."

What evidence we go have suggests that only about 16% of schools have reduced art and music time at all. And those that did reduce time in these areas only reduced time by less than an hour a week. Perhaps these schools were neglecting math and reading pre-NCLB. Do you know? No, you don't.

But teachers alone can't get kids all the way to proficiency, when disadvantaged children typically enter school already three years and 30 million words behind.

I hate throwing this word around, but this statement is a lie. The big lie, so to speak. Schools can substantially reduce all of the achievement gap that exists between low-SES and middle-class schools. We've known this for over thirty years now. The experiment has been replicated many times, most recently in Baltimore. Up to the fifth grade level as well.

[M]y message was twofold: first, let's put in place a federal education program that, unlike NCLB, provides space and opportunity for children to be taught a rich, well-rounded curriculum, with standards and accountability that support rather than undermine that curriculum; and second, let's--at the same time--try to address the outside factors like nutrition and health care that affect a child's ability to reach her full educational potential. And yes, I said that we also should try to help parents so they can better support their children's learning.

In Project Follow Through, comprehensive medical, dental, nutritional, and social services were provided to all of the thousands of students taking part in the experiment so these factors would not confound the results. Most of the interventions failed to achieve any student gains at all despite the provisioning of all these services. Many interventions performed below the performance of the control groups. Ooops.

The causal link has not been established. Repeating the rhetoric ad nauseum is not a substitute for data.

Ravitch knows better. Weingarten probably does as well, but there is self-interest at play.

I'll ask one more time. Show us the data. We're waiting.

The Myth of Fun and Interesting

Most educators have brought the myth that academic learning does not require discipline--that the best learning is easy and fun. They do not realize that it is fluent performance that is fun. The process of learning, of changing performance, is most often stressful and painful.

--Lindsey, O., (1992). Why aren't effective teaching tools widely adopted? Journal of Applied Behavioral Analysis, 22-26.

So begins chapter four of Vicki Snider's book Myths and Misconceptions about Teaching, What really happens in the classroom on the Myth of Fun and Interesting. This chapter is relevant to the ongoing debate over the educational value of the SLA student's project on the Dred Scott decision and problem-based education in general.

Snider describes the dangers of this myth well.

The myth of fun an interesting is an extension of the myth of process. When the process [of education] is emphasized, the entertainment value of a lesson and the students' level of engagement becomes the measure of the successful lesson. Teachers derive their reward and sense of satisfaction from creating fun and interesting lessons rather than from attaining specified learning outcomes.

...

There is some truth to the myth that learning should be fun. I can understand why teachers feel compelled to make lessons entertaining. I, too, like to design activities that engage students, create excitement, stimulate discussion, and make students laugh. I also know that these reinforcing moments do not necessarily guarantee that students have mastered the content. The exhilaration of my great lesson is more that offset by the letdown when I assess retention and application of skills and concepts.

Snider goes on to describe the potential harms that are done by the myth:

There are four harmful effects that result from overreliance on fun and interesting activities. First, fun activities lead to a lot of wasted instructional time. Second, activity-based instruction can make it difficult for learners to focus on what it is they are supposed to learn. Knowing what to pay attention to is called selective attention in the psychological literature and it is often a problem for young or naive learners or those with learning disabilities. Third, rather than increase motivation to learn, activities with a high entertainment value but a low content value may actually decrease the probability that a child will become a lifelong learner. Fourth, without effort and practice, individuals cannot master any intellectual or creative endeavor.

Snider makes an often overlooked, yet important point about lifetime learners:

People seldom decide to pursue a new intellectual area out of the blue; they become interested because they find themselves in a situation that reactivates some general knowledge that a teacher thought important years ago. If they have enough general knowledge, they can find out more through experience, by going to the library or looking on the Internet, or taking a class. The more specific knowledge they acquire, the more they are able to learn. Sometimes this positive reciprocal learning cycle leads to a depth of knowledge that allows a person to think critically and analytically. In other words, interest is the reward of learning, not the motivation for learning.

Lastly, Snider on the importance of developing fluency.

Fluency goes beyond accuracy. It is accuracy plus speed. Our traditional reliance on accuracy only, in the form of percentages, does not distinguish between students who have learned a skill, but still perform it with hesitation, and those that are fluent. failure to make these distinctions underlies many educational failures (Binder, Haughton, & Bateman, 2002). Students "progress by building one non-fluent skill on top of another until the whole skill set becomes too difficult to be enjoyable and students may respond to this stressful learning situation by becoming inattentive, misbehaving, or failing to complete homework and other assignments. All of these consequences are predictable. If the teacher responds by making learning more fun and interesting or by reducing accountability, the problem is solved in the short term, but the real issue remains unaddressed and the long-term result is low academic achievement.

Snider raises some important issues with PBL. Making school work fun and interesting is all well and good, but ultimately you have to look at what the student has actually learned to gauge the effectiveness of the teaching. The arguments I'm reading so far favoring PBL and SLA rely on some form of goal-post shifting with respect to what the student is expected to have learned. This isn't exactly a strong argument in favor or PBL and SLA if you know what I mean.

Teaching Content -- the Dred Scott Decision

We all can agree that history instruction should not be a parade of facts; however, facts will need to be learned, at least temporarily, to give the student something to think about. Here's one way to make that process less painful and boring for the student.

To facilitate learning, most of the instruction should relate to big ideas. Big ideas are the ideas that summarize content, are applicable again and again, can generate predictions and hypothesis, can be used to help structure the details of content, and help in remembering and reconstructing through inference content details. Most of the instruction should revolve around the big ideas. (Crawford 2004)

Students should be able to fluently articulate the knowledge they learn so that the knowledge can be used in higher-order activities, such as essays, comparisons, application, synthesis, evaluation, and the like. (Crawford 2004)

Here's an example of a big idea from a middle school history textbook that precedes the teaching of the Dred Scott decision.

The sectional disputes about slavery that began during the Constitutional Convention ended with two compromises. One compromise was that the Constitution said that slaves were to be counted for both representation in Congress and as property, but at three-fifths value. The other compromise kept Congress from stopping the slave trade until 1808, but allowed Congress Control over other aspects of trade. The northern and southern states had equal numbers of representatives in the Constitutional Convention and chose to reach a compromise. When each side is evenly matched, there is a balance of power. An example of balance of power is when two teams have the same number of players and the players have the same skill level. If one team gets a chance to put one extra-good player into the game, then the balance of power is upset and that team will probably win. That team will dominate.

University of Oregon (1995), Understanding U.S. History: Volume 1—Through the Civil War, Chapter 13: The Road to the Civil War pp. 318- 328.

The big idea of the unit is balance of power which gives the student something to think about as the material is learned and by doing so helps him integrate the material being learned.

By using the big idea of Balance of Power, the congressional compromises of the period, the Missouri compromise and the compromise of 1850, can be learned with some meaning, rather than as a mere parade of facts. For, example, here is how the Missouri compromise might be taught:

In 1819, the balance of power in the Senate was equal with 11 free states and 11 slave states. Adding new slave or free states to the United states could be thought of as adding extra players to a game—if one side got too many new players, the balance of power would shift to that side. But adding new players was exactly what was about to happen to the U.S. because the country was growing. It looked as if the balance of power might change.

In 1819, the people of the Missouri Territory asked to join the United States as a state. A territory was a region that was a part of the United States but not yet a state. Most of the white people living in Missouri wanted slavery. Northern Senators were against adding Missouri as a state because then there would be more slave states than free states with representation in the Senate. In 1819, the northern senators would not agree to let the South have a majority in the Senate. They argued for months against adding the Missouri Territory as a slave state.

After several months of debate, the people of Maine, which had been part of the state of Massachusetts, asked to join the United States as a free state. This opened the way for a compromise to be worked out. Henry Clay, a senator from Kentucky, proposed the compromise plan, called the Missouri Compromise. Henry Clay was one of the War Hawks who boasted before the War of 1812 that the Kentucky militia could conquer Canada. He also worked out a compromise for the Nullification Crisis.

Clay’s idea was to admit both states into the United States. One state, Maine, was free state which did not allow slavery. The other state, Missouri, did allow slavery. That maintained the balance of power.

The Missouri Compromise was a plan that drew a line across the Louisiana Purchase territory, south of which slavery would be allowed. Slavery would be outlawed in the rest of the Louisiana Purchase lands north of the imaginary line. The Missouri Compromise Line was drawn at the southern boundary of Missouri. Areas south of the Missouri Compromise line could become slaveholding states, but the land north of the line was to remain free from slavery. In 1820, the United States had not acquired the land west of the Louisiana Purchase. At the time it was passed, the Missouri Compromise established rules for all the land in the United States.

(Understanding U.S. History)

Now, see if you can answer the following question to see f you understand balance of power and how it affects the admission of territories as states.

Q: Arkansas was the next state to ask to join the United States. Arkansas was below the line of the Missouri Compromise, so it would be added as a slave state. What factors might influence Arkansas' admission as a state?

The same big idea can get you through the admission of Florida and Iowa, the election of President Polk, the concept of Manifest Destiny, the admission of California, New Mexico, Texas and the Oregon Territory, the Compromise of 1850, the South's attempt at seceding, the concept of popular sovereignty, and the passage of the Fugitive Slave Laws which culminate in the end of the balance of power era:

Although the Compromise of 1850 solved the problem of the southern states seceding from the United States, the solution was only temporary. Enforcing the Fugitive Slave Law and allowing popular sovereignty to decide about slavery only made the issues about slavery more difficult for Congress to solve. The ability of the Congress, the political parties, and the country as a whole to make any more compromises on the issue of slavery was coming to an end.

...

Until 1850, a balance of power between slave states and free states had existed. That balance of power resulted in compromises being made between the North and the South for many years. In the 1850s, three factors ended the ability of Congress to make compromises about slavery: (1) the idea of popular sovereignty, which began with the Compromise of 1850; (2) the Kansas-Nebraska Act of 1854, which led to violence and 200 deaths in Kansas; and (3) the Supreme Court’s Dred Scott decision in 1857.

(Understanding U.S. History)

The Dred Scott decision wasn't important because of the specific holding with respect to the slave Dred Scott (though I'm sure it was important to him), but was important becuase it eliminated Congress' ability to maintain the delicate balance of power that had existed and threatened to allow the unhindered growth of slavery:

The Dred Scott decision. The third factor that completely ended the ability of Congress to compromise on the issue of slavery was an 1857 Supreme Court ruling on slavery. The Court ruled that slaves had no right to sue in federal court because African Americans could not become citizens of the United States. The Supreme Court ruled that someone who owned slaves could keep slaves in any federal territory. The decision basically made slavery legal in any territory. The Supreme Court also ruled that there was nothing in the Constitution that allowed the federal Congress to outlaw slavery in any territory, and that only states could decide for themselves about slavery. That decision was the end of Congress’ ability to make compromises about slavery because Congress couldn’t pass any laws that limited slavery.

The Chief Justice who wrote the court’s decision was a southern Democrat who had always supported slavery. He felt that the decision in the Dred Scott case would end the debate over the expansion of slavery. He thought that once the court had decided to protect the property rights of slaveholders, the country would accept this decision. He was mistaken.

(Understanding U.S. History)

The challenge for techers is to teach all this material so that the students know it well enough to explain it without the referring to the textbook. This is the first prerequisite for knowledge to be used flexibly in higher order activities such as essays, comparisons, application, synthesis, evaluation, and the like.

There is no golden road to learning. Deep understanding in a domain (such as pre-civil war history) is not going to happen in a fact vacuum. You need something to think about first, before you can think about it deeply, i.e., understand the abstract functional relationships inherent in the material. If the student's "inquiry" hasn't led the student to learning the facts, at least temporarily, deep undersatnding isn't going to follow. This is what happened to the SLA student.

Tom Hoffman Attempts a Defense

Tom Hoffman of Tuttle SVC tries to defend SLA'a Dred Scott project that I recently critiqued. Hoffman attempts the old smoke and mirrors defense by attempting to portray the student's response as "deep understanding" of other stuff. You can be the judge of whether he's succeeded or not. I think he unwittingly proves my point as I indicate in the comments.

Hoffman's argument is another good window into the mind of the progressive educator. Deep understanding has been redefined to mean the amount of understanding you can achieve without knowing much content. To the rest of us, this is superficial understanding. History without historical facts or understanding of those facts. See the way that works?

I hope Hoffman will never try to "get my back" like he did here.

Chris Lehmann Responds

Chris Lehmann, Principal of Science Leadership Academy (SLA), responded to my criticism of one of SLA's projects in a recent post. Here is Chris's response with my comments.

One, let's start with the premise that you and I have fundamentally different views on educational philosophy, so I'd argue that you're predisposed to
disapprove of our school.

I'm not so sure about that. I assumed that our goals are similar: to maximize student learning. To the extent that I favor certain pedagogies over others, it is only because those pedagogies have evidence of superior results when we look at what the student has actually learned. But I am not beholden to any particular pedagogy for the sake of any particular educational philosophy.

Let's also state that, as you point out, we are completely transparent about our philosophy at SLA. Between my blog, the Family Night Book, the web site, etc, you can get a sense of what we believe pretty easily. Also, we encourage all
prospective students and parents to come spend a day at the school... not a special "everyone-visits" day, but any day. We want families to understand our educational philosophy because we want kids to make an informed decision about where they want to go to high school. So we're not trying to trick anyone into coming to SLA. Given that transparency, we've had a great interest in our school. We're also pleased with the academic results we've seen. So far, we are currently on track to have over a 90% four-year graduation rate from SLA, and by qualitative and quantitative metrics (attendance, course passing rates, PSAT scores, as well as early research by two PhD students), kids are doing very well.

I haven't seen any reported PSSA results for SLA yet. Will this year's results be reported?

So to the specific points you raise:

A) You show an interesting unwillingness to accept the scope of the piece or that the scope of the piece has merit. To do this work, the students had to examine primary source materials, learn about abolitionist societies, learn about the Dred Scott case through multiple lenses -- including the political one you favor. And then they had to create a piece of writing that a) showed historical understanding and b) made a decision to argue a point of view of one of the groups active at that time. All of that maps to the standards of the state for history.

Admittedly, the scope of the project is unclear based on what is provided in the handbook. I assumed, and I think fairly so, that the scope of a project for a high school history class would be a demonstration of historical understanding. Now, if the student expectations are diminished from this level, I'm curious to know in what way they are diminished and what is the expected pathway to this higher level since the curricular design is claimed to be backwards?

The handbook states that the project "can only be completed by showing both the skills and knowledge that are deemed to be critical to master the subject and demonstrate that deep level of understanding." What specific skills and specific knowledge were the students expected to have mastered? What was the level of specific understanding that was expected?

I do think that a high school student can be expected to perform the kind of analysis I provided if they are taught how to do it and taught the underlying content. In fact, I think middle-school student can be taught to this level. I'll provide an example in a future post.

My concern, and I think it is a valid concern, is that student learning is taking a backseat to the pedagogical fidelity. My opinion is that the student's understanding is banal and typical of what you see when the student lacks domain knowledge. (My third grade son comes home with stuff like this from school all the time even though he is capable of better.) My understanding is the theory is that it is better for the student to generate this knowledge on his own, rather than be told it. Fair enough. But the student's work product indicates that the required knowledge has not been generated, as the theory predicts, with the result being a superficial analysis. I just don't see how the student understanding can in anyway be considered to be deep at this point.

Moreover, during the course of the unit, students were engaged in a variety of instructional techniques. The assumptions you make about the kind of teaching that happens at SLA are wrong.

My assumptions were based on what was provided in the handbook and by what learning the student demonstrated. If I'm wrong, I'd like to understand how.

Moreover, even in my presentation that you cite, I talk about how traditional forms of assessment -- quizzes and tests -- have their place in our classrooms, they just are now lower on our hierarchy of assessment. Tests and quizzes are great ways to see if kids have learned how to handle skills and content in a narrowly defined context. What the student projects do is see if they can transfer those skills and content to a larger context.

The handbook states that "the primary assessment of student learning is through their projects" so I think it's fair to use the project, by itself, as the criterion of whether the learning meets the educational goals. I assumed that some teaching took place regarding the historic era in question and some background knowledge was taught, but I don't see how very much of this transfer took place.

B) I'd argue that [your] analysis of what was missing suggests your own bias toward what [you feel] is important to learn about that material. That's fine, the single greatest limiting factor in school is time. If you want to cover a great deal of material -- and even in "progressive" schools, history courses have a lot to cover, you are never going to get to every lens. And, by the way, while this assessment may not have asked kids to deal with the larger political lens, other work did. And yes, we do often question our balance with depth and breadth -- that's the question all good progressive schools should ask themselves, just as all good traditional schools should ask themselves about their balance of skills and content, information transferal and knowledge acquisition.

Again, the handbook indicates the goal was deep understanding and mastery of skills. I don't see either having taken place here at a ninth grade level. Which lens were expected if not the historic/political one considering this was a history class? I don't see depth nor breadth here, but then again I am not sure of the expectation which appears to be much less than what I consider to be high school level.

C) The level of analysis you suggest is warranted is collegiate -- or at least 11th or 12th grade -- in its complexity when this was a 9th grade piece of work. (We only had ninth graders when we published the book.) I think it holds up as a sophisticated, smart piece of work that shows an emergent sense of what it means to have a historical sense of the world.

Another way of saying this is that the student has not yet acquired an historic sense.

I'm going to post a middle school level text related to the same issue so we can judge the level of analysis we might expect. I get to hopefully by the weekend.

D) You and I have a fundamental disagreement over the value of skills v. content in student understanding. You believe that until the student has enough facts at their disposal, there is little benefit to asking them to "think like a historian." We believe that those skills develop with practice and that students' ability to develop the ability to apply an historical lens on the world requires frequent, guided, scaffolded practice. That's fine. We'll have to agree to disagree.

Again, the proof of the pudding is in the eating. I don't see how a series of superficial analyses is going to lead to expertise, nor I have I ever seen a deep analysis of an issue by one who was not in possession of a deep understanding of the underlying domain knowledge. Cog Sci tells us you don't acquire the latter until you've acquired the former. Nor is the student getting practice gaining expertise in this domain. The student isn't learning the underlying content. How is he going to learn how to analyze what he doesn't know?

Inquiry-driven learning doesn't mean you just set the kids off on Google. Guided inquiry means giving kids skills to access resources and make decisions on their own.

Has that been demonstrated here? I think just the opposite has been demonstrated. The resources weren't accessed and the right decisions were not made. Hallmarks of a novice.

I don't think you're going to read this comment and suddenly think, "Aha! I get it, SLA is wonderful!" But I also would hope that we could move beyond strawman arguments -- I'll promise not to argue that KIPP creates a bunch of automatons who merely can regurgitate what they have been told if you promise not to argue that SLA is some unstructured school where kids just are indoctrinated toward an ideological bias or kids merely learn some surface knowledge by surfing the web.

I do not assume that SLA is unstructured. I assume there is plenty of structure because no structure would be a disaster. I think that the problem is that the students' inquiry isn't leading to content knowledge and that the students' analysis suffers for it.

Regurgitation is not the goal, though inflexible knowledge is the typical starting point. However, not being able to even regurgitate is telling as well.

As always, I offer you the opportunity to come visit SLA. I don't think you'll like what you see there -- again, my goal is not to convince you that we're the one right school, but rather so you could see that there is more than one approach to schooling. I have no doubt that a thoughtful application of DI schoolwide can create an effective school where kids learn well. Can you entertain the same notion that an inquiry-driven approach that has been executed thoughtfully can do the same?

There is always the possibility, but I've yet to see the results with a population like that of SLA. I'm always looking to be convinced. Perhaps, there is a better example project for inclusion in next year's handbook.

Juxtaposition

Here's a nice juxtaposition from Teacher's Magazine's Blogboard on playing the teacher card*.

First we have Ryan's statement from I Thought A Think:

I think it can be universally accepted that teaching requires a certain skill set to transmit information to the students and get them to retain it. There's a science to teaching, and there's an art to teaching. I don't think it's out of line to suggest that if you haven't practiced the craft then you don't really have an authentic understanding of what's involved.

And then we have a comment from one of those teachers, Haley:

I agree with Ryan’s thinking in that it does take one to teach to fully understand the detailed process. Most people do not realize all of the details that are put into teaching and how every one of those details play a critical role in the learning process. For instance, in order to cater to all students’ needs, it is essential to incorporate different teaching styles to meet the varying learning styles students possess. It is imperative that teachers create lessons that are developed to meet the needs and interest levels of all students. I bet a lot of people did not even realize that there are different teaching and learning styles.

Moving right along.

*Playing the teacher card means claiming that teachers have some special knowledge with respect to education policy and should be deferred to.

Science Leadership Academy

I made it through about ten minutes of Chris Lehmann's keynote speech at this year's NECC entitled Progressive Pedagogy and the 21st Century. Nothing you haven't seen before, but here's an embedded link for you diehards.



Lehmann is the principal of the Science Leadership Academy (SLA). SLA is a Philadelphia-based magnet high school which claims to be, not unsurprisingly, an "inquiry-driven, project-based 21st Century school with a 1:1 laptop program." SLA also makes available its 2007 Family Night Book (PDF) for prospective students and their families. Caveat Emptor.

The Family Handbook and Lehman's speech paint a rosy picture of progressive education and technology. You're supposed to get the impression that progressive education is simply super and adding technology takes it one step higher -- to the super-duper level.

Normally, at this point I'd criticize the inquiry-driven, project-based brand of progressive-education practiced at SLA, but SLA has already done the heavy-lifting for me with its Family Handbook.

The Family handbook helpfully gives us actual examples of its student projects making it clear that Lehmann's lofty rhetoric doesn't survive its contact with reality.

According to the Family Handbook:

Although student projects, in general, generate more student interest they also often have involved multiple steps and drafts and can, quite often, require a great deal more effort than just studying for a test. One of the first things students realized when they first come to SLA is that projects here are different. The key to this difference lies in a concept that the faculty employs to create curriculum called “Backward Design.” Teachers in each course ask the question – “What are the enduring understandings students should have when they leave this class?” Teachers then create projects that can only be completed by showing both the skills and knowledge that are deemed to be critical to master the subject and demonstrate that deep level of understanding.

Furthermore, "the primary assessment of student learning is through their projects" (p. 9).

So let's take a look at one of these projects from the Family Handbook. I'm going to assume that this is a project that SLA is proud of and that the student example demonstrates mastery of both the skills and knowledge that are critical to master the subject and demonstrate a deep level of understanding.

The project comes from page 10 of the Family Handbook and pertains to African-American History. The project is:

To write a letter as the president of an abolitionist society to the group’s members upon hearing the decision in the Dred Scott case. Your letter should explain the decision, your perspective as a leader of the abolitionist cause, and what you want your membership to do about the decision. The content of the letter is restricted to information available in 1857.

It appears that the student is expected to consult the Supreme Court's 1857 Scott v Sanford decision (54 glorious pages of mid 19th century prose), and perhaps other secondary sources, and construct a sensible interpretation of the decision in light of pre-Civil war historical events like an expert historian might do. That's a tall order for a novice student lacking the deep-structured well of domain knowledge possessed by the historian. Lacking this domain knowledge, we would expect a superficial analysis of the decision focusing on surface features of the decision rather than the more important abstract, functional features needed for an expert analysis of the problem.

A decent high-school level analysis of Dred Scott might go something like this:

Briefly, the Dred Scott decision is important because it upset the political compromise at the federal level (the Missouri Compromise and the Kansas-Nebraska Act) which served to limit the spread of slavery. Each new state would now be able to determine for itself whether slavery would be permitted within its borders under the doctrine of popular sovereignty. At this time in U.S. history, the population of the western territories was growing rapidly and the U.S. was growing rapidly as territories petitioned for statehood. This rapid growth threatened to upset the existing balance between slave states and free states which maintained a gridlocked Senate. As long as the Senate was gridlocked, the North would not be able to pass a constitutional amendment banning slavery in all the states. The North was content with this arrangement as long as slavery was contained in the existing slave states and, thus, a political compromise had been reached which permitted the admission of free states and slave states in equal proportion. The Dred Scott decision threw out this political compromise in favor of popular sovereignty. Thus, the Dred Scott decision threatened to upset the existing political balance, would permit the growth of slavery, and would make it impossible to amend the Constitution to make slavery illegal in the U.S.

(This is my synthesis of about a half dozen relevant secondary sources)

A superficial analysis of Dred Scott, in contrast, would focus on the plight of the slave, Dred Scott, who was seeking his freedom in federal court. A superficial analysis would focus on Dred Scott and not the larger issue--the spread of slavery. With this in mind, let's take a look at the example student project that SLA chose to include in its handbook:

Greetings Members:

These past few months have been busy, with major changes being implemented in government decisions. In the Supreme Court, a case has recently been heard that affects our mission to help reverse the plight of African-Americans in this country. This case goes against our values as Pennsylvanians, Philadelphians, and abolitionists, and our beliefs in human rights.

In the case of Dred Scott v Sanford, the courts unfortunately favored Mrs. Emerson, Mr Scott’s “owner”. While under the rule of his master, Mr. Scott was taken to live in Illinois state and Wisconsin territory, respectively, both of which are free. Mr. Scott pleaded that since he had lived in those areas, he was entitled to his freedom, because it was illegal for him to be living there and be enslaved at the same time. The court eventually decided that Mr. Scott was not a citizen, being of the African persuasion, and therefore had no right to sue in a federal court. They went further as well, stating that our U.S. Congress has no right to declare certain states free and 0thers slave-owning.

This decision promotes the message that African Americans have no rights. In the majority write-up, Chief Justice Roger Taney writes: “They had for more than a century before been regarded as beings of an inferior order; … so far inferior that they had no rights which the white man was bound to respect; and that the negro might justly and lawfully be reduced to slavery for his benefit.” That statement is pure fiction. Negroes have had rights in our commonwealth for decades. One of our African American members, Mr. Forten, is a sail maker, and makes his living through his own business. He enjoys the right to own property and to make a profit. Mr. Forten would not benefit from being forced into backbreaking labor daily with insufficient food. No one would. The idea that slavery is beneficial to the slave is preposterous. No human would think it in his favor to be sold like a table and to work from sunrise to sunset, and then not get to reap a single seed he has sown.

Family Handbook, p. 10

This is a superficial analysis. The analysis focuses on the specific facts of the case, rather than the larger issue--the spread of slavery--presented by the decision. The student seems unaware that the (federal) citizenship issue was only important because Dred Scott was trying to gain redress in federal court based on diversity jurisdiction which required the plaintiff Scott to be a citizen of a different state than the defendant Sanford. The student fixates on the inflammatory language in the decision regarding plaintiff's citizenship which was merely of procedural importance. The analysis fails to cover any of the important issues presented by the decision, the historical context of the case, and why the case is historically important.

We shouldn't be surprised by the student's analysis. The student is a novice and is being asked to create a new interpretive narrative of facts stemming from the Dred Scott decision. The student, unlike the historian, is not an expert in pre-civil war history and how our republican form of government operates. The student lacks the background knowledge of the expert. Lacking this background knowledge, the student is going to have a difficult time separating the important facts from the unimportant facts, accessing what little relevant knowledge he possesses from his memory with the speed and accuracy needed to perform the analysis. The result is that the novice student is likely to focus on the surface features of the problem, rather than the abstract functional features of the problem that are needed to analyzes the problem. This is exactly what happened.

More importantly, the student's "inquiry" failed to teach the student what was supposed to be learned about the Missouri compromise, the Nebraska-Kansas act, bleeding Kansas, and all the other issues presented by slavery before the civil war. This wasn't supposed to be some silly busy-work project. This "project" was supposed to be "the primary assessment of student learning" showing "both the skills and knowledge that are deemed to be critical to master the subject and demonstrate that deep level of understanding." I think the project shows just the opposite. The student failed to acquire a deep understanding of the issues which was the entire object of the project.

The project continues with the student's recommendation for what the abolitionist society should do in light of the decision.

We must protest this unethical decision. Our society wishes its members to petition to lawmakers on the fairness of this ruling. The Chair of the Committee on the Judiciary in the Senate is one James Bayard Jr. of Delaware, a Democrat. His counterpart in the House of Representatives is George S. Houston, a Democrat hailing from Alabama. We beseech you to write to these men and protest this injustice! Help our brother in bonds who should rightfully be free. Every letter sent raises our chances to help Mr. Scott.

We thank you for your dutiful support of our efforts to help improve the condition of African Americans in our country and this Commonwealth. Many voices blend together to make a choir! With all of your voices singing, we know we make beautiful music. Please continue in your support so that one day we might all be free.

Petitioning the legislature's judiciary committee isn't going to accomplish much, especially for a 7-2 Supreme Court decision. And nevermind that the pre-civil war Democrats were mostly for the expansion of slavery, especially the southern Democrats, like the Senate member the student recommends petitioning. Again, the project shows that the student did not gain a deep understanding of the issues. A better answer would have been suggesting that its members persuade people to flood the new territories to affect the vote on whether slavery would be permitted under popular sovereignty which was the result of the decision.

Treating the student like a junior historian isn't going to make the student think like an historian. The student lacks the requisiste domain knowledge and the understanding of the abstract functional relationships inherent therein needed to perform the kind of analysis that an historian would be able to perform. This project isn't going to aid the student in acquiring that domain knowledge or give the student any practice thinking about the abstract relationahips which might have been gained by conducting a real analysis of the historical facts and the decision. Rather, the student wasted time conducting a superficial analysis which ignored all the relevant historical facts. The student missed a valuable opportunity to think about the important functional relationships presented by the exercise because the student was asked to create new content which is outsude his ability as a novice. This student didn't even gain practice in analyzing like an expert because he didn't analyze like an expert would have. It's a charade.

The fact that SLA included this project in the Family Handbook as an exemplary project is telling. This example supposedly shows the superior understanding supposedly conferred by the project-based inquiry pedagogy favored by SLA. Instead, what the example shows is that the "pedagogical" tail is wagging the "student learning" dog. That the student hasn't demonstrated a deep understanding isn't important. What is important is that SLA gets to teach according to its ideological bias.

It is somewhat embarrassing that SLA doesn't understand the difference between deep understanding and superficial understanding. Anyone can google "dred scott" and read the hits. But, it takes someone with some domain knowledge to separate the unimportant facts from the important ones and to synthesise the important facts into a good analysis. Sadly, this SLA student is no closer to that goal than he was before the project.

Update: Chris Lehmann stops by in the comments to provide a lengthy rebuttal of my arguments re SLA and this particular project. Chris actually advances the discussion and doesn't just give us talking points, whcih is a good thing. I'm tied up for the rest of the day and won't be able to respond until later. Play nice until then.

Update II: Here is my response.

Not Even Wrong

The New York Times reports on soon-to-be AFT presidente-for-life, Randi Weingarten's "new" vision for American education:

Can you imagine a federal law that promoted community schools — schools that serve the neediest children by bringing together under one roof all the services and activities they and their families need?

...

Imagine schools that are open all day and offer after-school and evening recreational activities, child care and preschool, tutoring and homework assistance.

...

Schools that include dental, medical and counseling clinics.

This is a very slick way of selling the proposed expansion of the public sector (and the expansion of the unionized jobs controlled by Weingarten that expansion entails).

Telemarketers do the same thing: Let me tell you about this great new service x we're offering ... Doesn't that sound great? let me verify your address ... can I go ahead and sign you up?

Telemarketer Weingarten wants you to focus on, i.e., imagine, all these great services provided for "the neediest children" instead of focusing on whether these services will boost student achievement and whether schools are the best vehicle for providing these non-educational services.

Weingarten just wants us to assume she is right. But, she's not right. She's not even wrong.

The problem is that she has no real basis for concluding that having schools take over the provisioning of these services will lead to increased student achievement.

She doesn't have evidence that providing more of these services will lead to increased student achievement.

That's because the research doesn't exist. There are no real world examples of schools providing these services and student achievement having risen. We have nothing, but for Weingarten's pretty words.

Call me a cynic, but I don't have much faith that public sector schools, especially those with unionized labor, are capable of providing these non-educational services any better than they currently provide educational services.

As long as a student (1) learns things easily from technically unsound teacher presentations, (2) readily retains what has been learned with technically unsound practice afterwards, and (3) has a strong familial support structure in place to keep the student on track and make sure the student's learning is progressing, then the student stands a good chance of becoming educated in public schools. This is because this student can be educated in virtually any school. We don't need a public sector monopoly to provide educational services to these students; any fool is up to the task.

Whether we need a public sector monopoly for providing educational services for the remaining kids is debatable. There certainly is no evidence that the public sector provides these services any better than the private sector. And, they've had an awfully long time trying to get their game together. Too long, maybe. It appears that they've given up. But for a few reformers, the present consensus, reflected in Weingarten's speech, is that students need to change for there to be improvement in student achievement. So Weingarten's idea is for schools to take over all the social services that they believe affect student achievement.

This is what we're supposed to be imagining -- allowing a dysfunctional monopoly to take over responsibilities outside of its core function. That makes little sense.

I find it easier to imagine just the opposite -- taking away the monopoly powers for providing educational services we've given to the public sector.

I'll print out this blog post and eat if, if anyone can provide a good reason why we shouldn't. The comments are open.

Update: Commenter Oldtimer informs us that the Cincinnati Public School stem has a school that provides all the social services Weingarten is advocating and has yet to show any improvement.

An Amusing read

Go check out this lengthy thread over at educationwonkette.

Academic guest posts a somewhat controversial, albeit politically-correct, view on race and education, gets challenged on it by commenters, and runs away without responding to the challenge.

It must be nice to have this kind of insulation from criticism in certain academic fields, but it doesn't help the reputation when you fail or are unable to defend your position in the real world.

New DI Program: Differentiated Reading

A new DI fluency building program, Differentiated Reading (PDF), was released today and is intended to be used with low performers who are reading at low fluency rates (less than 45 words a minute). The program is directed to children with whom simple fluency procedures have not proven to be effective. The program will be published commercially before the end of the year by SRA.

The article I linked to entitled Improving the Reading Rate of
Low Performers and is loaded with insights regarding the teaching of lower performers.

The frustration of slow readers

Trying to improve the reading rate of very low performers can be a frustrating experience for both learner and teacher. The learner typically knows that the goal is to read faster, without making a flurry of mistakes, and the learner tries, but the added effort most frequently leads to word guessing, word skipping, word stuttering, and to greatly increased physical signs of high energy, such as clenching their fists, taking deep breaths, and even sweating. The student knows how to try hard physically and thatʼs what he does. But it doesnʼt work for reading faster.

What Teachers Observe

The teacher may also notice that the studentʼs performance is not predictable from one day to the next. The typical pattern is for the learner to perform “better” on one day, and be very happy with his performance and the praise the teacher issues, but almost certainly, he reverts to his old habits on the next day and does poorly.

The teacher often concludes from observations that whatever it is that causes improvement is there one day and gone the next. The bottom-line conclusion is that something is wrong with the learnerʼs learning mechanisms.

This conclusion is thoughtful and comes after the teacher has tried
different approaches for improving rate-accuracy.

The Basic Rule

Teachers need an approach that permits students to show them through their reading behavior how much and how fast they can improve. The basic rule is that if students are properly motivated to read faster and donʼt, the reason is they canʼt.

Students Respond Logically

We donʼt want the task of learning to read a little faster to become an effort like Sisyphus trying to roll the rock out of the pit but never succeeding. This step is built around the fact that students respond to data. They are realistic. They know when they are failing and when they are progressing. If they receive good evidence they are doing well, and meeting reasonable expectations, they will keep trying and persist when they regress or when the material they read becomes a little more difficult.

If they canʼt see evidence of progress, they will tend to draw a conclusion we donʼt want them to draw—“I am a failure; I canʼt do it.”

The article describes the new program in detail and can be used by classroom teachers now, so there's no need to wait to use it.

Mystery Ingredient X

Tyler Cowen asks:

I don't think we have a recipe that says, "Take a child of two non-college educated parents, add primary education ingredient X, bake, and out comes a college-capable high school graduate." The mystery ingredient X has yet to be discovered.

Predictably, lots of commenters showed up and said the missing ingredient X is IQ. What they mean is that students continue to need a high IQ to succeed in today's education environment because, by and large, the prevalent educational techniques fail to simplify the complex concepts students need to learn to be "college-capable high school students." The result is that education remains largely inaccessible to those having low IQs.

I'd say mystery ingredient X will most likely involve something that simplifies the complex concepts that need to be learned, but currently aren't being learned by low-IQ students. (This also extends to low-SES students which largely overlap the low-IQ group.) DI provides a large portion of this mystery ingedient X at the elementary school level. In fact, this is speifically the goal of DI's design:

The net result of meeting these criteria is that DI materials appear to be easy. Possibly the most difficult concept for observers of DI programs to understand is that although the programs seem simple, they meet multiple design criteria that make them simple. The superficial impression of a program done right is that the authors may not understand some of the complexities of the content. The complexities, however, have been addressed and have been reduced to non-complexities that do not sacrifice the integrity of what is taught earlier or what is to be taught later. If the criteria are met, the prediction is that the student will generalize to a specified set of examples including those that have not been taught.

Rubric for Identifying Authentic DI Programs, pp. 18-19.

Today's Quotes

For most minority groups, then, and most particularly the Negro, schools provide no opportunity at all for them to overcome this initial deficiency; in fact, they fall farther behind the white majority in the development of several skills which are critical to making a living and participating fully in modern society. Whatever may be the combination of nonschool factors--poverty, community attitudes, low educational level of parents-which put minority children at a disadvantage in verbal and nonverbal skills when they enter the first grade, the fact is the schools have not overcome it.

Coleman Report, p. 20

The conclusion can then be drawn that improving the school of a minority pupil will increase his achievement more than will improving the school of a white child increase his. Similarly, the average minority pupil’s achievement will suffer more in a school of low quality than will the average white pupil’s. In short, whites, and to a lesser extent Oriental Americans, are less affected one way or the other by the quality of their schools than are minority pupils. This indicates that it is for the most disadvantaged children that improvements in school quality will make the most difference in achievement.

Coleman Report, p. 21

Relying on the Coleman Report

I'm embroiled in a bit of a debate over at Russo's with frequent commenter (though not here) John Thompson over the conclusions I drew for the Baltimore first grade reading achievement study posted below.

The argument revolves around the popular premise that the effects of low socio-economic status cause low student achievement. I argue that large-scale empirical research like the Baltimore study and Project Follow Through refute such causal interpretations which are merely based on the correlational data we have on SES and student achievement, like that found in the Coleman report.

John, however, raises the following point regarding the Coleman Report:

You should realize you wouldn't be digging yourself into such a deep hole if you would back off from gratuitous attacks on others, like the people who issued the Bolder Broader challenge, and didn't try to refute the Coleman Report with sweeping comments.

I am not trying to refute the Coleman report. I am refuting the causal implications John and his Broader, Bolder allies draw from the Coleman Report.

The Coleman Report of 1966 was based on achievement data on over a half million students. The Report noted the disparity that existed between at-risk students and those not at-risk. The report compared schools of equal physical characteristics serving both groups. Coleman's finding was that money spent on smaller classes, laboratories, counseling, higher teacher salaries, and higher teacher qualifications were not correlated with academic achievement.

If the physical characteristics of the schools and all the other factors made no difference in student performance, there still remains the possibility that the instruction at-risk students received was inadequate. This possibility, which has been shown to exist in such large scale controlled research studies as Project Follow through and the Baltimore study serves as clear indictment of the educational system and a clear premise that the instruction provided to at-risk kids needed to improve.

The premise goes like this: if the instruction is inadequate, school factors won't matter and SES effects will predominate. Conversely, if the instruction is adequate, school factors will predominate rather than SES factors. This premise is consistent with the Coleman Report findings which examined an educational system in which inadequate instruction predominated, before we discovered that adequate instruction did, in fact, exist.

This data has not been refuted by the Broader, Bolder proponents. They simply ignore it and hopes that no one will notice. This is precisely the reason I always bring it up--to watch them go through rhetorical gymnastics trying to explain it away.

Developmentally Appropriate Practice is Not Developmentally Appropriate

The Summer edition of American Educator is out and has another must-read article by cognitive scientist, Daniel Willingham, on the inappropriateness of developmentally appropriate practice in education.

I view "developmentally appropriate practice" as one of the many excuses educators use to avoid teaching at-risk kids, i.e. kids who are developmentally behind their peers, what they need to know to achieve academically. As with many of our most pernicious educational practices, this one has roots with Piaget. Here's a shocker:

let’s review Jean Piaget’s theory. Although development psychologists no longer believe that his theory is right, it is a good starting place

It is unfortunate that so many teachers and ed school professors haven't gotten this memo yet.

Willingham makes the point that there is considerable variability in children's cognition. If a child fails to understand a concept, for example, it does not mean that the task was somehow developmentally inappropriate. It often means that the task presented was flawed or otherwise, deficient.

For example, suppose you read Make Way for Ducklings to a preschool class. Midway through the story you ask, “What do you think will happen next?” and you are met with blank stares. You might think to yourself, “That question was developmentally inappropriate. It was too abstract to ask them to think about the future.” Maybe. But maybe no one has ever asked them to make a prediction about a story, and so they were just unsure of what to do, and would have answered readily if you had said, “Do you think the ducks will go back to the park or stay where they are?” Or maybe they hadn’t understood the story very well to that point, so they knew what you were asking, but they just didn’t know what might happen next. Or maybe they just don’t know that much about ducks.

If a child, or even the whole class, does not understand something, you should not assume that the task you posed was not developmentally appropriate. Maybe the students are missing the necessary background knowledge. Or maybe a different presentation of the same material would make it easier to understand.

This, I think, is the largest flaw of many of the common education memes like developmentally inappropriate practice. It provides a convenient excuse to focus the student's failure to learn on the student himself rather than on how the material was presented to the student. If the student has failed to understand, there is often an underlying reason which needs to be discovered, analyzed, and the presentation remedied to avoid the confusion preventing the student from learning. Labelling the task as developmentally inappropriate allows the teacher to avoid this difficult task. Being able to remedy the presentation to avoid students' failures of understand is one of the reasons why we pay for highly-educated professionals, and not merely trained technicians. Yet, oddly, educators are often reluctant to engage in this difficult activity, preferring the recourse of a myriad of labels that shift the blame for failing to learn to the students.

Many teachers don't even like having their curriculum scripted for them. Scripting the curriculum merely shifts the burden of creating teacher presentations that have been field tested and are known to be understandable by students to someone else. Being that teachers don't like doing this work on their own, you'd think they wouldn't mind if someone else did it for them. Yet, they don't, and I'm not quite sure why.

It's as if they want to be professionals and be treated like professionals, yet don't want to engage in the the hard work required of professionals to assure their services are being rendered properly. Willingham concludes:

If we accept that students’ failure to understand is not a matter of content, but either of presentation or a lack of background knowledge, then the natural extension is that no content should be off limits for school-age children.

Yet, developmentally appropriate practice frequently places content off-limits for many at-risk students because if is more difficult to present the material to them in a way that they understand.

As they say, go read the whole thing.

Apparently the poor can be taught to read if they're taught properly

Here's a large scale study that answers two important questions regarding the educability of "poor" students.

1. Poor kids often come to school far behind their middle class peers in reading ability, can this initial achievement gap be eliminated in a timely manner?
   
2. Can this initial achievement gap in reading ability be eliminated by a school-based instructional intervention?

The study is Academic Acceleration in First Grade Reading Using the Direct Instruction Model by Michael Rebar answers both questions in the affirmative.

I alluded to this study in an earlier post on Douglass High. As I indicated in that post, in the mid 90s Baltimore decided it would mandate the use of SRA's Open Court reading program in all its public elementary schools. At the same time, The Baltimore Curriculum Project contracted with the National Institute for Direct Instruction (NIFDI) to provide training, coaching, and support for reading instruction in eleven Baltimore schools.

This set up the conditions for a nice experiment. Eleven Baltimore City schools were found with similar demographics and achievement levels to the 11 NIFDI schools to serve as controls. Thus, the study involved three groups: the 11 NIFDI schools, the 11 matched control schools, and the remaining Baltimore schools. In total, between 1998 and 2003, 41,223 kindergartners and first graders in Baltimore participated in the study.

Here is the demographic break-down for all three groups:




The above table shows the average percentage of poor students (based on free and reduced lunch participation) in the NIFDI schools, the control schools, and the remaining Baltimore schools for each year of the experiment (1998-2003). As you can see, the poverty rate in Baltimore is high (about 72%) but the poverty rate at the NIFDI and control schools was even higher -- these were some of the poorest schools in the district. These schools were also the likely feeder schools for Douglass High.

The experiment was phased in over three years -- 1997-1999. Five additional schools initially chose to implement the NIFDI model, but dropped out; they are included in the Other Baltimore group. The number of schools in the Other Baltimore group varied between 103-122 schools.

In the spring of 1997, the Peabody Picture Vocabulary Test (PPVT) was administered to kindergartners in the NIFDI and Matched Control conditions. The PPVT is a widely used norm-referenced picture identification test that is highly predictive of future reading ability. The results serve as the best estimate of initial achievement reading level (i.e., prior to intervention).

The NIFDI schools received two 30 minute periods of reading instruction per day in K and 1. The curriculum was Reading Mastery Classic. The NIFDI schools also received an additional 30 minute period of language instruction using the DI program Language for Learning and Language for Thinking which focus on oral language development.

Prior to 1998, the Matched Control and Baltimore condition schools were free to use any curriculum program desired. There was no district-wide structured reading program and schools used a variety of instructional programs. In the fall of 1998, the district adopted Open Court Reading in kindergarten through second grade.

Let me translate that into English. Prior to the experiment, Baltimore schools taught Reading in whatever manner they desired with predictably bad results. In 1997 Baltimore schools were performing at the 27th percentile on the PPVT. In 1998, all but 11 of Baltimore's schools switched to the research-validated Open Court Program. The remaining 11 schools switched to DI under the guidance of NIFDI. Thus, the experiment allows us to make three comparisons: 1. between DI and Open Court, 2. between DI and the pre-experiment reading curricula, and 3. between Open Court and the pre-experiment reading curricula.

In the spring of 1998 and 1999 the Comprehensive Test of Basic Skills, Fourth Edition (CTBS/4) (CTB,1991) was administered to all Baltimore first graders. The Comprehensive Test of Basic Skills, Fifth Edition (CTBS/5–TerraNova) was administered to all first graders in 2000 through 2003. Here are the results., including the PPVT baseline results.



The NIFDI schools went from the 15th percentile in 1998 to the 75th percentile in 2003, a significant increase in student achievement. In contrast, the Matched Controlled schools (using Open Court) went from the 14th percentile in 1998 to the 40th percentile in 2003, also a significant increase, but not as much as the NIFDI schools. The Other Baltimore schools (using Open Court) went from the 24th percentile in 1998 to the 59th percentile in 2003, also a significant gain. Here are the effect sizes for each group.



As you can see, the NFDI intervention had a very large effect size of 1.87 standard deviations. Let's put this in context. The NFDI intervention was 750% more effective than lowering class sizes to 13-17 students. In fact, since most non-instructional interventions perform about as well or worse than lowering class size, the NIFDI DI intervention was at least 750% better than those as well.

Even the Matched Control Open Court schools had a large effect size of 0.90 standard deviations or 360% more effective than lowering class size. That's not too shabby at all.

Below is a graph of the NIFDI schools showing performance gains in normal curve equivalents.



Here's the analysis from the study:

The average student performance level in the NIFDI schools moved from the lowest quintile to well above the national mean. All NIFDI schools showed substantial achievement gains and several became beacon schools for the district. The three highest achieving schools in the district in 2003 were NIFDI schools. The Matched Control schools never achieved a mean score above the national average.

In all, 49 of the 103 Baltimore condition schools (48%) achieved a mean Total Reading NCE score of 55 or greater in the 2003 school year. In the NIFDI condition, ten of eleven schools (or 91% of these schools) reached this level. Only one of the eleven Matched Control condition schools reached an NCE score of 55; this school ranked 49th in the Baltimore condition.

The Fisher’s Exact Test indicates that schools in the NIFDI condition were significantly more likely to exceed the national mean than either the Baltimore or Matched Condition schools. The three highest achieving elementary schools in the district were NIFDI schools, achieving mean Total Reading NCE scores of 92, 90, and 83. The highest three schools in the Baltimore condition achieved scores of 78, 76, and 75. It is perhaps ironic that the highest achieving Baltimore school (with a mean NCE score of 78) is one of the DI schools that did not continue as a NIFDI school but continued to use the Direct Instruction reading program.

There you go. The top four schools in Baltimore were all schools using DI. Of course, the Baltimore schools, despite a long history of academic failure, still though they knew the best way to teach:

Despite the gains demonstrated with the NIFDI model in Baltimore in 1997 through 2003, the district did not systematically expand the model to other schools. In fact, during the course of this study the district mandated procedures that were at odds with model provisions. Teachers were required to attend district inservice trainings that advised them to do things differently than what the NIFDI model specifies. The central administration required schools to provide daily test-preparation periods for the entire school year. NIFDI advised the principals in their model not to do this because it believed greater gains would be possible by implementing the model rather than providing test preparation (which would also tend to artificially inflate achievement scores). In fact, the six principals who implemented the model most faithfully and ignored some of the district mandates achieved better performance than any of those who complied with district regulations.

So what do you think Baltimore did when faced with these improved scores?

That's right, they stopped the program and went back to what they were doing before the intervention. Unfortunately, I've been unable to find CTBS scores from 2004-2008.

So the answer to our initial questions is: Yes, we can eliminate the achievement gap between low-SES students and their middle-class peers in reading by the end of first grade by means of a school-based instructional reform. In fact, we can get low-SES kids to significantly outperform their middle-class peers. These achievement gains can be accomplished without any SES-based reforms or by reducing any of the symptoms associated with growing up in a low-SES environment.

So, how do all you Broader, Bolder types explain these results?