Tom Hoffman Attempts a Defense

Tom Hoffman of Tuttle SVC trys to defend SLA'a Dred Scott project that I recently critiqued. Hoffman attempts the old smoke and mirrors defense by attempting to portray the student's response as "deep understanding" of other stuff. You can be the judge of whether he's succeeded or not. I think he unwittingly proves my point as I indicate in the comments.

Hoffman's argument is another good window into the mind of the progressive educator. Deep understanding has been redefined to mean the amount of understanding you can achieve without knowing much content. To the rest of us, this is superficial understanding. History without historical facts or understanding of those facts. See the way that works?

I hope Hoffman will never try to "get my back" like he did here.

Chris Lehmann Responds

Chris Lehmann, Principal of Science Leadership Academy (SLA), responded to my criticism of one of SLA's projects in a recent post. Here is Chris's response with my comments.

One, let's start with the premise that you and I have fundamentally different views on educational philosophy, so I'd argue that you're predisposed to
disapprove of our school.

I'm not so sure about that. I assumed that our goals are similar: to maximize student learning. To the extent that I favor certain pedagogies over others, it is only because those pedagogies have evidence of superior results when we look at what the student has actually learned. But I am not beholden to any particular pedagogy for the sake of any particular educational philosophy.

Let's also state that, as you point out, we are completely transparent about our philosophy at SLA. Between my blog, the Family Night Book, the web site, etc, you can get a sense of what we believe pretty easily. Also, we encourage all
prospective students and parents to come spend a day at the school... not a special "everyone-visits" day, but any day. We want families to understand our educational philosophy because we want kids to make an informed decision about where they want to go to high school. So we're not trying to trick anyone into coming to SLA. Given that transparency, we've had a great interest in our school. We're also pleased with the academic results we've seen. So far, we are currently on track to have over a 90% four-year graduation rate from SLA, and by qualitative and quantitative metrics (attendance, course passing rates, PSAT scores, as well as early research by two PhD students), kids are doing very well.

I haven't seen any reported PSSA results for SLA yet. Will this year's results be reported?

So to the specific points you raise:

A) You show an interesting unwillingness to accept the scope of the piece or that the scope of the piece has merit. To do this work, the students had to examine primary source materials, learn about abolitionist societies, learn about the Dred Scott case through multiple lenses -- including the political one you favor. And then they had to create a piece of writing that a) showed historical understanding and b) made a decision to argue a point of view of one of the groups active at that time. All of that maps to the standards of the state for history.

Admittedly, the scope of the project is unclear based on what is provided in the handbook. I assumed, and I think fairly so, that the scope of a project for a high school history class would be a demonstration of historical understanding. Now, if the student expectations are diminished from this level, I'm curious to know in what way they are diminished and what is the expected pathway to this higher level since the curricular design is claimed to be backwards?

The handbook states that the project "can only be completed by showing both the skills and knowledge that are deemed to be critical to master the subject and demonstrate that deep level of understanding." What specific skills and specific knowledge were the students expected to have mastered? What was the level of specific understanding that was expected?

I do think that a high school student can be expected to perform the kind of analysis I provided if they are taught how to do it and taught the underlying content. In fact, I think middle-school student can be taught to this level. I'll provide an example in a future post.

My concern, and I think it is a valid concern, is that student learning is taking a backseat to the pedagogical fidelity. My opinion is that the student's understanding is banal and typical of what you see when the student lacks domain knowledge. (My third grade son comes home with stuff like this from school all the time even though he is capable of better.) My understanding is the theory is that it is better for the student to generate this knowledge on his own, rather than be told it. Fair enough. But the student's work product indicates that the required knowledge has not been generated, as the theory predicts, with the result being a superficial analysis. I just don't see how the student understanding can in anyway be considered to be deep at this point.

Moreover, during the course of the unit, students were engaged in a variety of instructional techniques. The assumptions you make about the kind of teaching that happens at SLA are wrong.

My assumptions were based on what was provided in the handbook and by what learning the student demonstrated. If I'm wrong, I'd like to understand how.

Moreover, even in my presentation that you cite, I talk about how traditional forms of assessment -- quizzes and tests -- have their place in our classrooms, they just are now lower on our hierarchy of assessment. Tests and quizzes are great ways to see if kids have learned how to handle skills and content in a narrowly defined context. What the student projects do is see if they can transfer those skills and content to a larger context.

The handbook states that "the primary assessment of student learning is through their projects" so I think it's fair to use the project, by itself, as the criterion of whether the learning meets the educational goals. I assumed that some teaching took place regarding the historic era in question and some background knowledge was taught, but I don't see how very much of this transfer took place.

B) I'd argue that [your] analysis of what was missing suggests your own bias toward what [you feel] is important to learn about that material. That's fine, the single greatest limiting factor in school is time. If you want to cover a great deal of material -- and even in "progressive" schools, history courses have a lot to cover, you are never going to get to every lens. And, by the way, while this assessment may not have asked kids to deal with the larger political lens, other work did. And yes, we do often question our balance with depth and breadth -- that's the question all good progressive schools should ask themselves, just as all good traditional schools should ask themselves about their balance of skills and content, information transferal and knowledge acquisition.

Again, the handbook indicates the goal was deep understanding and mastery of skills. I don't see either having taken place here at a ninth grade level. Which lens were expected if not the historic/political one considering this was a history class? I don't see depth nor breadth here, but then again I am not sure of the expectation which appears to be much less than what I consider to be high school level.

C) The level of analysis you suggest is warranted is collegiate -- or at least 11th or 12th grade -- in its complexity when this was a 9th grade piece of work. (We only had ninth graders when we published the book.) I think it holds up as a sophisticated, smart piece of work that shows an emergent sense of what it means to have a historical sense of the world.

Another way of saying this is that the student has not yet acquired an historic sense.

I'm going to post a middle school level text related to the same issue so we can judge the level of analysis we might expect. I get to hopefully by the weekend.

D) You and I have a fundamental disagreement over the value of skills v. content in student understanding. You believe that until the student has enough facts at their disposal, there is little benefit to asking them to "think like a historian." We believe that those skills develop with practice and that students' ability to develop the ability to apply an historical lens on the world requires frequent, guided, scaffolded practice. That's fine. We'll have to agree to disagree.

Again, the proof of the pudding is in the eating. I don't see how a series of superficial analyses is going to lead to expertise, nor I have I ever seen a deep analysis of an issue by one who was not in possession of a deep understanding of the underlying domain knowledge. Cog Sci tells us you don't acquire the latter until you've acquired the former. Nor is the student getting practice gaining expertise in this domain. The student isn't learning the underlying content. How is he going to learn how to analyze what he doesn't know?

Inquiry-driven learning doesn't mean you just set the kids off on Google. Guided inquiry means giving kids skills to access resources and make decisions on their own.

Has that been demonstrated here? I think just the opposite has been demonstrated. The resources weren't accessed and the right decisions were not made. Hallmarks of a novice.

I don't think you're going to read this comment and suddenly think, "Aha! I get it, SLA is wonderful!" But I also would hope that we could move beyond strawman arguments -- I'll promise not to argue that KIPP creates a bunch of automatons who merely can regurgitate what they have been told if you promise not to argue that SLA is some unstructured school where kids just are indoctrinated toward an ideological bias or kids merely learn some surface knowledge by surfing the web.

I do not assume that SLA is unstructured. I assume there is plenty of structure because no structure would be a disaster. I think that the problem is that the students' inquiry isn't leading to content knowledge and that the students' analysis suffers for it.

Regurgitation is not the goal, though inflexible knowledge is the typical starting point. However, not being able to even regurgitate is telling as well.

As always, I offer you the opportunity to come visit SLA. I don't think you'll like what you see there -- again, my goal is not to convince you that we're the one right school, but rather so you could see that there is more than one approach to schooling. I have no doubt that a thoughtful application of DI schoolwide can create an effective school where kids learn well. Can you entertain the same notion that an inquiry-driven approach that has been executed thoughtfully can do the same?

There is always the possibility, but I've yet to see the results with a population like that of SLA. I'm always looking to be convinced. Perhaps, there is a better example project for inclusion in next year's handbook.

Juxtaposition

Here's a nice juxtaposition from Teacher's Magazine's Blogboard on playing the teacher card*.

First we have Ryan's statement from I Thought A Think:

I think it can be universally accepted that teaching requires a certain skill set to transmit information to the students and get them to retain it. There's a science to teaching, and there's an art to teaching. I don't think it's out of line to suggest that if you haven't practiced the craft then you don't really have an authentic understanding of what's involved.

And then we have a comment from one of those teachers, Haley:

I agree with Ryan’s thinking in that it does take one to teach to fully understand the detailed process. Most people do not realize all of the details that are put into teaching and how every one of those details play a critical role in the learning process. For instance, in order to cater to all students’ needs, it is essential to incorporate different teaching styles to meet the varying learning styles students possess. It is imperative that teachers create lessons that are developed to meet the needs and interest levels of all students. I bet a lot of people did not even realize that there are different teaching and learning styles.

Moving right along.

*Playing the teacher card means claiming that teachers have some special knowledge with respect to education policy and should be deferred to.

Science Leadership Academy

I made it through about ten minutes of Chris Lehmann's keynote speech at this year's NECC entitled Progressive Pedagogy and the 21st Century. Nothing you haven't seen before, but here's an embedded link for you diehards.



Lehmann is the principal of the Science Leadership Academy (SLA). SLA is a Philadelphia-based magnet high school which claims to be, not unsurprisingly, an "inquiry-driven, project-based 21st Century school with a 1:1 laptop program." SLA also makes available its 2007 Family Night Book (PDF) for prospective students and their families. Caveat Emptor.

The Family Handbook and Lehman's speech paint a rosy picture of progressive education and technology. You're supposed to get the impression that progressive education is simply super and adding technology takes it one step higher -- to the super-duper level.

Normally, at this point I'd criticize the inquiry-driven, project-based brand of progressive-education practiced at SLA, but SLA has already done the heavy-lifting for me with its Family Handbook.

The Family handbook helpfully gives us actual examples of its student projects making it clear that Lehmann's lofty rhetoric doesn't survive its contact with reality.

According to the Family Handbook:

Although student projects, in general, generate more student interest they also often have involved multiple steps and drafts and can, quite often, require a great deal more effort than just studying for a test. One of the first things students realized when they first come to SLA is that projects here are different. The key to this difference lies in a concept that the faculty employs to create curriculum called “Backward Design.” Teachers in each course ask the question – “What are the enduring understandings students should have when they leave this class?” Teachers then create projects that can only be completed by showing both the skills and knowledge that are deemed to be critical to master the subject and demonstrate that deep level of understanding.

Furthermore, "the primary assessment of student learning is through their projects" (p. 9).

So let's take a look at one of these projects from the Family Handbook. I'm going to assume that this is a project that SLA is proud of and that the student example demonstrates mastery of both the skills and knowledge that are critical to master the subject and demonstrate a deep level of understanding.

The project comes from page 10 of the Family Handbook and pertains to African-American History. The project is:

To write a letter as the president of an abolitionist society to the group’s members upon hearing the decision in the Dred Scott case. Your letter should explain the decision, your perspective as a leader of the abolitionist cause, and what you want your membership to do about the decision. The content of the letter is restricted to information available in 1857.

It appears that the student is expected to consult the Supreme Court's 1857 Scott v Sanford decision (54 glorious pages of mid 19th century prose), and perhaps other secondary sources, and construct a sensible interpretation of the decision in light of pre-Civil war historical events like an expert historian might do. That's a tall order for a novice student lacking the deep-structured well of domain knowledge possessed by the historian. Lacking this domain knowledge, we would expect a superficial analysis of the decision focusing on surface features of the decision rather than the more important abstract, functional features needed for an expert analysis of the problem.

A decent high-school level analysis of Dred Scott might go something like this:

Briefly, the Dred Scott decision is important because it upset the political compromise at the federal level (the Missouri Compromise and the Kansas-Nebraska Act) which served to limit the spread of slavery. Each new state would now be able to determine for itself whether slavery would be permitted within its borders under the doctrine of popular sovereignty. At this time in U.S. history, the population of the western territories was growing rapidly and the U.S. was growing rapidly as territories petitioned for statehood. This rapid growth threatened to upset the existing balance between slave states and free states which maintained a gridlocked Senate. As long as the Senate was gridlocked, the North would not be able to pass a constitutional amendment banning slavery in all the states. The North was content with this arrangement as long as slavery was contained in the existing slave states and, thus, a political compromise had been reached which permitted the admission of free states and slave states in equal proportion. The Dred Scott decision threw out this political compromise in favor of popular sovereignty. Thus, the Dred Scott decision threatened to upset the existing political balance, would permit the growth of slavery, and would make it impossible to amend the Constitution to make slavery illegal in the U.S.

(This is my synthesis of about a half dozen relevant secondary sources)

A superficial analysis of Dred Scott, in contrast, would focus on the plight of the slave, Dred Scott, who was seeking his freedom in federal court. A superficial analysis would focus on Dred Scott and not the larger issue--the spread of slavery. With this in mind, let's take a look at the example student project that SLA chose to include in its handbook:

Greetings Members:

These past few months have been busy, with major changes being implemented in government decisions. In the Supreme Court, a case has recently been heard that affects our mission to help reverse the plight of African-Americans in this country. This case goes against our values as Pennsylvanians, Philadelphians, and abolitionists, and our beliefs in human rights.

In the case of Dred Scott v Sanford, the courts unfortunately favored Mrs. Emerson, Mr Scott’s “owner”. While under the rule of his master, Mr. Scott was taken to live in Illinois state and Wisconsin territory, respectively, both of which are free. Mr. Scott pleaded that since he had lived in those areas, he was entitled to his freedom, because it was illegal for him to be living there and be enslaved at the same time. The court eventually decided that Mr. Scott was not a citizen, being of the African persuasion, and therefore had no right to sue in a federal court. They went further as well, stating that our U.S. Congress has no right to declare certain states free and 0thers slave-owning.

This decision promotes the message that African Americans have no rights. In the majority write-up, Chief Justice Roger Taney writes: “They had for more than a century before been regarded as beings of an inferior order; … so far inferior that they had no rights which the white man was bound to respect; and that the negro might justly and lawfully be reduced to slavery for his benefit.” That statement is pure fiction. Negroes have had rights in our commonwealth for decades. One of our African American members, Mr. Forten, is a sail maker, and makes his living through his own business. He enjoys the right to own property and to make a profit. Mr. Forten would not benefit from being forced into backbreaking labor daily with insufficient food. No one would. The idea that slavery is beneficial to the slave is preposterous. No human would think it in his favor to be sold like a table and to work from sunrise to sunset, and then not get to reap a single seed he has sown.

Family Handbook, p. 10

This is a superficial analysis. The analysis focuses on the specific facts of the case, rather than the larger issue--the spread of slavery--presented by the decision. The student seems unaware that the (federal) citizenship issue was only important because Dred Scott was trying to gain redress in federal court based on diversity jurisdiction which required the plaintiff Scott to be a citizen of a different state than the defendant Sanford. The student fixates on the inflammatory language in the decision regarding plaintiff's citizenship which was merely of procedural importance. The analysis fails to cover any of the important issues presented by the decision, the historical context of the case, and why the case is historically important.

We shouldn't be surprised by the student's analysis. The student is a novice and is being asked to create a new interpretive narrative of facts stemming from the Dred Scott decision. The student, unlike the historian, is not an expert in pre-civil war history and how our republican form of government operates. The student lacks the background knowledge of the expert. Lacking this background knowledge, the student is going to have a difficult time separating the important facts from the unimportant facts, accessing what little relevant knowledge he possesses from his memory with the speed and accuracy needed to perform the analysis. The result is that the novice student is likely to focus on the surface features of the problem, rather than the abstract functional features of the problem that are needed to analyzes the problem. This is exactly what happened.

More importantly, the student's "inquiry" failed to teach the student what was supposed to be learned about the Missouri compromise, the Nebraska-Kansas act, bleeding Kansas, and all the other issues presented by slavery before the civil war. This wasn't supposed to be some silly busy-work project. This "project" was supposed to be "the primary assessment of student learning" showing "both the skills and knowledge that are deemed to be critical to master the subject and demonstrate that deep level of understanding." I think the project shows just the opposite. The student failed to acquire a deep understanding of the issues which was the entire object of the project.

The project continues with the student's recommendation for what the abolitionist society should do in light of the decision.

We must protest this unethical decision. Our society wishes its members to petition to lawmakers on the fairness of this ruling. The Chair of the Committee on the Judiciary in the Senate is one James Bayard Jr. of Delaware, a Democrat. His counterpart in the House of Representatives is George S. Houston, a Democrat hailing from Alabama. We beseech you to write to these men and protest this injustice! Help our brother in bonds who should rightfully be free. Every letter sent raises our chances to help Mr. Scott.

We thank you for your dutiful support of our efforts to help improve the condition of African Americans in our country and this Commonwealth. Many voices blend together to make a choir! With all of your voices singing, we know we make beautiful music. Please continue in your support so that one day we might all be free.

Petitioning the legislature's judiciary committee isn't going to accomplish much, especially for a 7-2 Supreme Court decision. And nevermind that the pre-civil war Democrats were mostly for the expansion of slavery, especially the southern Democrats, like the Senate member the student recommends petitioning. Again, the project shows that the student did not gain a deep understanding of the issues. A better answer would have been suggesting that its members persuade people to flood the new territories to affect the vote on whether slavery would be permitted under popular sovereignty which was the result of the decision.

Treating the student like a junior historian isn't going to make the student think like an historian. The student lacks the requisiste domain knowledge and the understanding of the abstract functional relationships inherent therein needed to perform the kind of analysis that an historian would be able to perform. This project isn't going to aid the student in acquiring that domain knowledge or give the student any practice thinking about the abstract relationahips which might have been gained by conducting a real analysis of the historical facts and the decision. Rather, the student wasted time conducting a superficial analysis which ignored all the relevant historical facts. The student missed a valuable opportunity to think about the important functional relationships presented by the exercise because the student was asked to create new content which is outsude his ability as a novice. This student didn't even gain practice in analyzing like an expert because he didn't analyze like an expert would have. It's a charade.

The fact that SLA included this project in the Family Handbook as an exemplary project is telling. This example supposedly shows the superior understanding supposedly conferred by the project-based inquiry pedagogy favored by SLA. Instead, what the example shows is that the "pedagogical" tail is wagging the "student learning" dog. That the student hasn't demonstrated a deep understanding isn't important. What is important is that SLA gets to teach according to its ideological bias.

It is somewhat embarrassing that SLA doesn't understand the difference between deep understanding and superficial understanding. Anyone can google "dred scott" and read the hits. But, it takes someone with some domain knowledge to separate the unimportant facts from the important ones and to synthesise the important facts into a good analysis. Sadly, this SLA student is no closer to that goal than he was before the project.

Update: Chris Lehmann stops by in the comments to provide a lengthy rebuttal of my arguments re SLA and this particular project. Chris actually advances the discussion and doesn't just give us talking points, whcih is a good thing. I'm tied up for the rest of the day and won't be able to respond until later. Play nice until then.

Update II: Here is my response.

Not Even Wrong

The New York Times reports on soon-to-be AFT presidente-for-life, Randi Weingarten's "new" vision for American education:

Can you imagine a federal law that promoted community schools — schools that serve the neediest children by bringing together under one roof all the services and activities they and their families need?

...

Imagine schools that are open all day and offer after-school and evening recreational activities, child care and preschool, tutoring and homework assistance.

...

Schools that include dental, medical and counseling clinics.

This is a very slick way of selling the proposed expansion of the public sector (and the expansion of the unionized jobs controlled by Weingarten that expansion entails).

Telemarketers do the same thing: Let me tell you about this great new service x we're offering ... Doesn't that sound great? let me verify your address ... can I go ahead and sign you up?

Telemarketer Weingarten wants you to focus on, i.e., imagine, all these great services provided for "the neediest children" instead of focusing on whether these services will boost student achievement and whether schools are the best vehicle for providing these non-educational services.

Weingarten just wants us to assume she is right. But, she's not right. She's not even wrong.

The problem is that she has no real basis for concluding that having schools take over the provisioning of these services will lead to increased student achievement.

She doesn't have evidence that providing more of these services will lead to increased student achievement.

That's because the research doesn't exist. There are no real world examples of schools providing these services and student achievement having risen. We have nothing, but for Weingarten's pretty words.

Call me a cynic, but I don't have much faith that public sector schools, especially those with unionized labor, are capable of providing these non-educational services any better than they currently provide educational services.

As long as a student (1) learns things easily from technically unsound teacher presentations, (2) readily retains what has been learned with technically unsound practice afterwards, and (3) has a strong familial support structure in place to keep the student on track and make sure the student's learning is progressing, then the student stands a good chance of becoming educated in public schools. This is because this student can be educated in virtually any school. We don't need a public sector monopoly to provide educational services to these students; any fool is up to the task.

Whether we need a public sector monopoly for providing educational services for the remaining kids is debatable. There certainly is no evidence that the public sector provides these services any better than the private sector. And, they've had an awfully long time trying to get their game together. Too long, maybe. It appears that they've given up. But for a few reformers, the present consensus, reflected in Weingarten's speech, is that students need to change for there to be improvement in student achievement. So Weingarten's idea is for schools to take over all the social services that they believe affect student achievement.

This is what we're supposed to be imagining -- allowing a dysfunctional monopoly to take over responsibilities outside of its core function. That makes little sense.

I find it easier to imagine just the opposite -- taking away the monopoly powers for providing educational services we've given to the public sector.

I'll print out this blog post and eat if, if anyone can provide a good reason why we shouldn't. The comments are open.

Update: Commenter Oldtimer informs us that the Cincinnati Public School stem has a school that provides all the social services Weingarten is advocating and has yet to show any improvement.

An Amusing read

Go check out this lengthy thread over at educationwonkette.

Academic guest posts a somewhat controversial, albeit politically-correct, view on race and education, gets challenged on it by commenters, and runs away without responding to the challenge.

It must be nice to have this kind of insulation from criticism in certain academic fields, but it doesn't help the reputation when you fail or are unable to defend your position in the real world.

New DI Program: Differentiated Reading

A new DI fluency building program, Differentiated Reading (PDF), was released today and is intended to be used with low performers who are reading at low fluency rates (less than 45 words a minute). The program is directed to children with whom simple fluency procedures have not proven to be effective. The program will be published commercially before the end of the year by SRA.

The article I linked to entitled Improving the Reading Rate of
Low Performers and is loaded with insights regarding the teaching of lower performers.

The frustration of slow readers

Trying to improve the reading rate of very low performers can be a frustrating experience for both learner and teacher. The learner typically knows that the goal is to read faster, without making a flurry of mistakes, and the learner tries, but the added effort most frequently leads to word guessing, word skipping, word stuttering, and to greatly increased physical signs of high energy, such as clenching their fists, taking deep breaths, and even sweating. The student knows how to try hard physically and thatʼs what he does. But it doesnʼt work for reading faster.

What Teachers Observe

The teacher may also notice that the studentʼs performance is not predictable from one day to the next. The typical pattern is for the learner to perform “better” on one day, and be very happy with his performance and the praise the teacher issues, but almost certainly, he reverts to his old habits on the next day and does poorly.

The teacher often concludes from observations that whatever it is that causes improvement is there one day and gone the next. The bottom-line conclusion is that something is wrong with the learnerʼs learning mechanisms.

This conclusion is thoughtful and comes after the teacher has tried
different approaches for improving rate-accuracy.

The Basic Rule

Teachers need an approach that permits students to show them through their reading behavior how much and how fast they can improve. The basic rule is that if students are properly motivated to read faster and donʼt, the reason is they canʼt.

Students Respond Logically

We donʼt want the task of learning to read a little faster to become an effort like Sisyphus trying to roll the rock out of the pit but never succeeding. This step is built around the fact that students respond to data. They are realistic. They know when they are failing and when they are progressing. If they receive good evidence they are doing well, and meeting reasonable expectations, they will keep trying and persist when they regress or when the material they read becomes a little more difficult.

If they canʼt see evidence of progress, they will tend to draw a conclusion we donʼt want them to draw—“I am a failure; I canʼt do it.”

The article describes the new program in detail and can be used by classroom teachers now, so there's no need to wait to use it.

Mystery Ingredient X

Tyler Cowen asks:

I don't think we have a recipe that says, "Take a child of two non-college educated parents, add primary education ingredient X, bake, and out comes a college-capable high school graduate." The mystery ingredient X has yet to be discovered.

Predictably, lots of commenters showed up and said the missing ingredient X is IQ. What they mean is that students continue to need a high IQ to succeed in today's education environment because, by and large, the prevalent educational techniques fail to simplify the complex concepts students need to learn to be "college-capable high school students." The result is that education remains largely inaccessible to those having low IQs.

I'd say mystery ingredient X will most likely involve something that simplifies the complex concepts that need to be learned, but currently aren't being learned by low-IQ students. (This also extends to low-SES students which largely overlap the low-IQ group.) DI provides a large portion of this mystery ingedient X at the elementary school level. In fact, this is speifically the goal of DI's design:

The net result of meeting these criteria is that DI materials appear to be easy. Possibly the most difficult concept for observers of DI programs to understand is that although the programs seem simple, they meet multiple design criteria that make them simple. The superficial impression of a program done right is that the authors may not understand some of the complexities of the content. The complexities, however, have been addressed and have been reduced to non-complexities that do not sacrifice the integrity of what is taught earlier or what is to be taught later. If the criteria are met, the prediction is that the student will generalize to a specified set of examples including those that have not been taught.

Rubric for Identifying Authentic DI Programs, pp. 18-19.

Today's Quotes

For most minority groups, then, and most particularly the Negro, schools provide no opportunity at all for them to overcome this initial deficiency; in fact, they fall farther behind the white majority in the development of several skills which are critical to making a living and participating fully in modern society. Whatever may be the combination of nonschool factors--poverty, community attitudes, low educational level of parents-which put minority children at a disadvantage in verbal and nonverbal skills when they enter the first grade, the fact is the schools have not overcome it.

Coleman Report, p. 20

The conclusion can then be drawn that improving the school of a minority pupil will increase his achievement more than will improving the school of a white child increase his. Similarly, the average minority pupil’s achievement will suffer more in a school of low quality than will the average white pupil’s. In short, whites, and to a lesser extent Oriental Americans, are less affected one way or the other by the quality of their schools than are minority pupils. This indicates that it is for the most disadvantaged children that improvements in school quality will make the most difference in achievement.

Coleman Report, p. 21

Relying on the Coleman Report

I'm embroiled in a bit of a debate over at Russo's with frequent commenter (though not here) John Thompson over the conclusions I drew for the Baltimore first grade reading achievement study posted below.

The argument revolves around the popular premise that the effects of low socio-economic status cause low student achievement. I argue that large-scale empirical research like the Baltimore study and Project Follow Through refute such causal interpretations which are merely based on the correlational data we have on SES and student achievement, like that found in the Coleman report.

John, however, raises the following point regarding the Coleman Report:

You should realize you wouldn't be digging yourself into such a deep hole if you would back off from gratuitous attacks on others, like the people who issued the Bolder Broader challenge, and didn't try to refute the Coleman Report with sweeping comments.

I am not trying to refute the Coleman report. I am refuting the causal implications John and his Broader, Bolder allies draw from the Coleman Report.

The Coleman Report of 1966 was based on achievement data on over a half million students. The Report noted the disparity that existed between at-risk students and those not at-risk. The report compared schools of equal physical characteristics serving both groups. Coleman's finding was that money spent on smaller classes, laboratories, counseling, higher teacher salaries, and higher teacher qualifications were not correlated with academic achievement.

If the physical characteristics of the schools and all the other factors made no difference in student performance, there still remains the possibility that the instruction at-risk students received was inadequate. This possibility, which has been shown to exist in such large scale controlled research studies as Project Follow through and the Baltimore study serves as clear indictment of the educational system and a clear premise that the instruction provided to at-risk kids needed to improve.

The premise goes like this: if the instruction is inadequate, school factors won't matter and SES effects will predominate. Conversely, if the instruction is adequate, school factors will predominate rather than SES factors. This premise is consistent with the Coleman Report findings which examined an educational system in which inadequate instruction predominated, before we discovered that adequate instruction did, in fact, exist.

This data has not been refuted by the Broader, Bolder proponents. They simply ignore it and hopes that no one will notice. This is precisely the reason I always bring it up--to watch them go through rhetorical gymnastics trying to explain it away.

Developmentally Appropriate Practice is Not Developmentally Appropriate

The Summer edition of American Educator is out and has another must-read article by cognitive scientist, Daniel Willingham, on the inappropriateness of developmentally appropriate practice in education.

I view "developmentally appropriate practice" as one of the many excuses educators use to avoid teaching at-risk kids, i.e. kids who are developmentally behind their peers, what they need to know to achieve academically. As with many of our most pernicious educational practices, this one has roots with Piaget. Here's a shocker:

let’s review Jean Piaget’s theory. Although development psychologists no longer believe that his theory is right, it is a good starting place

It is unfortunate that so many teachers and ed school professors haven't gotten this memo yet.

Willingham makes the point that there is considerable variability in children's cognition. If a child fails to understand a concept, for example, it does not mean that the task was somehow developmentally inappropriate. It often means that the task presented was flawed or otherwise, deficient.

For example, suppose you read Make Way for Ducklings to a preschool class. Midway through the story you ask, “What do you think will happen next?” and you are met with blank stares. You might think to yourself, “That question was developmentally inappropriate. It was too abstract to ask them to think about the future.” Maybe. But maybe no one has ever asked them to make a prediction about a story, and so they were just unsure of what to do, and would have answered readily if you had said, “Do you think the ducks will go back to the park or stay where they are?” Or maybe they hadn’t understood the story very well to that point, so they knew what you were asking, but they just didn’t know what might happen next. Or maybe they just don’t know that much about ducks.

If a child, or even the whole class, does not understand something, you should not assume that the task you posed was not developmentally appropriate. Maybe the students are missing the necessary background knowledge. Or maybe a different presentation of the same material would make it easier to understand.

This, I think, is the largest flaw of many of the common education memes like developmentally inappropriate practice. It provides a convenient excuse to focus the student's failure to learn on the student himself rather than on how the material was presented to the student. If the student has failed to understand, there is often an underlying reason which needs to be discovered, analyzed, and the presentation remedied to avoid the confusion preventing the student from learning. Labelling the task as developmentally inappropriate allows the teacher to avoid this difficult task. Being able to remedy the presentation to avoid students' failures of understand is one of the reasons why we pay for highly-educated professionals, and not merely trained technicians. Yet, oddly, educators are often reluctant to engage in this difficult activity, preferring the recourse of a myriad of labels that shift the blame for failing to learn to the students.

Many teachers don't even like having their curriculum scripted for them. Scripting the curriculum merely shifts the burden of creating teacher presentations that have been field tested and are known to be understandable by students to someone else. Being that teachers don't like doing this work on their own, you'd think they wouldn't mind if someone else did it for them. Yet, they don't, and I'm not quite sure why.

It's as if they want to be professionals and be treated like professionals, yet don't want to engage in the the hard work required of professionals to assure their services are being rendered properly. Willingham concludes:

If we accept that students’ failure to understand is not a matter of content, but either of presentation or a lack of background knowledge, then the natural extension is that no content should be off limits for school-age children.

Yet, developmentally appropriate practice frequently places content off-limits for many at-risk students because if is more difficult to present the material to them in a way that they understand.

As they say, go read the whole thing.

Apparently the poor can be taught to read if they're taught properly

Here's a large scale study that answers two important questions regarding the educability of "poor" students.

1. Poor kids often come to school far behind their middle class peers in reading ability, can this initial achievement gap be eliminated in a timely manner?
   
2. Can this initial achievement gap in reading ability be eliminated by a school-based instructional intervention?

The study is Academic Acceleration in First Grade Reading Using the Direct Instruction Model by Michael Rebar answers both questions in the affirmative.

I alluded to this study in an earlier post on Douglass High. As I indicated in that post, in the mid 90s Baltimore decided it would mandate the use of SRA's Open Court reading program in all its public elementary schools. At the same time, The Baltimore Curriculum Project contracted with the National Institute for Direct Instruction (NIFDI) to provide training, coaching, and support for reading instruction in eleven Baltimore schools.

This set up the conditions for a nice experiment. Eleven Baltimore City schools were found with similar demographics and achievement levels to the 11 NIFDI schools to serve as controls. Thus, the study involved three groups: the 11 NIFDI schools, the 11 matched control schools, and the remaining Baltimore schools. In total, between 1998 and 2003, 41,223 kindergartners and first graders in Baltimore participated in the study.

Here is the demographic break-down for all three groups:




The above table shows the average percentage of poor students (based on free and reduced lunch participation) in the NIFDI schools, the control schools, and the remaining Baltimore schools for each year of the experiment (1998-2003). As you can see, the poverty rate in Baltimore is high (about 72%) but the poverty rate at the NIFDI and control schools was even higher -- these were some of the poorest schools in the district. These schools were also the likely feeder schools for Douglass High.

The experiment was phased in over three years -- 1997-1999. Five additional schools initially chose to implement the NIFDI model, but dropped out; they are included in the Other Baltimore group. The number of schools in the Other Baltimore group varied between 103-122 schools.

In the spring of 1997, the Peabody Picture Vocabulary Test (PPVT) was administered to kindergartners in the NIFDI and Matched Control conditions. The PPVT is a widely used norm-referenced picture identification test that is highly predictive of future reading ability. The results serve as the best estimate of initial achievement reading level (i.e., prior to intervention).

The NIFDI schools received two 30 minute periods of reading instruction per day in K and 1. The curriculum was Reading Mastery Classic. The NIFDI schools also received an additional 30 minute period of language instruction using the DI program Language for Learning and Language for Thinking which focus on oral language development.

Prior to 1998, the Matched Control and Baltimore condition schools were free to use any curriculum program desired. There was no district-wide structured reading program and schools used a variety of instructional programs. In the fall of 1998, the district adopted Open Court Reading in kindergarten through second grade.

Let me translate that into English. Prior to the experiment, Baltimore schools taught Reading in whatever manner they desired with predictably bad results. In 1997 Baltimore schools were performing at the 27th percentile on the PPVT. In 1998, all but 11 of Baltimore's schools switched to the research-validated Open Court Program. The remaining 11 schools switched to DI under the guidance of NIFDI. Thus, the experiment allows us to make three comparisons: 1. between DI and Open Court, 2. between DI and the pre-experiment reading curricula, and 3. between Open Court and the pre-experiment reading curricula.

In the spring of 1998 and 1999 the Comprehensive Test of Basic Skills, Fourth Edition (CTBS/4) (CTB,1991) was administered to all Baltimore first graders. The Comprehensive Test of Basic Skills, Fifth Edition (CTBS/5–TerraNova) was administered to all first graders in 2000 through 2003. Here are the results., including the PPVT baseline results.



The NIFDI schools went from the 15th percentile in 1998 to the 75th percentile in 2003, a significant increase in student achievement. In contrast, the Matched Controlled schools (using Open Court) went from the 14th percentile in 1998 to the 40th percentile in 2003, also a significant increase, but not as much as the NIFDI schools. The Other Baltimore schools (using Open Court) went from the 24th percentile in 1998 to the 59th percentile in 2003, also a significant gain. Here are the effect sizes for each group.



As you can see, the NFDI intervention had a very large effect size of 1.87 standard deviations. Let's put this in context. The NFDI intervention was 750% more effective than lowering class sizes to 13-17 students. In fact, since most non-instructional interventions perform about as well or worse than lowering class size, the NIFDI DI intervention was at least 750% better than those as well.

Even the Matched Control Open Court schools had a large effect size of 0.90 standard deviations or 360% more effective than lowering class size. That's not too shabby at all.

Below is a graph of the NIFDI schools showing performance gains in normal curve equivalents.



Here's the analysis from the study:

The average student performance level in the NIFDI schools moved from the lowest quintile to well above the national mean. All NIFDI schools showed substantial achievement gains and several became beacon schools for the district. The three highest achieving schools in the district in 2003 were NIFDI schools. The Matched Control schools never achieved a mean score above the national average.

In all, 49 of the 103 Baltimore condition schools (48%) achieved a mean Total Reading NCE score of 55 or greater in the 2003 school year. In the NIFDI condition, ten of eleven schools (or 91% of these schools) reached this level. Only one of the eleven Matched Control condition schools reached an NCE score of 55; this school ranked 49th in the Baltimore condition.

The Fisher’s Exact Test indicates that schools in the NIFDI condition were significantly more likely to exceed the national mean than either the Baltimore or Matched Condition schools. The three highest achieving elementary schools in the district were NIFDI schools, achieving mean Total Reading NCE scores of 92, 90, and 83. The highest three schools in the Baltimore condition achieved scores of 78, 76, and 75. It is perhaps ironic that the highest achieving Baltimore school (with a mean NCE score of 78) is one of the DI schools that did not continue as a NIFDI school but continued to use the Direct Instruction reading program.

There you go. The top four schools in Baltimore were all schools using DI. Of course, the Baltimore schools, despite a long history of academic failure, still though they knew the best way to teach:

Despite the gains demonstrated with the NIFDI model in Baltimore in 1997 through 2003, the district did not systematically expand the model to other schools. In fact, during the course of this study the district mandated procedures that were at odds with model provisions. Teachers were required to attend district inservice trainings that advised them to do things differently than what the NIFDI model specifies. The central administration required schools to provide daily test-preparation periods for the entire school year. NIFDI advised the principals in their model not to do this because it believed greater gains would be possible by implementing the model rather than providing test preparation (which would also tend to artificially inflate achievement scores). In fact, the six principals who implemented the model most faithfully and ignored some of the district mandates achieved better performance than any of those who complied with district regulations.

So what do you think Baltimore did when faced with these improved scores?

That's right, they stopped the program and went back to what they were doing before the intervention. Unfortunately, I've been unable to find CTBS scores from 2004-2008.

So the answer to our initial questions is: Yes, we can eliminate the achievement gap between low-SES students and their middle-class peers in reading by the end of first grade by means of a school-based instructional reform. In fact, we can get low-SES kids to significantly outperform their middle-class peers. These achievement gains can be accomplished without any SES-based reforms or by reducing any of the symptoms associated with growing up in a low-SES environment.

So, how do all you Broader, Bolder types explain these results?

Douglass High: Pro NCLB?

I caught a partial view of HBO's Douglass High Documentary and I have a question.

Was the Documentary supposed to be anti-NCLB or pro-NCLB?

To me it seems pro-NCLB, though that was probably not intentional. I can't see how anyone could view the documentary and come away with the impression that the administrators of Douglass don't need independent oversight. Their idea of education seems to be seat time. If the student is capable of sitting in his seat during class, regardless of what he's learning, he'll be passed along for four years and graduate. What he's actually learned, if anything, is irrelevant.

I think the average person would be horrified at what passes for education at Douglass. It is probably best that they don't know because I think support for the notion of public education would be greatly diminished.

Sixty percent of Douglass' student drop out between ninth and twelfth grade. Granted, most of the student came to Douglass way behind where they should have been, but the performance of the 40% that didn't drop out was appalling. Only a tiny percentage of students tested as proficient in math and reading. These were the good students, with parental support. They all seemed well fed and well dressed (fashion sense notwithstanding). The teachers admitted that nearly none were at grade level, yet miraculously almost every student qualified for graduation in the end.

I don't think this is what the public has in mind when they signed on for public education.

More on Douglass High

The debate continues to rage over who or what is responsible for Douglass High's poor academic performance.

Douglass High's students' fate was determined long before they entered high school. First Grade performance is a good indicator of subsequent academic performance. If anything, first grade performance tends to be a high water mark.

When the students shown in the Douglass High documentary (shot in 2004-05) were in first grade, Baltimore's academic performance was pitiful--so pitiful that Baltimore decided to take drastic action in 1998. That year Baltimore mandated that almost all Baltimore schools adopt the Open Court reading program as their core reading curriculum. That year they also decided to test all first graders using CTBS. Here are the results.



In 1998, Baltimore students were performing at about the 24th percentile on CTBS/4. And, you can bet the Douglass was drawing students from schools falling below this average. The Douglass students shown in the Documentary were past first grade by 1998, but the low scores of the 1998 cohort are indicative of the quality of education they received in K and 1. Academically, you could write off 75% of Baltimore students. There is a very good chance that a struggling read in first grade will be a struggling reader in third grade and last I checked it's difficult to motivate a child to learn if they can't read very well.

After adopting Open Court, Baltimore's first grade reading performance shot up to the 59th percentile. These 2003 first graders are not yet in high school.

I think it's safe to assume that the incidence of toothaches, bellyaches, student hunger, racial discrimination, bad home life, uncaring parents, and all the other excuses teachers like to bandy about to excuse their performance remained about the same. The only thing that changed was the reading curriculum to something that has a validated research base. Seemingly by magic, reading performance improved by about a standard deviation. This, supposedly, is impossible.

The study from which I stole this table actually indicates that student reading performance in Baltimore could have (and actually was) improved by twice this amount in 11 high-poverty schools that adopted an even better reading curricula. You'll have to wait for the next post to find out more.

Nonetheless, it's safe to say that a large part of the problems present at Douglass High could have been prevented if the Baltimore schools actually did a better job doing what we pay them to do -- educating children.

Douglass High

Though my DVR has yet to record HBO's documentary, Hard Times at Douglass High, that's not going to stop me from opining anyway.

Douglass High's primary problem is that most of its students do not possess the skills needed to access a high school curriculum. Most of the students don't belong at Douglass High or any high school for that matter. Most of the them belong in remedial classes teaching elementary school material. This, of course, assumes that the students are still willing to sit in a classroom attempting to learn after experiencing nine years of academic failure. I doubt many of them are. I suspect many of them are causing serious disruptions to whatever classrooms they happen to be in.

Douglass High's problem is social promotion. The elementary and middle schools that feed into Douglass High failed to properly educate their students and passed the buck right on to Douglass High. Douglass was handed a thankless job.

Douglass High is the end point for a very broken educational system. Even if Douglass High were capable of remediating its students at a high rate, the results would still likely not show up in any state test.

Douglass High's primary problems lie elsewhere.

News from the trenches

My son, who begins third grade in the fall, just finished Connecting Math Concepts level C (CMC C). CMC C is billed as a 2nd or 3rd grade curriculum.

We went through the 120 lesson sequence sporadically on weekday nights after school and weekends. It took about nine months. Had we been more diligent, we would have gotten through it in five months.

In his day job, i.e., second grade, he covered most of the 2nd grade Everyday Math (EM) curriculum. Generally, we were ahead of what his school was teaching in EM, so EM mostly served as additional practice for what he was learning in CMC. Both curricula covered roughly the same skills, if anything CMC covered more topics than EM.

CMC provided significantly more distributed practice than EM which should not be surprising. The benefit of all this distributed practice was that at the end of the sequence he was able to take and pass a cumulative test of all the material covered in the sequence. In other words, he had retained material that had been covered up to nine months ago. We will jump right into level D because the more we do this summer, the less we will have to do during the school year, what with regular homework, piano lssons, various sports, and the like. I'm confident that he could go the entire summer without forgetting what he was taught, at least nothing that couldn't be refreshed with a few lessons of review. CMC Level D provides that review, but we'll skip those parts.

In CMC, lessons are easy to teach because they are scripted for an entire classroom. The difficult part is shortening and modifying the script to suit teaching a single student. I will present the script until he has learned what was intended and learn, which usually takes one example. Then I monitor his working of the remaining examples. Two thirds of each lesson is usually independent work practicing skills he's already learned. He does those exercises on his own, though I often let him skip parts of exercises if he balks or seems bored by the problems. Surprisingly, he will often work problems he already knows how to solve well without complaint.

We usually get through a lesson in about half an hour, fifteen minutes of which is independent work. I suspect presenting to an entire classroom would take longer. I also suspect that I'd have to do more teaching for average and lower performing students.

EM is known for its steep spiral. Topics come and go quickly. Nothing is taught to mastery. And, quite frankly, I don't see any of the supposed conceptual learning advantages I see touted. CMC does not teaching using a spiral and everything is taught to mastery. Needless to say I was surprised that CMC stayed ahead of the EM spiral the entire school year. To put it simply, CMC teaches much more efficiently than EM.

Perhaps too efficiently. My son has little tolerance for the hokey manipulative exercises that clutter EM, such as make a grid of dots to conceptualize multiplication. He knows how to do most addition, subtraction, multiplication, and some division problems mentally, so that is how he wants to solve problems -- mentally. I suspect there will be a clash next year when he writes down his "magic" solutions to problems. We shall see.

He knows the math part of math well. That's a good thing. he doesn't know some other important parts of math so well, however. he doesn't know the importance of neatness yet. he doesn't consistently provide the units for word problem answers. He doesn't consistently indicate the answers by, say, boxing them.

These problems are all my fault. I should have held him to a stricter standard, but I didn't for various reasons, mostly time considerations and to minimize crankiness when we did this extra work before bedtime. These problems are easy to remedy. Remedying deficient math knowledge not so much.

Next year he'll run through the EM spiral again reviewing, and for many students relearning, all the partially taught topics from second grade. In the meantime, he'll continue to overlearn the topics material covered in CMC D. This will lead to automaticity in elementary math skills which will be needed for learning algebra. That's the ultimate payoff -- learning algebra. If you don't know algebra, you're pretty much foreclosed from pursuing study in the hard sciences and engineering.

Facebook

Although I am way behind the curve, I just joined Facebook. Looks like fun. I sent out a bunch of invites from my yahoo address book. If I missed you, feel free to add me.

Question 6: Why not to use manipulatives for learning math

Ari-free asks:

in an interview, Zig Engelmann said that he doesn't believe in using manipulatives for teaching math. I thought it was universally accepted that kids start with counting blocks, fingers, etc

Here is what Engelmann and Carnine have written on the topic or the need for making mathematical connections:

Connecting Math Concepts presents skills in applications in a way that permits children to connect concepts.

...

If the program did an effective job of articulating connections between various concepts, the child who completes the first level would have a strong conceptual foundation on which to build...

How does Connecting Math Concepts forge these basic connections?

The principal vehicle is the number line. The number line is potentially dangerous for teaching number facts because it provides children with a method of figuring out the answer without attending to relationship. To figure out 4 plus 3, the children simply go to 4 on the number line and then move three spaces further. Although children learn the right answer, they don't easily learn the relationship between 4+2 and 4+3 because they have never been required to do anything that focuses on the relationship. The just follow the number line as a "rote" procedure.

A similar problem exists with any method that permits children to count and figure out the answer to problems like 4 plus 2. Finger operations are logical. (Hold up 4 fingers; hold up 2 more; count all.) However, these operations don't prompt systematic learning of relationships because children are not required to attend to relationships. Line-making operations suffer from the same problem. They provide children with a reliable method for figuring out answers to problems; however, they do not induce fact knowledge or knowledge of relationships. Furthermore, they often militate against systematic learning of facts because so long as the child is able to use finger-counting operation or a number line, what motivation is there for learning facts?

Connecting Math Concepts, Level A, Teacher's Guide, pp. 6-7.

Question 5: Various Questions about DI

Robert asks:

What is the total number of people/budget for R&D in DI? What portion of that is outside of Eugene Oregon?

I'm not privy to the inner workings of the various DI organizations like ADI and NIFDI and the Becker/Engelmann Corp, but I do know they are small organizations.

Why are not more of the materials made targeted at the home school market? 100 math lessons, 100 writing lessons..etc.

This is probably a licensing issue with SRA. I agree that it would be a good idea to adapt the DI materials for the homeschooling crowd which is a big untapped market.

It seems to me that DI is good at instilling basic skills, probably to about the 5th grade level. But that there must be a transition at some point to a more conceptual, self directed learning style after that. Such that an properly prepared 5th grader (or 6th or 7th etc) should be able to be given a algebra book and expected to learn form it and ask questions every few days if they need help. Given that I know this happens with some kids, I think it good to figure out if this could happen with more kids. Part of this also has to deal with instilling an internal drive, because an unmotivated child being given an algebra book isn't going to get real far.

Drawing back a bit, DI can be used to transmit known systematized information, but at some point the learner needs to get beyond that into creating knowledge of their own...it seems like there should be some more transitions between DI and Graduate School, but I am not sure what those transitions look like.

Any ideas?

I am somewhat familiar with most of the DI classroom materials. Most of the scaffolding is systematically removed between level 1 and level 6 with the intention that students completing level 6 should be ready for instruction in a more traditional classroom setting or independent work.

I don't know how successful this transition is with lower performers; the data is scant.

Does DI have any long term affect on measured IQ?

I doubt it. I doubt it even has an effect in the short term. The Follow Through students all took Ravens Progressive Matrix tests (which is highly g-loaded) which did not show significant improvement in IQ. However, IQ is often estimated from achievement tests and on achievement tests the DI students showed substantial improvement in achievement and IQ in the short term. But, the achievement tests require further achievement gains to maintain gains, so its difficult to attribute this achievement to DI once the students are out of DI.

What can be done to improve long term IQ?

Presently, it appears that we don't know of anything that is capable of improving IQ past adolescence.

What can be done to maximize a persons ability to capitalize on the IQ that they have? (outside if DI)

Clear teaching and sufficient practice for retention.

Question 4: What to do after 100 easy Lessons

Ari-free asks:

1) My sister is teaching her 4 year old daughter with "Teach Your Child to Read in 100 Easy Lessons" What should be the next step for her?

For most students, the next step is Reading Mastery III which you can find on EBay. Another option is Horizons 3/4 (the other Direct Instruction curriculum) which squeezes two years on instruction into one year. This is also available on EBay but more difficult to find.

2) What does Singapore Math (the new CA standards version) look like from the perspective of Direct Instruction?

I'd say that SM is a good coherent curriculum that requires a fair amount of teacher skill to teach. Lacking such skill, a good alternative is to use the DI math curriculum Connecting Math Concepts as the primary curriculum and supplement it with SM (and Saxon for the die-hard).

Question 3: What material should be overlearned

EHT asks:

[S]hould we expect each and every student to have total recall 9 months later? OR Should we expect students to be able to read a question regarding older content and be able to touch upon the right answer by reviewing the answer choices carefully and logically? AND shouldn't we expect students to forget certain things, but be able to have the skills to jog their memory such as how to check a reference source, etc.?

Willingham addresses this issue at length. So, let's go right to the source.

[T]he following types of material are worthy of practice:

1. The core skills and knowledge that will be used again and again. In this case, we give practice in order to ensure automaticity. The student who struggles to remember the rules of punctuation and usage (or must stop to look them up in a reference book) cannot devote sufficient working memory resources to building a compelling argument in his or her writing. The student who does not have simple math facts at his or her disposal will struggle with higher math.

2. The type of knowledge that students need to know well in the short term to enable long-term retention of key concepts. In this case, short-term overlearning is merited. For example, as noted earlier, a science teacher may want students to know a set of facts about certain species so that she can introduce an important abstract concept concerning evolution that depends on these facts. Or, a high school history teacher may want students to master the facts of several Supreme Court cases in order to build long-term understanding of a particular constitutional principle.

3. The type of knowledge we believe is important enough that students should remember it later in life. In this case, one might consider certain material so vital to an education that it is worthy of sustained practice over many years to assure that students remember it all of their life. A science teacher might spend the better part of a year emphasizing basic principles of evolution in the belief that the material is essential to consider oneself conversant in biology. Further, the curriculum might address and require practice in evolution in multiple years to assure that such knowledge will last a lifetime. Do we expect that a 40-year-old will have retained everything learned through the 12th grade? No, but do we expect that she will retain anything? Should she be able to grasp the basics of evolution or describe the different responsibilities of the three branches of the federal government or calculate the area of a circle? Exactly what sorts of knowledge merit the focus required to create long-lasting memory will be controversial, but that practice is required to create such memories is not.

How should practice be structured--should a teacher strive for overlearning in the short term or repeated learning over the long term? The answer will depend on whether the goal is automaticity in skills, short-term knowledge, or long-term knowledge--and what the teacher knows about the future curriculum students will encounter. For example, an English teacher might deem it very important that students understand the use of metaphor in poetry, but extensive, focused practice may not be practical or necessary. This knowledge will likely be developed over a number of years, and there will be opportunities for practice in the future. In other cases there will be future opportunities for practice, but the timeliness of the learning is important. For example, one teacher might provide just a cursory introduction to first-graders on how to tell time, figuring that the students will have ample opportunities for practice in the future. But another teacher might also reason that first-graders need to know how to tell time (so that, for example, they can monitor their activities during the day and be more self-directed) and so focus practice on this skill. Similarly, a French teacher may realize that students will have plenty of practice conjugating the verb être (to be) over the long term, but may justly believe that students must know this material early in their training or their ability to read, write, and understand French will be badly hampered.

Exactly when to engage students in practice, through what method, and for what duration are educational decisions that teachers will need to make on a regular basis. But, that students will only remember what they have extensively practiced--and that they will only remember for the long term that which they have practiced in a sustained way over many years--are realities that can’t be bypassed.

One of the problems with modern education is that little effort is made to distribute practice such that students have a fighting chance to retain the knowledge they have learned. Google is great tool, but it is not a substitute for a deep well of structured long-term knowledge.

Question 2: Why don't students retain what they've learned

Roger Sweeny asks:

During the year, students had "learned" various things. How do I know they learned them? Because they correctly answered questions on tests or used the appropriate formulas and ideas to solve problems (I teach physics and physical science).

But when they take the final, they show that they don't know lots of the things that they seemed to know earlier in the year.

So maybe it's incorrect to say that they actually "learned" those things. Maybe it's more accurate to say that they "memorized" various things and then forgot them.

Two questions: One, does anyone doing education research care about this? My impression is that it's a gigantic elephant in the room that no one wants to talk about. And yet teachers know it's there. Every teacher knows that if she gave the same test a month later, her students would do considerably worse.

The short answer is that learning something, even to mastery, is not sufficient. Merely mastering something does not guard it against the ravages of forgetfulness. In order to retain what you've learned you need to overlearn it past the point of mastery. Schools don't do this.

For the cognitive science behind all this read these three articles from Dan Willingham:

1. Students Remember...What They Think About

2. Why Students Think They Understand—When They Don’t

3. Practice Makes Perfect--But Only If You Practice Beyond the Point of Perfection

Roger continues:

So question two: Does anyone doing educational research use long-term learning as a measure of success? When researchers try to determine if various things "work," how many test for knowledge 24 hours later? a week later? a month later? more?

The DI people do, though the standardized testing instruments don't capture the long-term effects if learning.

Read Engelmann's Student-Program Alignment and Teaching to Mastery for one way how to truly teach to mastery with retention. Another way to do it is through Precision Teaching, as another commenter indicated.

The easiest way to learn how to teach to mastery is to take a look at one of the DI programs. I'd suggest getting a mid-level reading or math program from eBay (you must get the presentatio