M.I.T Drops Physics Lectures; NY Times Confused

The NY Times "reports" that M.I.T. has dropped its Introductory Physics lectures in favor of "smaller classes that emphasize hands-on, interactive, collaborative learning." Dropping the lecture is probably a good thing; but, the Times article makes little sense. Probably because the reporter, Sara Rimer, doesn't seem to understand the nature of science instruction and doesn't provide enough information for the reader to understand what's going on here.

Introductory Physics is a typically a four credit class. Actually, two semesters of four credits. This means that there's four hours of classroom instruction provided each semester. Typically, these courses are followed by (at least) a two credit lab course in which students take what they've learned and conduct highly scripted Physics experiments.

Here's how Introductory Physics is traditionally presented in a typical school week.

1. Student reads the next section in the textbook.
2. Professor explains the section, provides his insights, and works some problems from the section. (one hour)
3. Student attempts to solve the assigned problems from the section, either alone or in a study group.
4. Grad Students work with students in small groups reviewing the problems to ensure the student understands the material. (one hour)
5. Repeat steps 1-4 for next section. (two hours)

That's how it's supposed to work. In practice, it usually goes a little differently.

1. Student fails to read next section or doesn't understand next section.
2. Professor reviews exactly what is written in the textbook without providing any insight that the student could have gotten by reading the book on his own.
3. Student fails to do some, many, or all of the assigned problems either through lack of understanding or laziness.
4. Grad student reviews problems and student copies answers.
5. Repeat.

Note that the student is supposed to be spending at least eight hours a week studying and solving problems outside of the classroom. Some students front-load the work and do their work before the material was presented and the problems are reviewed like in the first example. Other students back-load the work and do their work after the material was initially presented and the problems solved like in my second example. Most students, however, fall out somewhere in the middle.

This is basic direct instruction using worked problem examples. Research shows that it is an effective way to teach novice students, and, by definition, students taking an introductory Physics class are novice students. These students have a long way to go before they are experts in Physics. Using worked problem examples is less effective with experts and non-novices with considerable domain knowledge. The difference is domain knowledge. The experts have it; the novices do not, at least not yet.

In fact, they won't have it the following semester either when they take Physics lab. Undergraduate Physics lab is closer to baking a cake from scratch than it is to real science. It is a highly scripted affair because the students do not yet know enough physics or how to conduct a real experiment on their own yet. They are novice scientists and the lab provides another opportunity to follow worked problems. In this case, the worked problems are the scripted experiments.

I think now we have enough background knowledge to make some sense out of the Times article.

For as long as anyone can remember, introductory physics at the Massachusetts Institute of Technology was taught in a vast windowless amphitheater known by its number, 26-100

...

The physics department has replaced the traditional large introductory lecture with smaller classes that emphasize hands-on, interactive, collaborative learning. Last fall, after years of experimentation and debate and resistance from students, who initially petitioned against it, the department made the change permanent. Already, attendance is up and the failure rate has dropped by more than 50 percent..

Right off the bat, I find it hard to believe that all four hours classroom time in Introductory Physics are present in a large ampitheater. Are there any science/engineering majors (especially those attending M.I.T) out there that that were taught like this. Undergraduate physics instruction is all about learning how to solve basic physics problems. BY necessity this will involve the student working hundreds of problems over the course of the semester on his own or with a study group. There's no getting around that fact. Even the most direct instruction of courses requires that the student work the problems on his own following an introduction by an expert (the professor) and concluding with a review of the problems with an expert (a grad student or the professor). This requires motivation on the part of the student. And that appears to be a problem at M.I.T, as you'll soon see.


Also note the petitioning of the students adn the failure of the Times to get to the bottom of that. We'll get to that later.

The traditional 50-minute lecture was geared more toward physics majors, said Eric Mazur, a physicist at Harvard who is a pioneer of the new approach, and whose work has influenced the change at M.I.T.

“The people who wanted to understand,” Professor Mazur said, “had the discipline, the urge, to sit down afterwards and say, ‘Let me figure this out.’ ” But for the majority, he said, a different approach is needed.

I think Professor Mazur is delusional. Physics forms a critical foundation for most of the students learning a hard science or engineering. Subsequent courses will build off of what is learned in introductory physics and the physics problems will be revisited and expanded upon often in subsequent years. So, a student who does not possess the urge to put in the hard necessary to learn physics is in for a rude awakening sophomore year. The years of coddling in high school are over; now is the time for real work.

“Just as you can’t become a marathon runner by watching marathons on TV,” Professor Mazur said, “likewise for science, you have to go through the thought processes of doing science and not just watch your instructor do it.”

That's stating the obvious now isn't it. And I find it hard to believe that M.I.T students were merely watching their instructor solve problems for four hours every week in a large amphitheater and not actually solving their own problems. I'm sure somewhere along the line students were being assigned problems to work out of class and that some time in-class was spent reviewing those problems and their solutions. Are we to believe that only the Physics majors were doing their homework?

Then we have this non-sequitur.

In an article in the education journal Change last year, Dr. Wieman noted that the human brain “can hold a maximum of about seven different items in its short-term working memory and can process no more than about four ideas at once.”

“But the number of new items that students are expected to remember and process in the typical hourlong science lecture is vastly greater,” he continued. “So we should not be surprised to find that students are able to take away only a small fraction of what is presented to them in that format.”

What does this have to do with anything related to this article. The magic number 7 is a problem under both the old way and the new way at M.I.T. Either way, the students are learning more than they can absorb. That's why they take notes and write stuff down. Students really bump up against the short term memory problem when they try to solve the problems until they have learned the underlying material. The new way of teaching doesn't fix that problem. The only thing that fixes that is lots of practice solving problems. Are the students getting more practice under the new system? Let's see.

At M.I.T., two introductory courses are still required — classical mechanics and electromagnetism — but today they meet in high-tech classrooms, where about 80 students sit at 13 round tables equipped with networked computers.

Instead of blackboards, the walls are covered with white boards and huge display screens.

Why are journalists such suckers for bright lights and fancy gizmos? I've yet to see any of this technology used in a way that is pedagogically superior to a blackboard and a slide projector.

Circulating with a team of teaching assistants, the professor makes brief presentations of general principles and engages the students as they work out related concepts in small groups.

Teachers and students conduct experiments together. The room buzzes. Conferring with tablemates, calling out questions and jumping up to write formulas on the white boards are all encouraged

This is the money graf. Here's where we find out that M.I.T hasn't really done away with the lecture, they've just shuffled the chairs. Instead of Lecture for an hour in a classroom and then solve problems for an hour in small groups with grad students, M.I.T. now has the professor lecture for a short period of time then the students solve problems for a short period of time with the help of the professor and grad students in the same room, repeat until the class is done. What's the difference?

I don't see the advantage, except maybe that the lazy students are being forced to do the work under the watchful eyes of the instructors instead of copying the problems they should have worked out before a later problem solving period. But, since they're working in groups now, there's no guarantee that they're not free-riding off of their neighbors instead of free-riding off of the grad student in the separate recitation period.

What the article describes is exactly what was going on in our problem solving classes with our grad student after our lecture. The only change is that we had blackboards.

M.I.T hasn't done away with the lecture; they've merely rearranged it in a way that is no more sound in a cognitive science sense than it was before. What M.I.T. is doing is providing another year of coddling. It's also still direct instruction. (Though I can't wait to see how Stephen Downes is going to try to spin it.)

And, the students aren't experimenting, they are solving problems. There is a big difference.

“There was a long tradition that what it meant to teach was to give a really well-prepared lecture,” said Peter Dourmashkin, a senior lecturer in physics at M.I.T. and a strong proponent of the new method. “It was the students’ job to figure it out.”

Our professor gave some really good lectures and then he ran one of the problem solving sessions. Is there a difference?

Apparently the problem is really an attendance problem.

John Belcher, a space physicist who arrived at M.I.T. 38 years ago and was instrumental in introducing the new teaching method nine years ago, was considered an outstanding lecturer. He won M.I.T.’s top teaching award and rave reviews from students. And yet, as each semester progressed, attendance in his introductory physics courses fell to 50 percent, as it did, he said, for nearly all of his colleagues.

“M.I.T. students are very busy,” Professor Belcher said. “They see the lecture as dispensable, that is that they can get it out of a book more efficiently than getting up, getting dressed and going to lecture.”

After three years, Professor Belcher had had enough. “I had poor attendance, and was failing 10 to 15 percent, and grading the tests and shaking my head in despair about how little was getting across,” he said. “And this is a subject — electromagnetism — that I love.”

Here's the thing. The problem solving sessions are critical to success. The lectures less so if the textbook presents the material well and the student reads it beforehand. This is especially so if the professor is a bad teacher. Under the new system the students are forced to endure the gas-bag "initial presentation,"i.e., mini-lectures, to get to the problem solving part.

Maybe that's why the students are petitioning. The silly lectures are no longer optional for the student. That's why attendance is up. Here's another reason why attendance is up:

Unlike in the lectures, attendance counts toward the final grade, and attendance is up to about 80 percent.

I suppose the clickers don't hurt.

“One of the newer professors, Gabriella Sciolla, who arrived in 2003, was teaching a TEAL class on circuits recently. She gauged the level of understanding in the room by throwing out a series of multiple-choice questions. The students “voted” with their wireless “personal response clickers” — the clickers are essential to TEAL — which transmitted the answers to a computer monitored by the professor and her assistants.

You know where they are,” Professor Sciolla said afterward. She can then adjust, slowing down or engaging students in guided discussions of their answers, as needed.

Lecturing in 26-100, she said, she could only look out at the sea of faces and hope the students were getting it.

Unless they had clickers because if they had clickers in the lecture hall, the professor would get the same feedback.

What I see here is a distinction without a difference. The learning is no more active then it was under the old system.

What is left unexplored by the Times is why there was protesting by the students. Students are no fans of boring lectures. And I'm sure under the old system there was plenty of in-class problem solving and opportunity for feedback. Under the old system students were supposedly left on their own to solve difficult physics problems. Now they get to do the same thing in a high-tech classroom with all their classmates and lots of teaching assistants milling around. So why the protests?

I suspect once we learn why, we'll get a better idea of the problems of the new system at M.I.T.

Teaching Content is Teaching Reading

Dan Willigham has a new video up on the importance of content instruction in reading and comprehension. Willingham and Don Hirsch have making these same points for some time now. But the video makes this information more accessible.




BTW, Willingham also has a new article on memory in the new American Educator which you should also read.

And, would you believe that I actually beat the inestimable Core Knowledge Blog in posting the video. Take that Pondiscio.

Spelling tackled in American Educator

Following my lead, American Educator has a good article on Spelling and how to teach it.

Not unsurprisingly, popular spelling instruction practices are based on flimsy pseudo-science:

One common perception we have encountered is that visual memory, analogous to taking a mental picture of the word, is the basis of spelling skill. Teachers often tell us that they teach spelling by encouraging whole-word memorization (e.g., using flashcards and having students write words 5 or 10 times) or by asking students to close their eyes and imagine words. We’ve encountered this perception that spelling relies on visual memory so many times that we became curious about when and how it originated—after all, it’s a far cry from Webster’s spellers. We traced it back to the 1920s: one of the earliest studies to stress the role of visual memory in spelling was published in 1926, and it found that deaf children spelled relatively well compared with normal children of similar reading experience.4 Based on this study, and the perception that the relationship between sounds and the letters that spell them is highly variable, many people concluded that learning to spell is essentially a matter of rote memorization. Thus, researchers recommended that spelling instruction emphasize the development of visual memory for whole words

Right, let's have them visually memorize whole words because there couldn't possibly be any other helpful information they could use. Apparently not; its a common belief that English is a highly irregular language. The article lays that trope to rest.

This is a question we hear often. If English spelling were completely arbitrary, one could argue that visual memorization would be the only option. However, spelling is not arbitrary. Researchers have estimated that the spellings of nearly 50 percent of English words are predictable based on sound-letter correspondences that can be taught (e.g., the spellings of the /k/ sound in back, cook, and tract are predictable to those who have learned the rules). And another 34 percent of words are predictable except for one sound (e.g., knit, boat, and two). If other information such as word origin and word meaning are considered, only 4 percent of English words are truly irregular and, as a result, may have to be learned visually (e.g., by using flashcards or by writing the words many times).

Far from being irregular and illogical, to the well-known linguists Noam Chomsky and Morris Halle, English is a “near optimal system for lexical representation.

...

There are three types of information that, once learned, make spelling much more predictable: (1) word origin and history, (2) syllable patterns and meaningful parts of words, and (3) letter patterns.”

It doesn't take a rocket scientist to figure this out. Research shows that children misspell irregular words more often than regular words. That should have been a good indication that visual memorization might not have been the best way to go.

The other thing is. Wouldn't using word origin and history, spelling patterns and meaningful parts of words, and letter patterns to spell words involve using and practicing critical thinking skills -- dare I say 21st Century skills, rather than brute memorization? Just sayin'.

The article is a good read. The only weak part is when the authors make some untested recommendations as to how they think spelling should be taught. At best, these recommendations are representative examples of what might possible be good practice once someone takes to time to develop an test a suitable instructional sequence. But, that work has not yet been done and the authors are a wee bit overconfident that their recommendations will be effective.

21st Century Skills

Jay Mathews has a good editorial on the inanity of the latest education fad, teaching 21st Century skills.

Granted, the 21st-century skills idea has important business and political advocates, including President-elect Barack Obama. It calls for students to learn to think and work creatively and collaboratively. There is nothing wrong with that. Young Plato and his classmates did the same thing in ancient Greece. But I see little guidance for classroom teachers in 21st-century skills materials. How are millions of students still struggling to acquire 19th-century skills in reading, writing and math supposed to learn this stuff?

There are ways, some teachers tell me. Tim Burgess, a physics and chemistry teacher in Alabama, said he tried coaxing students to think for themselves. He laid out clues and let students sort them out together -- and it worked. "Suddenly, it became clear how 21st-century thinking was far more important than the mounds of content we were expected to force-feed our victims (I mean students)," Burgess said.

The 21st Century skills movement is nothing more than an excuse for continuing not to teach content under the mistaken belief that if you teach students how to think (i.e., how to learn how to learn), content becomes irrelevant internet access.

Unfortunately that's not the way it works. Critical thinking skills are domain specific. If you want to think critically about the American Civil War you unfortunately need to know a lot of stuff about American history, European History, military history, the American Civil Wat itself, and lots of other bring stuff like that.

This doesn't necessarily mean that students need to spend lots of time memorizing minutiae, but they at least know enough general knowledge to be able to pass those silly internet tests that embarrassingly show that today's (and yesterday's) students don't, in fact, know this stuff. There must be some sort of mental framework in place for Google or Wikipedia to be useful.

Instantaneous access to information doesn't guarantee that one will know what to do with the information after it's located.

Although, I think there is one useful 21st Century skill that students should be taught: how to set the time on their VCR's to lose that technological incompetence badge of shame: 12:00.

Or maybe not.

(That lame ending joke is actually a good example of what I'm talking about. It depended upon my knowing a few pieces of minutiae: 1. That there is an annoying blinking HTML tag (something I've known for some time) and that VCRs are no longer being manufactured (something I learned last week). And my being able to quickly retrieve those facts in real time as an example of obsolete skills, the importance of knowing facts, and critical thinking (the ability to synthesize those facts to make the joke) to end the post. The other point is that a good comedian should never have to explain his jokes. I leave it up to you to deconstruct that one in the comments.)

Update 1: Apparently, IE doesn't properly display the BLINK HTML tag. That's probably a good thing. Use your imagination.

Update 2: Willingham beat me to the punch. "But these 21st-century skills require deep understanding of subject matter, a fact that these reports acknowledge, albeit briefly. As I have emphasized elsewhere, gaining a deep understanding is, not surprisingly, hard. Shallow understanding requires knowing some facts. Deep understanding requires knowing the facts AND knowing how they fit together, seeing the whole. It’s simply harder. And skills like “analysis” and “critical thinking” are tied to content; you analyze history differently than you analyze literature, a point I’ve emphasized here. If you don’t think that most of our students are gaining very deep knowledge of core subjects—and you shouldn’t—then there is not much point in calling for more emphasis on analysis and critical thinking unless you take the content problem seriously. You can’t have one without the other."

Update 3:

Bamboozling the Gifted

One of the reasons I've been neglecting the blog is that I've been forced to learn Pennsylvania's rules for gifted education.

Here's how gifted education is supposed to work according to the statute:

1. Student is identified as being gifted, i.e., an IQ of two standard deviations above the mean (with some leeway which allows schools to fudge the results a bit for students just missing the cutoff).

2. The gifted student's present level of educational performance is then determined to see where the student is academically. For example, a third grade student might be reading on a fifth grade level and doing math on a fourth grade level.

3. Then the student's instruction is supposed to be specially designed, i.e., individualized, to meet the needs of the student.

4. Annual goals (what the student is supposed to learn this year) and short term learning objectives (the steps the students is to take to learn the goals) are then developed.

5. And the whole plan is memorialized in a written document (GIEP) which must be approved by the student's parents.

That's how things are supposed to work in theory. In actuality, things typically work a little differently. Here's how it works in practice in most school districts:

1. Student is identified as being gifted.

2. School district recommends that the student particpate in its gifted pull-out program which typical entails "enrichment" not acceleration.

3. Student receives some "differentiated" school work (i.e., semi-random worksheets) in class (because the courts have determined that a gifted pull-out program is not sufficient by itself).

4. Fuzzy goals and learning outcomes are listed in the student's educational plan which are typically subjective, unquantifiable, and/or untestable.

5. Plan is presented to student's parents for approval without informing them that the district's recommendation is merely a preference and that other options are available tp the student.


I'd characterize this as the school's way of discharging the regulatory burdens of providing gifted education with the minimal amount of work and the minimal amount of additional academic expectations. Instead of the student's needs being paramount as intended by the law; the disctrict's administrative convenience is paramount.

As a parent of a regular education student you basically have no say in how your child is educated in the public school system. You don't agree with the school's choice of fashionable curriculum? Too bad; move to a new a new school district. But once your child is identified as gifted (or "special" at the other extreme) they become statutorily protected. Now the parent does have a say. But unfortunately, most parents willingly (if perhaps unwittingly) sign away this right as soon as they accept the district's recommendation which is, as I described above, designed to specifically appear to be doing something for the student without doing much of anything or being responsible for doing or accomplishing much of anything.

The school's favorite way of accomplishing this goal is to specify academic "enrichment" for the student. So, what is enrichment? It's one of those education weasel words. It could mean almost anything. But I think my definition of enrichment is a good functional definition

Enrichment is not acceleration.

That cuts right to the chase. If the student is receiving enrichment, he's not receiving acceleration. He might be learning more, but that "more" being learned isn't the stuff needed to make it to the next level.

Let's say the gufted student is capable of learning 50% faster than the regular education instructional pace. This means that in two years the student is capable of learning three years of academic content. If the student was in third grade and was being accelerated, he'd be ready to tackle sixth grade level work by the end of fourth grade (2 years). However, if the student were being enriched, he would likely only be prepared to do fifth grade work at the end of fourth grade.

Maybe an illustration would help.

The first three light blue ovals represent how much the regular student needs to learn. The light green circles represent how much our hypothetical gifted student learns in a given year (150%) in an enrichment program. The gifted student is clearly learning a lot more than the regular student for the three years of grades 3-5 depicted. At the end of the those three years, however, the student still is only prepared to do sixth grade work.

Let's contrast this with an acceleration program.


The student has learned the same amount of material, but the learning is focused in the direction of what the student needs to know to progress through the grades. The result is that after the same three years of learning, the accelerated student is ready to do work at grade 7.5 instead of grade 6 as in the enrichment example above.

Acceleration seems, at least to me, to be the preferred course of action for the gifted student. School districts, however, don't see it this way. The vast majority of schools only want to offer enrichment pull-out programs for their gifted students. Why do you suppose this is so?

I think that the reason is that there's increased accountability in accelerating the gifted student. In my example, the gifted student should be ready to do sixth grade level work by the end of 2 years instead of three. If the student isn't ready then something has gone wrong and the student hasn't learned what he was supposed to. Someone is going to be blamed and who wants that aggravation, especially considering these are the kids who should be coasting through the system and Taking up less of the teacher's time, allowing her to focus on the other kids.

The other reason is that acceleration programs present administrative challenges for the school since these gifted kids will have to be separately tracked ot perhaps taught in a different grade for some subjects.

Nonetheless, the statute clearly places the student's needs above the administrative problems of the schools, so this last factor shouldn't be an issue in theory. In practice, you know it is. This is a monopoly we're dealing with and monopolies don't care about their customers -- where else are they going to go? And who cares anyway, the same amount of tax dollars are still going to flow into the coffers every year.

Today's Quote

It's time to admit that public education operates like a planned economy, a bureaucratic system in which everybody's role is spelled out in advance and there are few incentives for innovation and productivity. It's no surprise that our school system doesn't improve: it's more resembles the communist economy than our own market economy.

-Al Shanker, President AFT

Efficiency and Spelling

It's no secret that I'm not a fan of constructivist and child-centered teaching practices.

One of the main reasons why I don't like these practices is that they are even less efficient than traditional teaching practices. And traditional practices aren't very efficient either. In fact they are downright primitive compared to what we know about how children learn.

Let's take the teaching of spelling as one of the worst offenders.

Spelling continues to be taught, when it is taught at all, as it has been for decades. Students are given a list of words (10-15) on Monday and then tested on Friday to see if the words were learned. Then a new list of words is given and the process repeats. What happens to the old list of words? They disappear forever.

More formally, a week of massed practice is followed up with zero distributed practice. Not unpredictably, the students quickly forget what they've learned. All that effort is wasted. Retention is left to happenstance. Maybe the student will use the word in his writing before the spelling is forgotten. Maybe he won't. Maybe she'll read the word in her reading and think about the spelling, maybe she won't.

This is not an efficient way to learn spelling. It is a waste of time. Unless the student happens to be one of those smart kids that learns easily, reads voraciously, writes prolifically, and has exceptional retention. Inefficient teaching methods handicap those that aren't smart.

Further, it seems that the preferred way to teach spelling is through brute memorization. Often, the word lists do not capitalize on phonetic or morphographic efficiencies. Rote memorization appears to be the rule for learning spelling.

Then we have some of the inane exercises used to teach spelling. My favorite is "write a sentence for each spelling word." This often requires that the student is familiar with the meaning of word, familiar enough to use it coherently in a sentence. If the student doesn't know the word, it must be looked up in a dictionary. The hope is that the words used by the dictionary to define the word are understood by the student. Often they are not. This leads to more looking up until a definition the child understands has been found. At this point the child can formulate an understandable definition of the original word assuming all of this can be juggled in short term memory. Now the child is ready to make-up a sentence which requires creativity and knowing the rules of grammar, among other things. It's quite a lot for the student to attend to. We know that students remember what they think about, so you can bet that spelling only plays a minor role in this difficult exercise.

Who wants to defend the traditional way to teach spelling?

And who has a better way to teach spelling that addresses the problems I've discussed above?

Whose National Standards

Diane Ravitch is touting National Standards again. So is KIPP's Michael Feinberg.

I don't understand the love for standards, especially the national variety.

Imagine your ideological enemies being the ones in power drafting the standards. Now imagine that they, as they are wont to do, draft standards that not only favor their ideological brethren, but also might preclude you from practicing your favored ideological method. You can be certain they won't disfavor or handicap themselves.

Spend five minutes thinking about what you think are the best education outcomes and methods. Now spend another five minutes devising ways to disfavor those outcomes and methods. It's alarmingly easy to do.

Now tell me that you're still for a national standard that will apply to each and every state. They'll be no escape, unless you move to Canada. Or Mexico.

We need a diet

I'm a big plan of efficiency. So instead of analyzing all the bad education plans out there, I'm going to point out the shortcomings of the best -- Andy Rotherham's and Sara Mead's policy paper Changing The Game: The Federal Role in Supporting 21st Century Educational Innovation.

Here's the short version for the lazy:

Bad federal governmental intervention is the cause of much of our education woes, so we propose more federal intervention, but the good kind, i.e., the kind we like.

Now I like Andy and Sara. They are smart commentators on education policy. I am at least sympathetic, and often agree, with many of the views on education policy. But this time around Andy and Sara think that they can foster educational innovation and free-market-like solutions by putting the federal government's thumb on the scale and ignoring the reason why the free-market works in the first place.

Andy and Sara think they'll do a better job guiding the thumb than their equally smart predecessors. What they don't realize is that the thumb is the problem in the first place. This is a mistake that smart people tend to make. They think that they are smarter than the accumulated wisdom of the market. History shows they are not.

People, even smart people, are bad at making accurate predictions with respect to which innovations will succeed and which will fail. The recently deceased Michael Crichton makes a similar point with respect to finding solutions to the pollution problems facing people a hundred years ago.

Let's think back to people in 1900 in, say, New York. If they worried about people in 2000, what would they worry about? Probably: Where would people get enough horses? And what would they do about all the horseshit? Horse pollution was bad in 1900, think how much worse it would be a century later, with so many more people riding horses?

But of course, within a few years, nobody rode horses except for sport. And in 2000, France was getting 80% its power from an energy source that was unknown in 1900. Germany, Switzerland, Belgium and Japan were getting more than 30% from this source, unknown in 1900. Remember, people in 1900 didn't know what an atom was. They didn't know its structure. They also didn't know what a radio was, or an airport, or a movie, or a television, or a computer, or a cell phone, or a jet, an antibiotic, a rocket, a satellite, an MRI, ICU, IUD, IBM, IRA, ERA, EEG, EPA, IRS, DOD, PCP, HTML, internet. interferon, instant replay, remote sensing, remote control, speed dialing, gene therapy, gene splicing, genes, spot welding, heat-seeking, bipolar, prozac, leotards, lap dancing, email, tape recorder, CDs, airbags, plastic explosive, plastic, robots, cars, liposuction, transduction, superconduction, dish antennas, step aerobics, smoothies, twelve-step, ultrasound, nylon, rayon, teflon, fiber optics, carpal tunnel, laser surgery, laparoscopy, corneal transplant, kidney transplant, AIDS… None of this would have meant anything to a person in the year 1900. They wouldn't know what you are talking about.

Now. You tell me you can predict the world of 2100. Tell me it's even worth thinking about. Our models just carry the present into the future. They're bound to be wrong. Everybody who gives a moment's thought knows it.

So where does the free market come in? I'll let P. J. O'Rouke explain:

What will destroy our country and us is not the financial crisis but the fact that liberals think the free market is some kind of sect or cult, which conservatives have asked Americans to take on faith. That's not what the free market is. The free market is just a measurement, a device to tell us what people are willing to pay for any given thing at any given moment. The free market is a bathroom scale. You may hate what you see when you step on the scale. "Jeeze, 230 pounds!" But you can't pass a law making yourself weigh 185. Liberals think you can. And voters--all the voters, right up to the tippy-top corner office of Goldman Sachs--think so too.

With NCLB we finally bought the scale and made sure everyone weighed themselves. Many in education think that was a mistake and want us to throw out the scale. That's silly: how are we to know the diet works without a scale.

Others don't mind keeping the scale provided they can erase the objective markings and replace them with their own subjective ones. That's equally silly: you don't let the purveyors of the diet regime determine how to measure their own success.

And still others thought that merely weighing everyone and reporting their weights once a year would be sufficient to drop all those pounds. You still need a sensible diet in place for that to work. We didn't get many sensible diets. We got lots of excuses and test-prep diets: the kind of temporary diets that boxers do right before the weigh-in before a big fight.

What Andy and Sara want to do is legislate. i.e., fund, the "innovative" diets they think work best. That's only a small part of the problem. The bigger problem is getting the failed diets off the government teat and, unfortunately, that will include many of the diets Andy and Sara like. Andy and Sara's pseudo-free-market approach doesn't provide such a mechanism. And that is its fatal flaw. A real properly-functioning free-market works by ruthlessly eliminating the losers which involves a lot of short term pain, just like a real diet. That's the part that Andy and Sara leave out. Government won't defund, or starve, its losers voluntarily. That's not the nature of politics. And that's why political solutions, like Andy and Sara's, won't work.

Change

I've come back from my unannounced hiatus to discover that we have a brand new president.

A president that is for change. And, apparently, hope as well.

I "hope" that none of you wasted any time reading either candidate's platform. What politicians say they are going to do is very different from what they actually do once you've given them power. But you can rest assured that once elected their actions they will be consistent with them accruing power and ensuring that they retain power by getting re-elected. Keep that in mind because what you've just been promised (by both candidates) is inconsistent with their desire for power. Suffice it to say that you will be disappointed, and you would have been disappointed regardless of who was elected. That is the nature of politics.

Here is my prediction for education:

There will be change. That change will be superficial with respect to improving academic performance. It is extremely difficult to improve academic performance. The odds of academic performance improving in the next eight years in an educationally significant way are virtually nil.

It is easier to reduce academic performance by unwittingly changing things for the worse. This is because educating children is a difficult orchestration of detail that is difficult to get right and easy to screw-up. This remains true even though our current system remains horridly inefficient with much of the orchestration being badly out of tune.

Nonetheless the most likely scenario is that the change will produce no significant effect on outcomes. That is the history of education reform.

I wish my new president well but I don't have much hope that he is capable of improving education. He doesn't know how. And, as a result, he has no basis for selecting an education secretary that knows any better. Even an ideologically blind random selection is unlikely to produce better results because the field is replete with charlatans. Even if he were lucky enough to pick a winner, it is unlikely that that person could overcome the obstacles and vested interests in place that are anathema to improving academic performance.

We're going to get change. We always do. NCLB was change. But change doesn't guarantee improvement. Did you jump to that conclusion? I hope not. What you will get is something different, but that difference will likely not be an improvement.

There will be no shortage of wishful thinking and opinions of advisors. But since those opinions are almost certainly based on faulty science and informed by political correctness you should not necessarily expect beneficial results. Unless you're counting on luck. That's always a possibility. Even broken clocks are correct twice a day. Though, unfortunately, a clock that is five minutes slow is never correct.

That's what you're going to get -- an education secretary that is slow, broken, or both. Kind of like the current one.

So here's my prediction: the change you get in education will be different but not an improvement.

Let's hope that I am wrong. But don't count on it.

The WWC falls down on the job again

The What Works Clearinghouse (WWC) does a noble job of identifying much of the junk science research that plagues education research and masquerades as real research. The WWC, however, is not without its faults.
I have noted at least two instances in which the WWC has given its imprimatur to very questionable research.

In August, the WWC released a report on Reading Mastery-- one of the most researched reading programs in existence. Despite the fact that other reputable organizations have found that much of the Reading Mastery research base passes scientific muster, the WWC did not find a single study that met its standards. Clearly something was amiss.

The author of Reading Mastery, Zig Engelmann, has just weighed in on the WWC's latest shenanigans -- Machinations of What Works Clearinghouse. Basically Zig says that WWC failed to locate a large portion of the extant post 1985 Reading Mastery research base, improperly excluded the entirety (38 studies) of the pre-1985 research base, and used dubious criteria for excluding at least one study it did consider. I suggest you read the whole thing. I'll elaborate on two points that Zig raises.

Dubious Rationale for Excluding Pre 1985 Research

The WWC arbitrarily limits its research review to studies reported no earlier than 1985 (unless the WWC principal investigator deems the study important enough to report). This 1985 cut-off makes little sense. Beginning reading performance hasn't changed much since 1985. In fact, we have readily available evidence that it hasn't changed much since as early as 1971. That evidence is the NAEP Long-Term Trend in Reading test data (not to be confused with the plain ol' NAEP test which changes frequently). Here's a graph of the performance of nine year olds (4th grade):

As you can see, the performance of nine-year olds in reading has stayed remarkably flat during the period 1971 - 2004 with little difference between pre-1985 scores and post-1985 scores. My back of the envelope calculation is that the change between 1971 and 1999 is less than a quarter of a standard deviation, i.e., not educationally significant. In fact, scores in 1980 were higher across the board than they were in 1999. Only in the post-1999 do scores rise above the 1980 high-water mark.

Since we have reliable data going back to 1971 showing similar performance in early reading, there is no compelling reason to arbitrarily set the cut-off at 1985. The rationale the WWC offers is lame:

... the fact that preschool enrollment has increased, combined with the fact that more preschool and kindergarten programs run full-day, means that students in the early grades may be better prepared to receive reading instruction today than students 25 years ago. Moreover, it is possible that any changes in reading readiness over this period may not have been evenly distributed, since differences in reading ability by socioeconomic status and race are apparent at the kindergarten level . . . Any of these changes could have implications for the effectiveness of an intervention. If school readiness has increased, then an intervention that was effective 25 years ago may not be effective in more recent years. (p. 2, Appendix A)

Perhaps the WWC hasn't heard, but there isn't any evidence that preschool, full-day kindergarten, and Headstart provide any lasting effects that don't quickly fade out. In fact all of the potential causes given by the WWC (for none have been confirmed by research) must be superficial and superfluous to reading performance, since the NAEP data shows that none of them have had a significant effect on reading performance.

This is a somewhat embarrassing admission coming from the WWC what with its lofty evidentiary standards and all. I also suggest you read Zig's evisceration of this argument which concludes:

The assertion that the children are better prepared now and therefore what was effective 25 years ago might not be effective now is logically impossible. Lower performers make all the mistakes that higher performers make. They make additional mistakes that higher performers don’t make and their mistakes are more persistent, more difficult to correct. Therefore, if the program is easier for them now because of their higher degree of undefined ―readiness, they will make fewer mistakes and progress through the program sequence faster.

...

[B]eginning reading for grades K–3 is stable because nothing of significance has changed in the last 40 years. The instructional goal is the same—to teach children strategies and information that would permit them to read material that could be easily covered with a vocabulary of 4,000 words. The frequency of these words has not changed. The syntax of the language has not changed significantly. For these reasons, the content of the first four levels of Reading Mastery has not changed over the years.

I am not aware of any properly conducted scientific research which has a shelf life of only 20 years. Research doesn't go bad. I'm not going to stop taking penicillin based drugs while the research gets updated because the basic research was conducted 80 years ago. And, I see little reason for the WWC to exclude any properly conducted research on Reading Mastery, such as the Project Follow Through, or for any other educational program for that matter.

Dubious Confounding Factors

It's bad enough that the WWC failed to even locate, much less consider, a majority of the extant Reading Mastery research. It's even worse that they set an arbitrary cut-off date that excluded at least 38 studies on Reading Mastery. However, improperly excluding a study (which otherwise meets all the selection criteria) based on the fact that the new teachers were provided initial training goes beyond the pale.

The RITE study (Carlson and Francis, 2002) which involved 9300 students and 277 teachers (Zig claims that it is "probably the second largest instructional study ever conducted (after Project Follow Through") met all of the WWC exceedingly high selection criteria. However, the WWC excluded the study because "support [was] provided to teachers through the RITE program" which the WWC believes to be a confounding factor. Here's the confounding "support" the teachers received:

This support consisted of summer training, less than two hours of monitoring during the year, and help from a designated trainer. Nearly half of the teachers (137) were in their first year of teaching Reading Mastery. The training focused on how to provide positive reinforcement, how to correct specific errors, how to organize and manage the classroom so that one small group is in reading instruction while the other two groups are engaged in independent work and are not disrupting the instruction... The teachers were trained to teach Reading Mastery exactly the way the [Teacher's] Guide describes it, with all the technical details in place.

This is not only a ridiculous reason for excluding an otherwise acceptable study, but also against the WWC's own protocols which permits the inclusion of "commercial programs and products that [have] an external developer who: Provides technical assistance (e.g., provides instructions/guidance on the implementation of the intervention)." (p. 6, Protocol)

The WWC excluded many other otherwise acceptable Reading Mastery studies based on "confounding factors." I wonder how many were confounding factors related to initial training like the RITE study. I know that more than one study was excluded because the control group initially performed at least half a standard deviation above the Reading Mastery group, yet despite this advantage, the Reading Mastery group outperformed the control group by the end of the study. I'm thinking that the magnitude of the effect size more than compensates for the reliability issue caused by initial discrepancy which favored the control group.

In any event, there you have it. The WWC failing to do their job properly yet again. This is beginning to become a pattern.

Postal Service More Loved Than Public Schools

According to Lisa Snell:

An August 2008 poll conducted by Education Next and Harvard University finds that Americans think less of their schools than of their police departments and post offices. When asked to grade the post office, 70 percent of respondents gave an "A" or "B." In contrast, only 20 percent of Americans said public schools deserve an "A" or a "B." Twenty-six percent of the country actually gave their public schools a grade of "D" or "F." And African-Americans are even more down on public schools, 31 percent gave public schools a "D" or an "F."

I'm not surprised. The post office delivers my mail faithfully, albeit expensively and with a substandard tracking system, regardless of my social status, my ability to receive mail, or my mail receiving style.

Learning Styles Are Bunk

Dan Willingham has another video out on the non-existence of learning styles.



Willingham goes into much more detail on learning styles in his Summer 2005 American Educator article: Do Visual, Auditory, and Kinesthetic Learners Need Visual, Auditory, and Kinesthetic Instruction?

Vicki Snyder also makes a similar point in Myths and Misconceptions about Teaching: What Really Happens in the Classroom. Learning styles are presented as the fifth myth of teaching. (I reviewed the book here.):

Myth #5: the myth of learning styles refers to the popular idea that teaching methods should be matched to students' unique characteristics. Although individualization is desirable, learning style assumes that certain learner characteristics are intrinsic when they may in fact be the result of experiential factors that are amenable to instruction. As a result, teachers may inadvertently deny low-performing students opportunities to learn.

The myth of learning styles is based on three faulty premises: learning styles are intrinsic, learning styles can be assessed, learning styles can be matched to instructional styles. Snyder points out that all three premises are untrue.

In any event, as far as teaching goes, we only really care about the differences and similarities that influence learning and instruction. Of course, the vast majority of differences between children have little or nothing to do with how kids learn. Often these differences are expressed in terms of "learning styles and modalities," "multiple intelligences," and "differing interests." All of these so-called differences are similar in that none has any empirical support nor has any been shown to have an effect on learning or instruction.

This is because the content of instruction dictates about 90% of what has to be taught:

Content, and the nature of content, doesn't change according to the interests of children, nor according to any other characteristic of children. If we were trying to teach a gorilla to read, the nature of reading wouldn't change. Obviously, when it comes to the nature of content, differences among learners don't have much to do with anything.

Learning style differences are usually assessed informally through teacher observation. Teachers, however, often know little about inducing real learning. These learning styles are often expressed as superficial external traits like visual, auditory, tactile or kinesthetic which mask the underlying complex cognitive traits. For example, children cope with their inability to read in ways that might superficially seem like a learning style, but that actually reflect poor reading skills. It's easy to misinterpret certain behaviors.
Consider the following examples.

• Sometimes elementary teachers say that poor readers are auditory learners because they can't track words with their fingers. It's more likely that they can't read the words. Usually these auditory learners can keep their eyes riveted to a television or video game screen for hours.
  
• Sometimes elementary teachers say that poor readers are visual learners because they memorize and rely on picture clues rather than sounding out words. It's more likely that they revert to visual clues because they can't read the words. Without knowledge of the underlying sound structure of language, they have little choice but to rely on memorization and guessing.
  
• Sometimes high school teachers say that poor readers are auditory learners because they need the text read aloud or explained to them.
  Sometimes high school teachers say that poor readers are visual learners because they need pictures, graphics, and visual displays to explain the text to them.
  
• When students are labeled tactile/kinesthetic learners, they often need hands-on experience, group work, and activities to learn, not because of their learning style but because they need structure, assistance, and feedback on difficult or unfamiliar tasks.

In all of these examples, the source of the observed behavior is poor reading skills. To ignore the basic problem in no way benefits the students.

The point is that all kids (and humans) share some characteristics that are useful for learning, and, therefore, instruction has to accommodate those samenesses among learners, rather than the many differences among them. Learning styles and "intelligences" and student interests and modalities couldn't possibly have too much influence on learning, not when the nature of content doesn't vary among learners, and not when some of those things that make us all human are so central to learning.Efficient instructional programs make every effort to communicate the essential nature of content to all learners (because it is the same for all learners), and they make every effort to take full advantage of the ways all humans generalize more accurately and efficiently. What is the same about children is their innate capacity for language, to learn to read and think inductively and deductively. What is the same about all children is that they will learn if given appropriate instruction. They may learn at different rates and may need different amounts of structure and practice to master academic skills and concepts, but they can learn.

The IQ Conundrum for Broader, Bolder

Here are some charts from Gersten, R., Becker, W., Heiry, T., & White. (1984). Entry IQ and yearly academic growth in children in Direct Instruction programs: A longitudinal study of low SES children. Educational Evaluation and Policy Analysis, 6(2), 109-121. that show the gains made by the low-SES DI students in Project Follow Through for a range of IQ blocks from under 71 (2 sd below the mean) to above 130 (2 sd above the mean).

There are six IQ blocks shown on the chart. From left to right:

Block One: IQ below 71
Block Two: IQ between 71 and 90
Block Three: IQ between 91 and 100
Block Four: IQ between 101 and 110
Block Five: IQ between 111 and 130
Block Six: IQ above 130

For each IQ block the mean standard score has been graphed at the end of grades 1, 2, and 3.

There are arrows (<, <<, <<<) along the Y axis (mean standard score) that show the national median for each grade. I've (helpfully) drawn a blue line at the third grade national mean, as you can see, only the kids in blocks with IQs above 100 are performing above about the national median for math and only those above 110 for reading. (The blue line only has meaning with respect to the third grade scores (the top point). You could draw horizontal lines from the double arrow (second grade) and compare it to the middle point and from the single arrow and compare it to the bottom point.)

Click on each chart to enlarge.

This chart is for total reading for the Metropolitan Achievement Test.



This chart is for total math for the MAT.



Here is Becker's interpretation of the charts:

The data showed almost no contribution to "learning rate" (pretest to posttest gains) for IQ. If IQ were correlated with gains, lower-IQ children would make smaller gains and higher IQ children would make larger gains. This does not happen for Reading on the Wide Range Achievement Test (decoding) [Ed: Not shown.] or comprehension on the MAT, there is no IQ effect gains from the end of grade one to the end of grade two (most of the gains are about equal), but there is an effect for the gain from the end of grade two to the end of grade three. I believe this effect is due to the fact that the end of third grade test for Reading Comprehension on the Metropolitan uses an uncontrolled, adult-level vocabulary (as found in fourth grade texts). Since vocabulary instruction in school does not progress gradually to the adult level (but jumps from a carefully controlled vocabulary to an adult vocabulary after third grade), the test at this level is now measuring something not taught in school. Thus, students who score higher on a test of verbal skills (IQ) do better on a test of verbal skills (Reading Comprehension) when the content was not systematically taught in school. (A caution: The data may have imposed a ceiling effect on the brighter students; the program stressed preventing failures and thus teachers may have given more effort to teaching lower performers. Even if this is the case, however, the data are noteworthy in showing what can be done "gainwise" for lower-IQ children.)

Here is my observation. I understand Becker's comparable gains argument, but look at the mean percentile ranks for each IQ block:

Math End of Third Grade

Block One (IQ below 71): 24th
Block Two (IQ between 71 and 90): 39th
Block Three (IQ between 91 and 100): 47th
Block Four (IQ between 101 and 110): 61st
Block Five (IQ between 111 and 130): 69th
Block Six (IQ above 130): 88th

Reading End of Third Grade

Block One (IQ below 71): 11th
Block Two (IQ between 71 and 90): 29th
Block Three (IQ between 91 and 100): 34th
Block Four (IQ between 101 and 110): 44th
Block Five (IQ between 111 and 130): 58th
Block Six (IQ above 130): 81st

Also notice the gradual slippage from first to third grades in Reading even for the smartest kids. There is no slippage in math. Interesting.

I don't see how the lower IQ kids are going to be able to learn in a regular classroom given these percentiles. That would seem to foreclose a college education for these students and probably an academic high school education. Am I wrong?

And for the Broader, Bolder crowd, given that many low-SES students have lower IQs and that SES inerventions have not been able to to show a significant effect on IQ past about third grade, how exactly are your proposed SES interventions going to get around this IQ conundrum. Look the high-IQ, low-SES kids are performing well. The low-IQ ones aren't. I'd like to hear a rational argument that makes sense of this.

Day 17: Still Waiting

Two weeks after I first called for some evidence on the effectiveness of Broader, Bolder, I finally received a (sort-of) response from Big-Labor Fat-Cat Leo Casey.

Leo must have had a few of his underlings poring over the ERIC databases non-stop finding the requested evidence. Here is Leo's evidence. I am leaving in all the internal citations and footnotes.

Classroom teachers recognize immediately the educational value of providing a comprehensive array of services to students living in poverty. They have seen the effects of undiagnosed and untreated eye problems on a student’s ability to learn how to read, and of untreated ear infections on a student’s ability to hear what is being said in the classroom. They know that the lack of proper medical care heightens the severity of childhood illnesses and makes them last longer, leading to more absences from school for students who need every day of school they can get. They have seen asthma reach epidemic proportions among students living in poverty, and they know that the lack of preventive and prophylactic medical care leads to more frequent attacks of a more severe nature, and more absences from school. They understand that screening for lead poisoning happens least among children in poverty, even though their living conditions make them the most likely victims, with all of the negative effects on cognitive functions. They know that the stresses of life in poverty make mental health and social work services for students and their families all that more important, and yet they are least likely to receive them. They see how the transience that marks poverty disrupts the education of students again and again, as the families of students are constantly on the move. In short, teachers know that the students living in poverty lack the health and social services routinely available to middle class and upper class students, despite the fact that they need them even more. And they know that the absence of these services has a detrimental impact on the education, as well as the general well-being, of students living in poverty.

I emphasized Leo's evidentiary citations since they do not conform to the generally accepted norm. Leo's logic goes something like this: Leo knows best because Leo knows best. The circularity of this argument is surpassed only by its arrogance.

There is, of course, little actual research backing up Leo's claims. This is fortunate for Leo since in the few instances where there is research, it proves Leo wrong. Let's take a look at one of those claims.

They have seen asthma reach epidemic proportions among students living in poverty, and they know that the lack of preventive and prophylactic medical care leads to more frequent attacks of a more severe nature, and more absences from school.

As luck would have it, we actually have legitimate research on the efficacy of an asthma intervention. Here are the results.

• An asthma self-management program incorporating health education and parental involvement increased academic grades for low-income minority children but not standardized test scores. (Evans et al.)
  
  
• A subsequent study of the asthma self-management program was expanded to include health education for asthmatic children and their classmates, orientation for school principals and counselors, briefings for school custodians, school fairs including caretakers, and communication with clinicians demonstrated higher grades for science but not math or reading and fewer absences attributed to asthma as reported by parents but not fewer school-recorded absences. (Clark et al.)
  

Notice how the subjective measures (teachers' grades and parental reporting of grades) conflict with the objective measures (standardized test results and school-recorded absences).

Apparently, this isn't the sort of evidence that Leo is looking for. Leo isn't looking for any evidence:

Disingenuous calls for “evidence” that community schools work require a willful myopia on the effect on life in poverty on education — a blindness made possible by a complete unfamiliarity with the real world of the classroom.

If you ask Leo to provide support for his (expensive) opinions, you're being disingenuous. If you don't trust Leo that community schools work, you're being willfully myopic to poverty's effects on education. Of course, based on Leo's educational track record, if you're still foolish enough to be taking Leo at his word at this point, you'd have to be priapic.

I'll take disingenuous and myopic over priapic any day. I'm sufficiently hyperopic to know better than to take Leo at his word. Especially when that word calls for yet another bromide that gives more money and power to Leo.

Prediction Time

Following up on my last post on Charles Murray's new book, real education, it's time to see what Murray predicts will be the results from the grand experiment he proposes:

On measures involving interpersonal and intrapersonal ability. I expect statistically significant but substantively modest gains. On measures of actual knowledge, the experimental group will score dramatically higher than the members of the comparison group, perhaps 30-plus percentile points higher (technically more than a standard deviation). On measures of reading and math achievement, the differences will be no more than 15 to 20 percentile points (about half a standard deviation). Three years after the experiment ends, all of the differences will have shrunk. The differences in reading and math will be no more than 8 to 12 percentile points (no more than a third of a standard deviation) and may have disappeared altogether.

More formally, I predict that the magnitude of each academic effect will be a function of the g loading of the measure. Measures of retention of simple factual material have the lowest g loadings and will show the largest gains. For highly g-loaded measures such as reading comprehension and math, what has been accomplished by the last half-century of preschool and elementary school will be shown to be about as good as we can do, no matter how much money is spent.

This is a decent prediction. The I think that Murray overestimates the ease at which facts can be taught to and retained by low-IQ students and underestimates their ability with respect to math and reading comprehension.

Facts are difficult to learn because facts must be mostly learned on a case by case basis which is not readily amenable to acceleration. Math and reading (decoding and comprehension) are easier to teach because these skills, can be accelerated (even though teaching language and vocabulary remain problematic). But I knew that from the Follow through and the Baltimore Curriculum Project data. The data shows that we can get at least about three-quarters to a standard deviation improvement on average by the end of elementary school, better if we discount the schools that are so incompetent that they are unable to implement well-tested programs with fidelity.

Murray's point with respect to fade-out is well taken, but I'll leave that for another post.

Real Education: A Call for an Educational Experiment

I'm reading Charles Murray's latest book, Real Education: Four Simple Truths for Bringing America's Schools Back to Reality.

I agree with some of the points Murray makes and I disagree with others.

In any event Murray proposes a very good idea in Chapter 5:

Hence my second proposal, for a study that would be the most expensive educational demonstration project in history and would take as much as fifteen or twenty years from beginning to end. I state in the form of a challenge to everyone who is convinced that we can tach low-ability children far more than we are currently teaching them: Put up or shut up... Here is the proposal:

select children who test low in accademic ability but are not clinically retarded--say, children with measured IQs from 80 to 95, which demarcate the 10th to 37th percentiles. Make the number of the children in the study large enough that the results cannot be explained away as an accidents of small samples. Then provide these children with the best elementary education that anyone knows how to provide. Build new facilities or renovate existing ones. Hire the best teachers and create model curriculum. Measure how well the children are doing at the end of elementary school, and compare their progress with that of other children matched for IQ, family background, and whatever other variables are considered important.

...

The people who conduct the experiment should be free to use any teaching techniques, any class sizes, any amount of one-on-one tutoring, and type of technological aid. They shouldn't worry about making the program financially affordable for wider application, but instead bring to bear every resource that anyone can think of, at whatever cost that will maximize the education that these children acquire. Or to put it another way, their mission is to conduct the experiment in such a way, if it fails to produce success, there will be no excuses. Only three ground rules are nonnegotiable:

• The organization that selects the experimental and control samples and tests the children must be completely independent of and isolated from the organization that conducts the experiment.
  
• The design must protect against teaching to the test and test-practice effects.
  
• The design must include a test for fadeout, conducted three years after the experimental education ends.

Great idea. Sound familiar?

That's what I thought too. So I dashed off an email to Murray informing him that we'd already done something very similar thirty years ago: Project Follow Through.

Murray wrote back that he thought something was out there (even though people kept telling him there wasn't) and hoped that Real Education would surface it. Sure enough it had and I gave him a crash course on PFT.

In the post I'll tell you what Murray predicted would be the results of this grand experiment and we'll see how well his predictions matched the results of PFT.

SES and Rotten Instruction

Deep in a comment thread over at Sherman Dorn's blog, Dick Schutz makes an excellent point:

The only thing that the [standardized] tests are sensitive to is SES and racial/ethnic characteristic. If those two variables were partialled out statistically, the results would show that schools are pretty feckless instructionally.

That's not "news." It's been around since the Coleman Report of the 1960's. But the popular conclusion is that we have to "change society." The "obvious conclusion" has been overlooked--change instruction. When you do, you find that the correlation with accomplishments and SES is near-zero. That's empirical reality, not a statistical manipulation.

At least at the elementary level and if we don't include comprehension with uncontrolled vocabulary, and if we teach the higher-SES kids like we do currently, but his larger point is valid. SES matters quite a bit as long as instruction is rotten.

More Visual Aids

I have two more charts related to my earlier post on school expenditures and student performance.

For the first chart I calculated the differential between total student pass rate and the total pass rate predicted from the regression between percentage of economically disadvantaged students and pass rates. (Basically this crudely controls for amount of economic disadvantage in a school district.) Then I divided the pass rate differential by the standard deviation to arrive at a z-score so you can more clearly interpret student achievement. Then I plotted this z-score against the differential between the total expenditures for a school and the median total expenditures ($10711.5). (Does that make sense? Let me know.)



For the next chart I did the same thing. this time, though, I used the pass rate for economically disadvantaged students only since this is a better indicator of how well these districts do with at-risk students. The results are similar.




It should be clear from these charts that at these funding rates student achievement is not affected by school expenditures. There are plenty of schools that perform well with low expenditures and plenty that fail even with high expenditures.

Bronze: Where the Least Motivated Find the Will to Succeed

A blog reader, Carol Glenn, has come up with a new proposed after-school program that incorporates DI and Core Knowledge. Heer idea is in contention over at ideablob where she can win $10,000 in seed money if you vote for her. Check out her business plan and cast your vote.

Here's a good visual representation of Carol's plan:

| View | Upload your own

Today's Chart

Update: more visual aids here.

Arnold Kling suggests another way to present education data to determine if funding matters in education:

On the X-axis, plot the percentage of students in a county who are above the FARMS line (that is the "free and reduced meals" indicator of poverty). On the Y-axis, plot the percentage of students that pass the math exam. For each county in Maryland, put a data point on the chart. Next to each data point, put the County's ranking in terms of expenditure per pupil.

Next, draw the line of best fit through the data points. Counties that fall above the line are adding relatively more value than counties below the line. If education spending matters, then Montgomery County and other high-spending schools should be above the line. It would be interesting to see whether this is in fact the case.

Here is the chart for 497 school districts in Pennsylvania for 2005. On the X axis I have the percentage of students in the district that do not receive free and reduced meals. On the Y-axis I have the pass rate for PA's PSSA 11th grade exam (math and reading). There were too many school districts to add the funding data but I did do the calculations.



For school districts falling below the regression line the average total expenditures was $11,417.

For school districts falling above the regression line the average total expenditures was $11,214.

The overperforming schools actually spent less on average. Go figure.

Back in March I ran few different regressions on expenditures and FARM perfromance, household incomes, teacher salaries and parental education. The results are not always what you'd expect.

Update: Brett from DeHavilland blog has the numbers from Tennessee. Brett writes, "After looking at the correlation between TCAP and poverty rates, we looked at correlation between free/reduced lunch rates and value-added performance of the schools. Virtually no correlation to be found: in other words, some schools with 100% free/reduced lunch rates are contributing tremendously to student learning, and some with almost no free/reduced lunch participants are dropping the ball." Notice the variance (R²) is virtually the same as what I calculated for PA. (Note: Brett 's graph shows the percentage of FARM students not non-FARM students.)



Update II: Unbroken window runs the data for New York and finds the same relationship. Although, it appears that the some schools are maxing out the test and distorting the data.

Inquiry Physics

I was looking for something in my storeroom earlier today and stumbled upon my old College Physics textbook. Seven hundred pages of pure applied brutality that every science and engineering student had to complete.

Physics I was the course that sent a large portion of my freshman college class for greener pastures over at the business school. This happened despite the fact that almost every student had taken a high school physics course, so this was the second time through this material.

If my recollection serves me correctly, Physics instruction was supposed to go something like this. The student was supposed to read one or more sections of the textbook every week and attend a lecture given by the professor elucidating the sections we were to have read. A few dozen problems from those sections were assigned to us to work out. Then we attended three hours of recitation classes given by graduate students who worked through some of the problems we had been assigned to make sure we understood what was going on.

This brutal pace kept up for fourteen weeks and we covered nearly the entire textbook. During that time, we worked through hundreds and hundreds of problems. We were permitted to take into each exam one sheet of paper with whatever we could fit thereon. Otherwise, the exams were closed book. Nonetheless, the average grade for each exam was almost always less than 50%.

Here's my question.

How could one possibly teach an inquiry-based (problem-based learning) Physics course and possibly hope to get through more than say a quarter of the syllabus of a lecture-based course? I don't even see how this might work for a high school level course.

Update: Based on Stephen Downes' comment, I sense some confusion. I consider this to be a direct instruction/lecture based course not an inquiry based course. The pace is brutal for a lecture based course. I can't imagine covering this amount of content in a true inquiry based course.

Snake Oil is still Snake Oil even when its Broader and Bolder

I've been perusing the various Background Papers for the Broader, Bolder Initiative looking for some valid research pertaining to an actual implementation of one of the Broader, Bolder ideas. What I didn't find was:

Nevertheless, there is solid evidence that policies aimed directly at education-related social and economic disadvantages can improve school performance and student achievement. The persistent failure of policy makers to act on that evidence—in tandem with a school-improvement agenda—is a major reason why the association between social and economic disadvantage and low student achievement remains so strong.

What I find is a lot of observational studies that find various correlations between traits associated with at-risk children and their families and the fact that these children tend to perform worse than their mainstream peers. The causal jump is then assumed.

For example, studies have shown that at-risk kids report in questionnaires that they experience more hunger, which might be broadly defined to include everything from extreme malnutrition to missing a snack once a week, than their middle-class peers. Since the performance of at-risk kids is less than their mainstream peers, Broader, Bolder reasons that hunger causes distraction and distraction causes lower performance. And, therefore, we should provide more nutrition to at-risk kids.

But since there's no such thing as the nutrition fairy, this broader, bolder plan has to implemented somehow. For example, we might fund the public schools so that they might provide free and reduced lunches to qualifying at-risk students. Actually, we do that already. Then, how about if we fund breakfast programs for qualifying at-risk students. We do that too. I'm confused.

You see, the question isn't whether we should be providing more nutrition to at-risk students. That low-hanging fruit has already been picked. The question now is whether we should expand or supplement these existing programs and are educationally significant gains in student achievement to be forthcoming.

This is the question that should have been answered before Broader, Bolder issued their manifesto. But it wasn't. At best, we have some small scale research, usually rife with methodological flaws, that was conducted so that the researcher could provide evidence that their belief was correct. You can always find evidence that some kids learned something when you changed some condition. Round up enough experimental subjects and conduct enough experiments and you are bound to find some statistically significant, though not necessarily educationally significant, increase in performance whether by chance, good fortune, or hoax.

What we want is research in which the researcher started withe assumption that their idea was wrong. Then the researcher collects evidence that shows either that the researcher's beliefs are false or not false. This is called testing the null hypothesis. Kozloff gives us an example of this type of research:

I believe program X works, but I'm going to assume that it doesn't work and I'm going to collect data to try to show that it doesn't work. If the data do not show that X does not work, I will conclude that maybe it does work. Maybe.

We don't see this kind of research cited by Broader, Bolder. What we see is "research" that is attempting to persuade us that we should join the researcher in accepting his beliefs and that the researcher is not very interested in the possibility that he is wrong.

Broader, Bolder needs a healthy dosage of humility, especially since so many of its bromides remain untested.

(Picture adapted from Telling the Difference Between Baloney and Serious Claims About What Works, Kozloff and Madigan, DI News, Summer 2007)

Today's Best Education Paragraph

From Jay Greene:

Besides neither being unfunded nor a mandate, the argument that NCLB is an unfunded mandate is especially odd because it makes one wonder what all of the funding that schools received before NCLB was for. It’s as if the unfunded mandate crowd is saying: “The $10,000 per pupil we already get just pays for warehousing. If you actually want us to educate kids, that’ll cost ya extra.” Remember, that NCLB just asks states to establish and meet their own goals. Didn’t they have goals before NCLB?

Oh, we were supposed to educate them as well with that money?

Bogus Bowl V

Go take Teach Effectively's latest Bogus Bowl poll.

Which of the following do you consider to be the most bogus reason for failing to teach prospective teachers how to employ teaching procedures that have been documented to be effective?

• Professors want future teachers to find their own teaching styles.
• Professors don't want to stifle future teachers' creativity.
• Professors say that using research-based practice is only one small part of what future teachers need to know.
• Professors believe that there is not one best way to teach.