Parental Involvement in Low-Income Areas

Uncle Jay has a good column on how the need for parental involvement in low-income schools isn't all that critical.

The story of his school and others like it suggests that the importance of parental involvement, at least in the most disadvantaged neighborhoods, has been exaggerated, probably because middle-class commentators have been imposing their suburban experiences on very different situations. Unchallenged, this misunderstanding of what works for low-income children could stymie efforts to improve the country's worst schools.

The best school leaders say that they don't need much parental involvement when they are hiring staff, creating class schedules and putting discipline procedures in place.

...

Low-income parents may often be distracted just trying to make a living, but they know what works. Once they see a school keeping its promises, they provide the kind of support found in suburban schools. But it's important to remember that good schooling must come before parental support, not the other way around.

That seems about right to me. Parents need to make sure their kids get to school timely and regularly. But then the school needs to demonstrate that it is doing its job. Or why would parents think its important to send their kids to school every day?

Reboot III

In this post I'll lay out the final main points in my plan for school reform.

Lots of Data, Set Free

The most important aspect of the plan is data. Useful data must be made freely available to all parents and students so that parents can make sense of the various educational offerings. Parents must be able to answer the following questions:

• What is the predicted performance of my child in this education program based on the past performance of children, like my child, in this program?
  
  
• How much better (or worse) is the predicted performance of my child in education program A compared to educational programs B, C, ... , n.
  

The disaggregation of data made available has to go far beyond that provided under NCLB. For example, a male student with two Hispanic parents - one with a high school degree and the other with a college STEM degree, a family income of $ x, from neighborhood z, with various scores on prior tests taken should be able to drill down through the data and find out similar students like him and see how they fared in a particular course.

Such a system is predicated on accurate demographic data being collected, initial placement exams being taken when the student enters formal education, and all subsequent testing results.

Furthermore, the data should be made publicly available (while respecting FERPA regs) so that organizations, like S& P, can provide services for the public to use to evaluate the various educational offerings.

Maintain the Default System, Allowing Educators to Experiment with Their Top-Down Schemes

Under my plan, in which teachers take responsibility for student learning within the agreed upon obligations agreed upon by the students, there will remain some students that no teachers believe thay are capable of educating. These students must still be provided with a free appropriate education, so an educational system of last resort must be maintained. In all liklihood, states will maintain a scaled-down version of their present public education system for these children (and those parenst who favor the present system and wish to keep their children in it). These children will remain a political problem and states and the federal government will continue to employ top-down edicts and experiments in an attempt to solve the problem. They'll also continue to pump money into the system. None of it will work because the nature of the present precludes such a system working. Nonetheless, eventually some educators will be enticed by the per pupil funding available (the reward) and agree to the risks inherent in educating this kids.

Funding

I've left this one for last because I have no ecific ideas how to employ an equitable funding system. Basically, some kids are much easier to educate than others and, as such, require more effort to educate. This effort is likely to cost more and you'd think there should be additional compensation for those endeavoring to educate these kids.

There's also an argument that the present funding levels are more than adequate to educate most kids.

I don't have an answer for this problem. Nor does anyone else.

I do know how you might go about solving the problem. First you gather all the educators who think they know how to educate these kids. Then have all of these educators bid to take as many schools worth of children that they think they can handle. The losing educators can then be offered to participate in a small scale experiment and take a random sample of children at the winning bid price and see what they can do. The winning bidder will be paid only for the children they are capable of successfully educating. If the winning bidder (and any of the successful losing bidders) is sufficinetly successful, they should be provided any needed funding to scale up.

****

That's it. The minimal framework needed for a bottom-up evolution of the present system within a top-down regulatory framework which provides for a fair environment for professional educators to work and permits for top-down efforts for the difficult to educate children.

I'll discuss how a system might work in practice in my next post.

Reboot II

I noted in the previous post that the public education system needs a reboot to get a fresh start.

To use another metaphor, the public school system needs to be declared bankrupt so it can be restructured, eliminating all the conditions that drove it into bankruptcy in the first place.

The primary reason for the reboot, as opposed to less drastic measures, is to extinguish all the bad contracts, obligations, and relationships they've entangled themselves in and have had thrust upon them by political means.

As I pointed out in the previous post, after the great reboot of aught nine you'd have students, teachers, buildings, equipment/instructional material, and existing funding. All that is needed is to restructure the relationship between these existing elements so the incentives are better aligned while avoiding the mistakes of the present system.

So how should the system be rebuilt?

Power to the People

The first, and most important, fix is to return the power to the people. Education funding should go directly to the students to spend on education services as they see fit. One way to do this would be to set up education savings accounts for each student, like medical savings accounts, that can only be used for education services.

People pay taxes for education. A certain percentage of those taxes go to fund the education for the benefit of the public. But a person should a get a credit for a percentage of the taxes they pay each year that goes into their education savings account for at least the benefit of their own children's education. The idea is to provide parents (and students) control over their children's education commensurate with resources they've contributed into the system free from interference from the government or otherwise. To the extent that students are being subsidized from public funds, then it is likely that taxpayers, through their elected representatives, will want some control how those funds are spent. That seems to me to be a fair compromise.

Teachers Will Be Professionals

Teachers need to be professionalized, whether they like it or not, with all the benefits and responsibilities that flow from that status. We need professionals to do the hard work of education because we need educators to be responsible for student outcomes. Teachers should be like treated like doctors, lawyers, accountants, and other professionals. Doctors don't have to cure every patient and lawyers don't have to win every dispute, but they are required to render their services competently. If they fail to do so they risk having their license revoked and having to compensate for their malpractice. The upside is that as professionals, teachers will be free to render their services (within the guidelines established by the profession) in the manner of their choosing and free from the silly micro-regulations in effect today because educators have failed to police themselves like other professions.

The main benefit for teachers for professionalizing is that teachers themselves get to decide how they will organize themselves to offer their services. Any professional teacher can put out a shingle and offer services as a sole practitioner. Or than can partner up with one or more lawyers and offer services as a partnership. They can enter into more complicated organizational structures as they grow, much like modern day law firms and doctor practices.

Of course, since educators haven't yet developed their own code of professional responsibility (especially one that people trust), they're going to need some objective criteria for determining when services have been rendered adequately. This can be accomplished by ...

All Educational Services Will be Rendered on a Contractual Basis

Students and their teacher would have a contract setting forth the manner in which educational services will be rendered and criteria against which learning will be measured. Students will have obligations and so would teachers. Ideally, the system might work like this:

The teachers would determine her entrance requirements and placement criteria. The teacher should be able to determine if the student has the skills,knowledge, and other factors needed to succeed in the class. The teacher would also determine the obligations of the students accepted for instruction, such attendance criteria, homework criteria, and the like. Teachers are in the best position to determine whether a student is capable of succeeding with the instructional methods that will be employed and the teacher's assessment of her own skills. This is only fair since the teacher will be on the hook for educating the students she accepts. Bear in mind that the more stringent the teacher's requirements, the more difficult it will be for the teacher to attract students, so this process should find an efficient equilibrium point that satisfies both teachers and students.

teachers will also be required to spell out everything that will take place in the classroom and exactly what skills and knowledge will be taught and learned by the students -- what will be taught, how it will be taught, what curriculum will be used, how will the curriculum be supplemented, and the like. The student and her parents should be able to determine in advance exactly what will be taking place in the classroom and what will be learned.

Most importantly, the education contract will specify the class's exit criteria and whether it has been met by the student. The exit criteria should include the content specified by the State which will likely be a minimal skills test to assure the public that it has gotten the expected value from its investment. The exit criteria should also include any additional content, if any, that the teacher has promised to teach in the education contract.

Also, spelled out in the education contract should be the downstream teachers/programs that accept this teacher's final exam as fulfilling their entrance requirements. For example, the final arithmetic teacher would specify in her education contract which algebra teachers accept a passing grade in her class as fulfilling their algebra class prerequisites. This will encourage teachers to work together to develop their own standards that will carry students from the beginning of their formal education to the end, whether it be college or work.

As professionals, teachers would be responsible for assuring that all the students they accept learn everything they've promised to teach and pass the final exam. Otherwise, the teacher will be given a brief period (say two weeks) to cure the student's deficiencies through remediation. Failing to cure the student's deficiencies will result in the teaching forfeiting some or all of the funding she received to educate the student which will go to remediating the student.

Education Colleges Will No Longer Have a Monopoly on Teacher Preparation

Any person with an undergraduate degree should be able to teach if she is capable of passing the state's licensing exam and background check. This would include degrees from college's of education.

College's of education mostly teach pedagogy. But pedagogy should be determined by practicing teachers. Some teachers may find value in the pedagogy taught in Ed colleges in which case they should be free to hire Ed school graduates. Other teachers may feel that content knowledge is more valuable and that pedagogy is best taught by them according to their own philosophy in which case they should be free to hire graduates of their choosing.

Classrooms should be allocated to teachers who've attracted sufficient students in the district the school Building serves

Since the community owns the school buildings and the equipment therein, these resources should be offered to any teacher who has attracted sufficient students from the district to consume the eduction services they're offering.

Any licensed teacher (or group of teachers) should be able to offer educational services to students in any school district. For example, before every semester there might be an educational services "fair" in which all teachers interested in offering educational services in a particular school district advertise their offerings and attempt to attract students. Each classroom should be rated as to the minimum and maximum students it can hold. Once a teacher has attracted the minimum number of students in the district that teacher would be entitled to lease a classroom in that district for that semester. That teacher determines the maximum number of students she believes she can educate. The number of students successfully educated per semester determines the compensation that teacher receives. This provides an incentive for teachers to maximize the efficiency of the services they provide.

If students don't like the educational services offered in their home district, they are free to go to any other school district that has room. In this way, classroom space will be efficiently allocated. This also permits niche educational offerings to pull students from different districts.

I'm going to stop here for this post and let you chew this over. I haven't included everything that a well run public school system should have, such as funding allocation, but I think you have enough to see how my proposed system better aligns the incentives needed to improve education.

No doubt I've failed to include much and have failed to account for various factors. That's what the comments are for to point out my mistakes and help to improve on this basic framework. Or to argue that certain parts should simply be thrown out.

Do your worst.

I'll attempt to draft a post that includes the rest of the framework while responding to your critiques.

Blog Update

As some commenters may have noticed I've activated the CAPTCHA verification system and the comment moderation system for posts older than 30 days old. I had to do this to combat the influx of Chinese spam I've been receiving over the course of the last week. Last night alone I had to manually delete, one by one, about 200 comments that were spam. Let's see if this drives them away. if it does I'll remove the CAPTCHA system.

I'm currently trying to finish my reboot post. It's turning out to be a little more difficult than I originally thought.

I'll answer Brian's and Dick's comments after I've posted the conclusion of the reboot post.

Reboot

The administration's tentative rhetoric-heavy, action-light education policies aren't going to work.

Wishful thinking isn't going to make up the deficiencies.

The problem isn't necessarily that the Administration's policies are bad, though in this case most of them are. (The Bush Administration's policies weren't much better.)

And let's not forget that each state has the primary role in education anyway. That's federalism and it's usually a good thing. Mississippi's education needs are much different than Massachusetts'. Why should their education policies be the same? Moreover, no one has found the recipe for providing a good education for all students yet, so the need for experimentation remains. And, the more laboratories the better until one state finds a system that works and can be replicated.

To improve education, the Administration needs to take an important first step:

Admit defeat.

The current system simply doesn't work well for many children (and teachers). The incentives are all screwed up. There is not enough "choice" in the system, hence all the "wars." There is a "reading war" because some parents don't agree with the reading instruction services favored by some educators, yet have no choice when it comes to selecting those services. Educators should be able to choose what instructional services they offer and parents and students should be able to choose which educator's services they want to consume. There would be a market for both "progressive" education services and "traditional" education services.

Henry Ford used to offer his Model T automobile in any color the customer wanted provided they wanted black. Ford's competitors soon offered a choice of colors. Everybody was happy, except perhaps Henry. In education you're only happy if you like black. If you don't, you're at "war" with the system because you don't have a choice.

But I digress.

The first step to improve education has to be to admit defeat. We picked a bad system a century or so ago and it didn't deliver on its promises. We tried to educate the masses and we failed. We don't educate the masses; we educate the same small group of students that have always been easily educable. For the rest, we offer an expensive facsimile of education that fails to educate. We're good at claiming we've educated; not so good on delivering the education. We're good at shifting goal posts to make it appear that we're doing a good job, but few people are fooled.

Worse yet, the present system has attracted many powerful entrenched special interests whose best interest is to maintain the status quo. Those special interests are not students or teachers.

Instead of playing the blame game, it is more productive to say that the education incentives are not aligned. They need to be aligned for the system to work. Until they are aligned, no "reform" is going to work. There is lots of data proving this point.

To get the incentives aligned you need to take the right second step after you admit defeat. Admitting defeat provides the political will to take the second step. And, there's only one right second step. And that is to ...

Reboot.

The system needs to be rebooted.

We need a do over.

All the existing ties need to be severed. Then they can be rebuilt. The right way.

After the reboot what would have when the system comes back online?

• A bunch of students with varying education needs.
• A bunch of teachers with varying teaching abilities.
• A bunch of administrators and support personnel with varying administrative capabilities.
• A bunch of school buildings where education services and transportation services can be provided.
• A lot of instructional material and many publishers willing to supply almost any instructional need.
• Over $10k per student for operational expenses from existing tax revenue.
• An information superhighway capable of providing all the information services needed for a transparent network linking parents, students, taxpayers, and educators.

That's more than enough to get us started on Education System 2.0 - 21^st century edition.

Actually, you need to do one more thing. Throw out all the existing rules and regulations and institute a ten year moratorium on all education-specific rules, like we did with the Internet when it was opened up to consumers. When the reboot goes into effect, you want only a bare minimum of rules in place to bootstrap the new system.

Then its just a matter of realigning the incentives which is easy.

That'll be the next post.

New P21 Video Found Lacking

Is it me, or is the the new P21 Video lacking some 21st century skills or, at least, 21st century production values?



I mean it's not exactly a good use of technology or the video format.

Would you buy a suit from this guy.

Obama's Five Ponies of Education Speech

Today, President Obama promised us five ponies to cure our education woes.

It started off well enough:

My outstanding Secretary of Education Arne Duncan will use only one test when deciding what ideas to support with your precious tax dollars. It’s not whether an idea is liberal or conservative, but whether it works.

But then he goes on to offer us five ideas which largely don't work.

In other words, we won't be getting five ponies; we'll be getting at best one pony, three broken down mares, and a gelding.

And a large pile of horseshit for the next generation to clean-up.

Let's take a look at each of our ponies.

Pony One: Early childhood initiatives.

Obama sells this pony by pointing us to some non-educational benefits.

For every dollar we invest in these programs, we get nearly ten dollars back in reduced welfare rolls, fewer health costs, and less crime.

He has to do this because these early childhood initiatives tend not to have any educational benefits that don't wash out rapidly once the child enters the public school system.

He also offers what appears to be an preschool version of NCLB by offering grants to states that:

[d]evelop a cutting-edge plan to raise the quality of [their] early learning programs. Show us how you’ll work to ensure that children are better prepared for success by the time they enter kindergarten.

Letting states develop their own plans and then define what success looks like didn't work too well under NCLB.

This pony might have offered some hope if the funding went to actual preschool programs that directly remedied identifiable deficits that many low-SES children enter school with: low language skills.


Pony Two: Better standards and assessments.

Next, Obama wants to challenge states to enact better standards and assessments.

Why would they ever want to do a thing like that? It's much easier to enact low standards that appear to show that students are achieving, rather than actually getting them to achieve to high standards. Where is the incentive? And Obama isn't providing any new incentives in this speech. States will either answer the "challenge" or they won't. My money is on "they won't."

Obama is also promising funding for data systems "to track how much progress a student is making and where that student is struggling."

This is one of the few good ideas in the speech. Although, why states aren't doing this for themselves is beyond me.

Pony Three: Recruiting, preparing, and rewarding outstanding teachers.

Then Obama attacks teachers. Apparently, the ones we have now aren't any good. I disagree.

When the automakers ran their businesses into the ground and came begging for relief, the CEOs got the spanking from Congress, not the assembly line workers.

The problem isn't the teachers. The problem is with the management from top to bottom. They are the ones who made the silly rules, who created the impossible work environment, that entered into bad contracts wit the unions, and hired teachers from education schools that failed to prepare teachers with the skills they need.

And, let's be honest, you'd have to be a little crazy to want to work in many schools today given the conditions. I think most teachers simply don't realize what's in store for them. That's changing with the Internet.


Pony Four: Promoting innovation and excellence in America’s schools.

Obama promises to remove the current caps on charter schools. This is a good thing. No student should be forced to go to a failing school. There's no guarantee that a particular charter will be any good, but at least the student gets a choice and isn't forced to attend a bad school.

Obama also ants to increase the school day and year. That's great, students get to waste even more time in crappy schools. Let's focus on improving the schools first and then we'll worry about gaining further efficiencies by lengthening the time requirements.

He's also going to develop "new strategies to make sure at-risk students don’t give up on their education" by dropping out in high school. Here's a good way: improving their K-8 education. I thought that was what this whole speech was about in the first place.


Pony Five: Providing every American with a quality higher education ...

... by making it more expensive to go to college by floating more "grant" money into the system.

This one fails Econ 101. It also ignores the primary problem preventing students from attending college: they aren't sufficiently prepared for the rigors of college. And, there's precious little in this new plan that's going to help here. Let's work on that.

This plan is long on lofty rhetoric, but short on anything that stands a good chance of working, much less has an established track record. Basically, more of what we've come to expect from education from the Feds.

21st Century Skills, Not So new

21st Century skills aren't so 21st century. They've been peddled under various names for a long time.



Flawed in 1940; flawed in 2009.

Now with no more research base than they had back then. Yet educators are still "convinced" of their efficacy.

Of education and strawmen

Now this is funny.

Stephen Downes, in a post arguing that Joanne Jacobs has erected strawmen in her post on the 21st Century Skills debate, erects a few of his own.

Each of these three formulations serves to misrepresents actual positions held by educators that do stand up to scrutiny: that knowledge depends on critical thinking skills (not the other way around), that teachers address multiple cognitive needs (not just drill and recall), and that self-direction does not entail content-free learning.

Let's take each in turn.

Knowledge depends on critical thinking skills (not the other way around)

This is a demonstrably false statement.

Knowledge is defined as (i) expertise, and skills acquired by a person through experience or education; the theoretical or practical understanding of a subject, (ii) what is known in a particular field or in total; facts and information or (iii) awareness or familiarity gained by experience of a fact or situation.

Since knowledge includes facts and information, and facts and information can be acquired by rote (albeit inefficiently), knowledge does not depend on critical thinking skills.

The converse (Critical thinking skills depend on knowledge) is, however, true. You can't think critically unless you've learned a procedure for thinking critically (such as the critical reading procedure I laid out) without the knowledge of the steps (this would be information and facts) that make up that procedure.

That teachers address multiple cognitive needs (not just drill and recall)

This would be a strawman. Does anyone actually think that teachers should only address drill and recall?

That self-direction does not entail content-free learning

Another strawman. Does anyone actually believe that self-direction entails content-free learning? I suppose there is a risk that the self-chosen direction might not entail enough content, but this gets dangerously close to making a point that is educationally relevant.

The journalistic standards of the New York Times continue to decline

Critical thinking is easier when you know something about a subject. Take for example the following statement:

For decades, educators have warned against teaching children to read, they say — many children who are taught to read become struggling readers who hate to read — so teaching a child to read instead of allowing them to learn to read naturally can undermine the joy of reading for its own sake and can even lead to cheating.

Clearly this is a ridiculous statement (though it comes extremely close to the logic propounded by whole-language ideologues). And, it is easily identified as based on faulty logic. The problem is faulty reading instruction is bad, not all reading instruction. The statement can be easily remedied:

For decades, educators have warned against faulty reading instruction, they say — many children who are taught to read poorly become struggling readers who hate to read — so faulty reading instruction can undermine the joy of reading for its own sake and can even lead to cheating.

That's much better.

Let's try the exercise again with the lede paragraph of this Times article:

For decades, psychologists have warned against giving children prizes or money for their performance in school. “Extrinsic” rewards, they say — a stuffed animal for a 4-year-old who learns her alphabet, cash for a good report card in middle or high school — can undermine the joy of learning for its own sake and can even lead to cheating.

This statement suffers from the same deficiencies as the first statement, but it's more difficult to understand why unless you know something about basic principles of behavior. The problem isn't that all "extrinsic" rewards can backfire, just poorly designed extrinsic reward systems. Let's fix the statement up.

For decades, psychologists have warned against giving children prizes or money for their performance in school when the children are already motivated to learn. “Extrinsic” rewards, they say in this situation — a stuffed animal for a 4-year-old who learns her alphabet, cash for a good report card in middle or high school — can undermine the joy of learning for its own sake and can even lead to cheating.

The basic rule is to never use a stronger motivational system than you need to get the job done. If praise and grades will do the job then there's no need to implement a token reward system, such as cash or rewards for grades. If a token reward system is needed to motivate a child, then use the least invasive rewards that'll motivate the child to engage in the desired behavior (i.e., learning) and fade out the system as soon as possible. For example, trying rewarding with free time or other reinforcing activities before offering cash, candy, treats, or other tangible rewards.

This is not exactly controversial. About the only person who might disagree with such a statement is Alfie Kohn, but then again he's not a psychologist.

The Times then goes on to use the faulty lede to set up a false dichotomy with the view of "economists" and "businesspeople":

But many economists and businesspeople disagree, and their views often prevail in the educational marketplace. Reward programs that pay students are under way in many cities. In some places, students can bring home hundreds of dollars for, say, taking an Advanced Placement course and scoring well on the exam.

The Times has framed the debate poorly for the reader to understand the real issues and the possible dangers of these kinds of reward programs. And do you really trust the Times as it butchers the statement of its lead "he said - she said" expert:

Whether such efforts work or backfire “continues to be a raging debate,” said Barbara A. Marinak, an assistant professor of education at Penn State, who opposes using prizes as incentives.

I question whether Dr. Marinak would support the paraphrase as written. Seems like they dug awfully deep to find someone who supports the unqualified "extrinsic rewards likely to backfire" position. Especially when it's pretty easy to find an article co-authored by Dr. Marinak that supports the use of rewards for motivating children to read:

If your reading program uses incentives, consider using rewards that are proximal to reading. The importance of reading-related rewards may go beyond recognizing the relationship between reward proximity and the desired behavior. It could be that the real value of reading-related rewards is that both the desired behavior (reading) and the reward (books, self-selection, time) define a classroom culture that supports and nurtures the intrinsic motivation to read.

And the Times wonders why it is slowly going bankrupt.

Some kids excel regardless of the school

A new study from across the pond shows that bad schools are relative:

Middle-class parents obsessed with getting their children into the best schools may be wasting their time and money, academics say today.

They found that children from privileged backgrounds excelled when they were deliberately sent to inner-city comprehensives by parents opposed to private schooling.

Most of the children “performed brilliantly” at GCSE and A level and 15 per cent of those who went on to university took places at Oxford or Cambridge.

This shouldn't be surprising. Most schools teach roughly the same hodge-podge of topics and skills. Teacher skill varies, but it's generally suficient to induce learning in a typical middle-class child with educated parents.

These kids will thrive in the better public schools and also in schools that we don't typically consider to be that good. But, that's because these kids are full of the "less privileged" students, whatever the hell that's supposed to mean.

Thanks to NCLB, we know that a large achievement gap exists betwen the lower-SES kids and the higher-SES kids, and black and Hispanic kids and white and Asian kids. Even in affluent schools districts.

And, thanks to studies like the Minnesota Transracial Adoption Study we know that familial environment doesn't make much, if any, of a difference either when it comes to student success.

The only thing that does seem to matter is improved instruction, the earlier the better.

Arne Learned How to Exaggerate

Super-blogger Russo points us to an interview with our new Education Secretary, Arne Duncan, in which he makes the following claim:

And the Chicago public schools are desperately underfunded by the state.

This is misleading statement. The source of the funding is largely irrelevant (I understand that many states and the Feds often attach certain strings to funding that can be unproductive). What is important is the Chicago School District's actual funding.

How much do you think a chronically underfunded school district has to spend? Five? Six thousand dollars per pupil?

According to the district's own budget reporting the number is closer to "$10,555 operating expenditure per pupil."

And even that is a gross understatement.

The Chicago School District spends more than that. The $10,555 per pupil figure is for operating expenses (Operation Expenses Budget: $ 4.648 Billion ) . (It also seems to include about 31,759 extra students over the officially reported 408,601 student who attend schools in the district)

The total expenditures for Fiscal Year 2008 (full budget, p. 11: pdf) were actually $ 5.786 Billion. That means that the Chicago School District spent between $13,140 and $14,161 per pupil. (Depending on whether you count the 31,759 mystery students )
That's a lot of chicken now matter how you slice it.

Watch what you ask for

For years we've been hearing from educators that their failure to improve under NCLB is because NCLB wasn't "fully funded." Only with more money could we expect education utopia.

Educators just lost that convenient excuse. The recently passed stimulus plan just gave educators all they've been asking for and then some.

And, the Administration is (naively) expecting to see results and soon. Here's Biden talking to educators.

I genuinely need your help to make this work because, folks, look at it this way. We've been given all the ammunition. If we shoot and miss, if we squander the opportunity, tell me how long you think it's going to take for another American president to go and ask for more dollars to correct the education system.

...

You've got a president and vice president absolutely committed to having all the tools you need to finally get it right in American public education

He doesn't realize it yet, but he'll regret those words.

Public schools have lots of problems, but lack of money isn't one of them (though fiscal management certainly is).



This graph only captures a fraction of educational spending (mostly operating expenditures) but is sufficient to show the real dollar increase schools have gotten since the 1970s. Don't try looking for an increase in achievement during this doubling of funding, you won't find it.

Educators now find themselves in a pickle -- they've been given everything they've been asking for and which they've promised us will lead to miracles. Now they have to perform.

Of course, we already know that they're not going to be able to perform. And, quite frankly, I think they know it too.

But it will be interesting to see how this plays out over the next few years. Expect lots of backpedaling and "I don't remember saying that" type of stuff.

The NYT Doesn't Understand What a Charter School Is

At least based on this article.

Let's start with the title which doesn't make any sense.

Charters Offer More Choices in Harlem, but Stir Concern for Public Schools

Charter schools are public schools. What the Times means to say is that charters are stirring concern for non-charter public schools (or whatever retronym you prefer).

Even the definition given by the Times is off.

Charter schools, which are publicly financed but have latitude in how they operate

What the Times means is that charters aren't constrained by the legacy of conditions and agreements they agreed to or had imposed upon them during the decades of non-competition which, while generally true, misses the main distinction.

The main difference between charter schools and non-charter schools is that charter schools have to earn their customers, i.e., students and then retain them, thus, creating a quasi-market.

This is, of course, no guarantee that charter schools will perform any better than non-charter schools. Many won't. The market doesn't guarantee winners. The market works by ruthlessly weeding out the losers.

It's pretty clear that the Times doesn't now why or how a market works. But, they are quickly learning that lesson as they slowly drive themselves out of business as their financial losses continue to mount.

The School District of Philadelphia's Imagine 2014 Plan

Philadlephia got itself a brand spanking new "CEO" of its schools who promptly issued her plan to reform Philadelphia's well-funded yet under-performing schools. Here it goes.

It's full of pretty words and lofty goals and, by my calculation, stands a zero percent chance of working.

The plan's primary tell is found on page 8.

Elementary Schools

Reading

o Ensure that all elementary students are proficient in reading by third grade.

o Utilize multiple assessment tools to identify students who need targeted reading interventions.

o Provide students reading below the 40th percentile with the opportunity to receive half‐hour daily lessons for 12 ‐ 20 weeks from teachers specially trained in the Reading Recovery model.

o Improve reading fluency and comprehension of ELL and elementary students with language deficits by using guided oral reading instruction and building sight word knowledge in order to recognize words quickly when reading.

Guided reading instruction and Reading Recovery for the strugglers that's a recipe for reading failure.

Last I checked, poor readers don't typically find academic success. They also tend to behave poorly -- better to act up than to look stupid in front of your peers.

And just in case a few students learn to read with some proficiency they're going to be hit with:

instructional practices, such as the use of cooperative learning, inquiry‐based instruction, thinking maps, project‐based learning, simulations, hands‐on learning, and integrated technology

because this works so well with students having little background knowledge and low language skills, i.e., many of Philadelphia's currently failing students.

If they had a way to short these plans, you could make a killing.

Some Critical Thinking Skills Are Critical

This is a long one kids, so bear with me. There is a payoff in the end.

Wikipedia provides a reasonable definition of critical thinking:

Critical thinking is purposeful and reflective judgment about what to believe or do in response to observations, experience, verbal or written expressions, or arguments. Critical thinking might involve determining the meaning and significance of what is observed or expressed, or, concerning a given inference or argument, determining whether there is adequate justification to accept the conclusion as true.

Critically evaluating assertions, arguments, and proposals, whether presented orally or in print, is an important comprehension skill. Many personal, professional, and social decisions are based on what we are told by other people. Because faulty arguments and propaganda are so common, critical thinking has a role in almost every important decision we make.

The consensus is that the typical K-12 education does not result in students with good critical thinking skills. I agree with this consensus.

What I do not agree with; however, is the notion that critical thinking is some generalized skill that is independent of domain knowledge.

Let's see why this is so by looking at skills needed to critically read a passage of text. I am going to use the procedure outlined in Direct Instruction Reading, 4th edition, chapter 22, which is a simplified version of the skill suitable for elementary school students, but, which sadly most students never acquire.

The four steps in the critical-reading process can be treated as the major component skills:

(1) identify the author's conclusion; that is, what does the author want the reader to believe?

(2) Determine what evidence is presented; that is, what does the author present to convince the reader? Evidence or opinion?

(3) Determine the trustworthiness of the author, that is, can the reader trust what the author says?

• Does the evidence come from a qualified person?
• Does the person have biases?

(4) Determine if the conclusion derives from the evidence. Identify any faulty arguments.

• Tradition, either old or new (sometimes called a bandwagon effect)
• Improper generalization
• Confusing correlation with causation (or coincidence)

Those who think critical thinking/reading is a generalized skill are confusing the general procedure with the act of performing the procedure itself. A student might have learned the procedure, but be unable to perform the procedure adequately for a particular text. That's where domain knowledge comes into play.

The best way to see this is by way of example. So, let's use the procedure to critically read a typical passage that students might encounter.

Thomas Edison, the inventor of the lightbulb, was seriously concerned about the increasing use of alternating current as a form of electricity. Edison believed that because alternating current involved so much more current than direct current, alternating current was a threat to the nation. Many fires were caused by alternating currents. In fact, alternating current was used in Sing Sing to electrocute criminals. Direct current was used with light bulbs for many years. Edison felt direct current was still the best form of electricity.

Another example of the dangers of alternating current has just occurred. A house wired with alternating current caught fire and burned to the ground. The fire started when an electrical wire became so hot that a wall caught fire. Alternating current will eventually cause a fire whenever it is used. Direct current rather than alternating current should be used for lighting.

The Daily Post used direct current to light its press room for over a year. Reporters are much happier now. They write more interesting stories. Sales of the newspaper have increased dramatically. The Daily Post is now the most popular newspaper.

Identify the Author's Conclusion

First the students must use details from the passage (seriously concerned, a threat to the nation, direct current is still the best) to form a main idea (or author's conclusion). Identifying an author's conclusion is a continuation of summarization skills.

Right off the bat you can see that critical reading/thinking is dependent on domain knowledge and the student's ability to extract the author's conclusion from the text and justify that conclusion with supporting details from the text. A student might be able to extract the main idea from Dr/ Seuss's Green Eggs and Ham, but be unable to extract the main idea from one of Plato's Dialogues. If critical thinking/reading were a generalized skill, then this wouldn't be the case. The student's inability to identify the author's conclusion will impact his ability to perform the remainder of the procedure.


Discriminating Evidence from Opinion

The second step is for the student to decide whether the author's conclusion is based on opinion or evidence. If it is based on opinion, students must understand that the conclusion is nothing more than a suggestion by the author about what people should think. A conclusion based on opinion does not imply that the student should believe or act on it.

In the example passage, both opinion and evidence are used to support the author's conclusion. The statement that alternating current is a threat to the nation is opinion. The other details are evidence used to justify the author's conclusion (occurrence of fires, electrocution of criminals, initial use of direct current as an energy source.)

This step is also intertwined with domain knowledge. Being able to discriminate between fact and opinion requires that you understand the underlying statement in the first place.

Fact or opinion?

In my judgment, the total enthalpy of any non-isolated thermodynamic system tends to decrease over time, approaching a minimum value.

It's not an opinion, even though it uses a phrase ("In my judgment") that typically indicates opinion. It is evidence. The statement is the second law of thermodynamics. But, you wouldn't know this unless you knew thermodynamics, and, thus, you wouldn't be able to discriminate between opinion and evidence.

In actuality the statement is inaccurate evidence. The statement is not the second law of thermodynamics; I changed some of the terms to make it inaccurate. Again, you wouldn't know this unless you knew thermodynamics.


Determining the Trustworthiness of an Author

The third step consists of several questions, all relating to the reliability or trustworthiness of the person presenting the argument.

Question (a) is whether the evidence comes from a qualified person. Since Edison was definitely an expert on electricity in the late 1800s, he was qualified.

Again, this determination requires that you know something about Thomas Edison. That's domain knowledge.

Question (b) concerns biases the expert might have. In Edison's case, two major biases existed. One was his deep personal and involvement in a company that provided direct current. He stood to lose money if alternating current replaced direct current. Also, his reputation was at stake. He became famous, in part, because of his discovery of the lightbulb and a distribution system for electricity based on the use of direct current. If alternating current replaced direct current, his reputation might be diminished.

Again, we need domain knowledge knowledge to make the determination.

Since Edison's biases contribute to the passage's conclusion, the evidence he cites may not be trustworthy.

Since there is doubt about the trustworthiness of the author, students must seek information from different experts. The statement that direct current is the best form of electricity is disputed by many experts. Alternating current can be transmitted great distances, but direct current cannot (at least during the relevant period). If remote areas are to receive electricity at a reasonable rate, alternating current is a necessity.

So, since the expert is biased and alternate interpretations of the evidence are compelling, the evidence is probably not trustworthy.

Again, the student must know quite a bit about electricity and its ability to be transmitted to make seek out the appropriate sources and understand the information presented. take a look at this article on electric power transmission and see if you can make sense of the differences between direct current transmission and alternating current transmission. And then you have to be aware of the history of electric power transmission to understand that the information this article presents was unknown during the time that our passage was written. But, how would you know this without quite a bit of domain knowledge in both electricity and the history of electrical power transmission? Google may be your friend, but he's not this good of a friend for you.

Identifying Faulty Arguments

The final step in the critical-reading process is deciding whether a conclusion legitimately derives from the evidence. In many arguments, valid evidence will be presented, but then a conclusion will be drawn that does not derive from the evidence. In the alternating-current example, one possible interpretation is that since direct current has been used with lightbulbs for many years, it should continue to be the best form of electricity. This faulty argument illustrates the use of tradition: what has been the best must continue to be the best. Conclusions based on tradition are not necessarily true. What has worked well may continue to be the best procedure, or a better procedure may be developed. Students can disregard conclusions based on tradition. (Note that the same attitude can be taken toward newly developing traditions; i.e., "Everybody is starting to use alternating current; therefore, you should, too." A conclusion that a product or procedure is better because it is popular is faulty.)

In the second paragraph, there is an example of improper generalization. One valid example is presented, but then a conclusion is drawn that applies to all examples. One fire caused by alternating current does not mean that alternating current will cause a fire every place it is used. Improper generalization occurs often: "I saw a rich person who was rude. What makes rich people rude?" "We sat next to a long-haired man in the movies. He smelled. I'll bet he hadn't bathed in weeks. Long-hairs should take better care of their bodies."

The third paragraph involves a confusion of causation and correlation. An event that is associated with success or some other positive outcome through coincidence is erroneously concluded to be the cause of the positive outcome.

Direct-current lighting is associated with happier reporters, more interesting stories, and greater sales; however, direct current did not necessarily cause reporters to be happier. The electric lighting that produced the positive outcomes could have been achieved with direct or alternating current. Conclusions suggesting causation that are, in fact, based on correlation can be disregarded. Confusion of correlation and causation is often made: "Joe Blow uses Squirt-Squirt deodorant, and girls always chase him." "Sally took You-Bet-Your-Life vitamins every day. She lived to be 106."

Of course, there are many other logical fallacies and faulty arguments that the student should know, but these three are a good start for primary grade students since they are among the most common.

Sadly,this last step is often completely ignored in most schools. Most students are simply not taught how to identify faulty arguments at all. And let me be crystal clear here: it should be taught. And, I'm guessing that the reason why it isn't taught more often, or at least learned by most students, is because it is the culmination of basic reading instruction which is rarely reached by many students.

Many students never learn how to decode accurately and proficiently in grade-level texts. This is criminal, quite frankly, because this is something we have known how to effectively teach for some time now. Then students need to comprehend what they've decoded well enough to extract the main idea and the justifications therefor. Most students can't do this competently when the main idea is presented to them as an alternate in a multiple choice format, much less being able to generate one on their own. And, if a student can't do this initial step in critical reading, how is she going to accomplish the accomplish the remaining steps which require an understanding of the author's conclusion in the first place?


So, what we have is two separate problems. One problem is that students are often not taught how to spot faulty arguments and logical fallacies which are important skills needed to think and read critically. No one seriously disputes that this is a problem. These skills should be taught. Period.

But, even if students are taught these skills, they will not necessarily be able to think and read critically because knowing how to identify and distinguish faulty arguments and logical fallacies is not the same thing as critically thinking/reading as many proponents of 21st Century skills seem to think. Critical thinking/reading involves much more than identifying faulty arguments, as I described above. And, the ability to identify faulty arguments is not the generalized form of critical thinking as some seem to think.

There is another problem that must also be dealt with. That problem is that students can't comprehend well enough to extract the author's conclusion, to discriminate evidence from opinion, and to determine if the author is trustworthy. All of these skills require and are a function of the student's domain knowledge. You have to know a lot of stuff, to think about a lot of stuff.

This is just a roundabout of saying that there are prerequisites to being able to think/read critically that also need to be addressed (and have never been adequately addressed) before students can make use of the "21st century skills" many think are important, like being able to identify faulty arguments and logical fallacies.

What's the sense of teaching a student astrophysics if the student doesn't know how to do basic arithmetic or understand basic science? Perhaps this is why schools haven't traditional taught these things in the first place.

Rebranding our way to better schools

Our new Education Secretary, Arne Duncan, wants to fix our education woes by, wait for it:

“Let’s rebrand it,” he said in an interview. “Give it a new name.”

How about the We Still Don't Know How to Spend the Money Effectively Act?

That gets right at the heart of the matter. It also implies a solution: find out how to spend the money effectively.

This seems to me like the logical starting point. So, who knows of any federal programs with the goal of finding out what predictably works backed by evidence?

Today's Video

Classroom Management presentation by Saul Axelrod.



Lots of good tips. Long.

Direct Link.

Alfie Kohn and the Murray Gell-Mann Amnesia effect Part II

Continuing on from Part I.

We're now getting into the last prong of Kohn's main argument, after which he appears to take a kitchen sink approach and throws in all the remaining negative information he could find.

Kohn's last argument is based on a logical fallacy:

Finally, outside evaluators of the project – as well as an official review by the U. S. General Accounting Office – determined that there were still other problems in its design and analysis that undermined the basic findings. Their overall conclusion, published in the Harvard Educational Review, was that, “because of misclassification of the models, inadequate measurement of results, and flawed statistical analysis,” the study simply “does not demonstrate that models emphasizing basic skills are superior to other models.”

As a preliminary matter, Kohn fails to mention that the "outside evaluators" he's referring to, House et al. were funded by the Ford Foundation which had also funded a few of the losing models in PFT. As such, Kohn's source has "potential bias" issues which Kohn fails to alert his readers to. Kohn also fails to alert his readers to all the other similar "outside evaluators" which analyzed both the PFT data and House's analysis and came to a different conclusion. These other outside evaluators are no more biased than House, so its curious to see why Kohn would fail to mention them.

Next comes the first of Kohn's logical fallacies. Kohn commits the fallacy of division (or whole-to-part fallacy) when Kohn claims PFT "simply 'does not demonstrate that models emphasizing basic skills are superior to other models.'" Even if the basic skills models as a whole weren't superior to the other models doesn't mean that the DI model alone wasn't. The data certainly shows that the DI model was the superior performer.

The point here is that Kohn is criticizing DI yet has already started to veer off and is trying to mislead readers by dragging in information pertaining to other programs or the more general classification of basic skills programs.

Next Kohn makes an appeal to authority, another logical fallacy, when he mentions that the results were "published in the Harvard Educational Review." I'd be willing to cut Kohn some slack had he mentioned all the other journals of the research he cites. But this is the only one he cites. Coincidence? What I do know is that he failed to mention another study on PFT that was also published in the Harvard Educational Review that affirmed the findings of PFT. Another convenient oversight.

Kohn again buries the weakest the parts of his argument in a footnote. Kohn parrots the House study's findings which is based on a reanalysis of the PFT data. This reanalysis was not without it's own problems as set forth in another study by Bereiter et al which reanalyzed the House reanalysis (It should be mentioned that this study has the same potential bias problems as the House study since Bereiter had professional ties to DI before PFT).

Let us therefore consider carefully what the House committee did in their reanalysis. First, they used site means rather than individual scores as the unit of analysis. This decision automatically reduced the Follow Through planned variation experiment from a very large one, with an N of thousands, to a rather small one, with an N in the neighborhood of one hundred. As previously indicated, we endorse this decision. However, it seems to us that when one has opted to convert a large experiment into a small one, it is important to make certain adjustments in strategy. This the House committee failed to do. If an experiment is very large, one can afford to be cavalier about problems of power, since the large N will presumably make it possible to detect true effects against considerable background noise. In a small experiment, one must be watchful and try to control as much random error as possible in order to avoid masking a true effect.

However, instead of trying to perform the most powerful analysis possible in the circumstances, the House committee weakened their analysis in a number of ways that seem to have no warrant. First, they chose to compare Follow Through models on the basis of Follow Through/Non-Follow Through differences, thus unnecessarily adding error variance associated with the Non-Follow Through groups. Next, they chose to use adjusted differences based on the "local" analysis, thus maximizing error due to mismatch. Next, they based their analysis on only a part of the available data. They excluded data from the second kindergarten-entering cohort, one of the largest cohorts, even though these data formed part of the basis for the conclusions they were criticizing. This puzzling exclusion reduced the number of sites considered, thus reducing the likelihood of finding significant differences. Finally, they divided each effect-size score by the standard deviation of test scores in the particular cohort in which the effect was observed. This manipulation served no apparent purpose. And minor though its effects may be, such as they are would be in the direction of adding further error variance to the analysis.

The upshot of all these methodological choices was that, while the House group's reanalysis largely confirmed the ranking of models arrived at by Abt Associates, it showed the differences to be small and insignificant. Given the House committee's methodology, this result is not surprising. The procedures they adopted were not biased in the sense of favoring one Follow Through model over another; hence it was to be expected that their analysis, using the same effect measures as Abt, would replicate the rankings obtained by Abt. (The rank differences shown in Table 7 of the House report are probably mostly the result of the House committee's exclusion of data from one of the cohorts on which the Abt rankings were based.) On the other hand, the procedures adopted by the House committee all tended in the direction of maximizing random error, thus tending to make differences appear small and insignificant.

It is one thing to fail to mention that your source is potentially biased due to its financial ties to both some of the losing programs in PFT and the "outside" evaluators. But failing to mention that your source has been itself criticized as systematically adopting procedures which all have the effect of minimizing the direction of error, many with dubious or no scientific validity, in favor of the very programs with the corporate ties is quite another. The prudent advocate would at least attempt to explain these criticisms away, but in any event, readers should be made aware of these significant infirmities in your underlying studies.

Kohn concludes his main argument with a gratuitous swipe:

Furthermore, even if Direct Instruction really was better than other models at the time of the study, to cite that result today as proof of its superiority is to assume that educators have learned nothing in the intervening three decades about effective ways of teaching young children. The value of newer approaches – including Whole Language, as we’ll see -- means that comparative data from the 1960s now have a sharply limited relevance.

I bet Kohn wishes he could take that crack about whole language (an educational philosophy so bad it's advocates had to re-brand it to disassociate it from the lengthy trail of negative data it amassed) back now.

And, what evidence is there that today's educators have learned anything since the mid 1970's (not the 60s as Kohn claims)? The longitudinal NAEP data tells a much different story.

Next Kohn gets into the kitchen sink part of his argument. Let's take each in turn.

1. First Kohn cites some newspaper accounts on DI. Kohn admits these accounts are anecdotal. I agree with Kohn, but I'm wondering why he included them in his argument anyway. I'm guessing lurid innuendo. I could go into detail refuting the points Kohn recounts, but it's not worth the effort for anecdotes such as these.
2. Next, Kohn makes another fallacy of division when he states "it’s common knowledge among many inner-city educators that children often make little if any meaningful progress with skills-based instruction." Of course, there's lots of data from inner-city schools pertaining to DI that show that this statement isn't true with respect to DI.
3. Last, Kohn claims that there is "a lot more research dating back to the same era as the Follow Through project supports a very different conclusion" and then goes about citing various longitudinal studies which purport to show better long-term outcomes for various child-centered P-3 programs (which Kohn prefers) to DI programs. Apparently, Kohn hasn't heard of "confounding variables." And, chooses to ignre th efact that many of these studies were conducted by the sponsors of these programs. And, that some had serious methodological flaws and fails to mention that the High Scope study was also sponsored by the same people responsible for one of the worst performers in PFT.

Here's Kohn's big conclusion:

Still, with the single exception of the Follow-Through study (where a skills-oriented model produced gains on a skills-oriented test, and even then, only at some sites), the results are striking for their consistent message that a tightly structured, traditionally academic model for young children provides virtually no lasting benefits and proves to be potentially harmful in many respects.

This is only true for the studies Kohn has chosen to cite which are only the "negative" ones. Kohn has basically cherry-picked the studies and excluded all the ones showing positive effects, such as Gary Adams' meta analysis and the underlying studies. Kohn also cites his research uncritically and has ignored all the criticism directed at the studies he cites. He doesn't even offer an explanation, he simply ignores them. He also ignores all the potential bias problems and methodological flaws in his cited studies. Kohn seems to have an unrealistically high standard for DI studies and a very low one for research with conclusions he likes. I find it hard to believe that any study cited by KOhn in any of his books could withstand the constraints imposed by House et al. on the PFT data.

In short, Kohn's "hard evidence" against DI appears to be almost exclusively opinion, rather than fact. Little of this opinion is supported by data, though Kohn's use of selective quotes from "research" attempts to convey that impression. And, completely ignoring all the contrary evidence and presenting such a one-sided evaluation based on that cherry-picked "evidence" is reprehensible for anyone claiming to to be dispassionate.

In this analysis of DI, Kohn has shown himself to be an untrustworthy advocate and the same pattern of scholarly malfeasance is evident in all his writings I've read.

Alfie Kohn and the Murray Gell-Mann Amnesia effect

(the introduction can be found here)

In the comments of the recent Willingham-Kohn dust-up, edu-blogger Stuart Buck brought up DI and Kohn responded by citing this article by him which immediately reminded me of the Murray Gell-Mann Amnesia effect.

The late Michael Crichton once gave a speech describing what he termed the Murray Gell-Mann Amnesia effect.

Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well... You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.

In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.

I know enough about the research on DI to know that Kohn's description of the DI research qualifies as one of the worst hatchet jobs in education policy advocacy. As such, it should serve as evidence that Alfie Kohn might not be a trustworthy source on education policy and that his analysis of education research should be closely scrutinized in order to stave off the Murray Gell-Mann Amnesia effect.

But don't take my word for it, let's review Kohn's description of the DI research.

After we get past an initial paragraph of over-heated inflammatory language, Kohn's first argument is related to the results of Project Follow Through (PFT) is:

Of course, even if these results could be taken at face value, we don’t have any basis for assuming that the model would work for anyone other than disadvantaged children of primary school age.

But, PFT involved some schools in middle class neighborhoods and middle-class kids took part and were evaluated as part of the research. In fact, a very diverse set of students were evaluated.

The DI model was the best performing model for "disadvantaged children" as Kohn acknowledges, but it was also the best performing model for high-performing students, White children, Native Americans, African-American students, Hispanic students, English language learners, urban children, rural children, and the very lowest of the disadvantaged children. So there is a basis, a good basis, for assuming that the DI model works for most primary school aged children. And, in fact, these results have been replicated numerous times in subsequent studies which Kohn also fails to acknowledge.

Kohn's next argument is a repetition of the "variability" argument that others have levied against the PFT results:

To begin with, the primary research analysts wrote that the “clearest finding” of Follow Through was not the superiority of any one style of teaching but the fact that “each model’s performance varies widely from site to site.”[1] In fact, the variation in results from one location to the next of a given model of instruction was greater than the variation between one model and the next. That means the site that kids happened to attend was a better predictor of how well they learned than was the style of teaching (skills-based, child-centered, or whatever).

This is spurious conclusion because most of the variability is attributed to the inclusion of two cohorts from the Grand Rapids site in the analysis which had severed ties with the DI sponsor well before the end of the study. This is well documented. The data from the Grand rapids site was the only "DI site" with low performance (a half standard deviation below the mean of the other DI sites). The Grand Rapids site is the only "DI site" that consistently fell below national norms. In fact, most of the variability in the remaining DI sites is above National norms.

In addition, subsequent research, involving researchers with at least some professional/reputational ties to DI, has shown that the variability between sites is mostly attributable to demographic factors and experimental error, and not to the DI program.

We disagree with both Abt and House et al. in that we do not find variability among sites to be so great that it overshadows variability among models. It appears that a large part of the variability observed by Abt and House et al. was due to demographic factors and experimental error. Once this variability is brought under control, it becomes evident that differences between models are quite large in relation to the unexplained variability within models.

In any event, even if the variability finding was characterized as the main finding by the primary researchers, this in no way diminishes the finding that DI was the superior performing program across the board for all measures tested for all groups tests. It's still a valid finding and has been upheld by numerous researchers examining the findings since the initial evaluation.

So, best performing program and the only program whose variability was mostly above national norms does not a valid criticism make. Strike two for Kohn.

Next Kohn, attacks the testing instruments used in PFT:

Second, the primary measure of success used in the study was a standardized multiple-choice test of basic skills called the Metropolitan Achievement Test. [(MAT)]

The MAT is not just a test of basic skills (such as Listening for Sound (sound-symbol relationships), Word Knowledge (vocabulary words), Word Analysis (word identification), Mathematic Computation (math calculations), Spelling, and Language (punctuation, capitalization, and word usage)).

It is also a test of cognitive skills as well. Several Metropolitan subtests measure indirect cognitive consequences of learning, such as the Reading subtest (which is, in effect, paragraph comprehension), the Mathematics Problem-Solving subtest, and the Mathematics Concepts test (knowledge of math principles and relationships).

This is important because Kohn goes on to claim:

While children were also given other cognitive and psychological assessments, these measures were so poorly chosen as to be virtually worthless.

Even if the other cognitive and psychological assessments were "poorly chosen" it does not diminish the fact that the MAT is a well respected test of both basic and cognitive/conceptual skills, as ackowledged by subsequent researchers. The other cognitive/conceptual skills test used was the Raven's Colored Progressive Matrices, but it did not prove to discriminate between models or show change in scores over time.

Also, the affective skills were assessed using two instruments: the Intellectual Achievement Responsibility Scale (to assess whether children attribute their success (+) or failures (-) to themselves or external forces) and the Coopersmith Self-Esteem Inventory (to assess how children feel about themselves, the way they think other people feel about them, and their feelings about school).

Kohn buries the reason why he believes the cognitive and affective skills tests were "poorly chosen" in a footnote.

There is strong reason to doubt whether tests billed as measuring complex “cognitive, conceptual skills” really did so. Even the primary analysts conceded that “the measures on the cognitive and affective domains are much less appropriate” than is the main skills test (Stebbins et al., 35). A group of experts on experimental design commissioned to review the study went even further, stating that the project “amounts essentially to a comparative study of the effects of Follow Through models on the mechanics of reading, writing, and arithmetic” (House et al., 1978, p. 145). (This raises the interesting question of whether it is even possible to measure the conceptual understanding or cognitive sophistication of young children with a standardized test.)

Let's take these "strong reasons" in order. Even if the primary analysts believed that these tests were "much less appropriate" this doesn't they didn't measure what they purported to measure. There is no evidence that they didn't. It could also be that the primary researchers believed that the basic skills tests were best suited for measuring what K-3 students are typically expected to know. In any event, the opinion that the tests were "much less appropriate" does not lead one to conclude that the tests didn't measure "complex 'cognitive, conceptual skills.'” This is an empirical question and Kohn provides no empirical support for his conclusion.

Next, Kohn relies on the opinions of the "experts" commissioned and funded by the Ford Foundation (which I'll get to later) for the proposition that PFT only measured the affects of "the mechanics of reading, writing, and arithmetic." Apparently, what these experts were getting at was that students who hadn't learned the mechanics of reading, writing, and doing arithmetic might not be able to demonstrate their cognitive skills. This is also an empirical question. But, these experts provided no empirical support for their conclusion. Other researchers, however, did look into the question once it was raised.

Conceivably, certain models-let us say those that avowedly emphasize "cognitive" objectives-are doing a superior job of teaching the more cognitive aspects of reading and mathematics, but the effects are being obscured by the fact that performance on the appropriate subtests depends on mechanical proficiency as well as on higher-level cognitive capabilities. If so, these hidden effects might be revealed by using performance on the more "mechanical" subtests as covariates.

This we did. Model differences in Reading (comprehension) performance were examined, including Word Knowledge as a covariate. Differences in Mathematics Problem Solving were examined, including Mathematics Computation among the covariates. In both cases the analyses of covariance revealed no significant differences among models. This is not a surprising result, given the high correlation among Metropolitan subtests. Taking out the variance due to one subtest leaves little variance in another. Yet it was not a forgone conclusion that the results would be negative. If the models that proclaimed cognitive objectives actually achieved those objectives, it would be reasonable to expect those achievements to show up in our analyses.

So, again we have no valid reason for discounting the results of the non-basic skills tests. Unsupported opinion is not a valid reason last I checked.

Lastly, Kohn raises a question:

This raises the interesting question of whether it is even possible to measure the conceptual understanding or cognitive sophistication of young children with a standardized test.

And then conspicuously fails to answer it. This is the poor man's version of debate. Moreover, nothing that precedes this "interesting question" is capable of actually raising it.Also , I'm not sure if Kohn is trying to claim that you can't measure these skills or that you can't measure these skills with a standardized test. Though, Kohn provides no support for either question.

Kohn's innuendo is that the students might have had conceptual understanding or cognitive sophistication that we must accept on faith despite the evidence that more students in the non-DI models were incapable of demonstrating these magical immeasurable skills on simple tests of comprehension of written paragraphs, mathematical problem-solving and knowledge of math principles and relationships, not to mention all the other "basic skills" that were measured.

Unfortunately, Kohn fails on all three counts to provide evidence that would compel a reader to follow him down his opinionated path that PFT only measured basic skills and that the other measures were "virtually worthless." Maybe this is why he buried this one in a footnote.

Next Kohn claims:

Some of the nontraditional educators involved in the study weren’t informed that their programs were going to end up being judged on this basis.

First of all, the DI educators were also non-traditional in as much as the other models' educators were. DI is about as far removed from traditional pedagogy as the other models.

Also, even if some of the other educators claimed that they were never initially told that their models weren't going to be judged on reading comprehension, math problem solving, and the like, they would have quickly learned what was coming down the pike since the PFT students were extensively tested throughout the study. And, it was the third and fourth cohorts that formed the cohorts of the evaluation. Whoever claims to not have known initially would certainly have found out during the time time the first two cohorts passed through.

Next Kohn claims:

The Direct Instruction teachers methodically prepared their students to succeed on a skills test and, to some extent at least, it worked.

Actually, the DI model systematically prepared their students to read, understand the conventions of language, to spell, and to do arithmetic with an emphasis "placed on the children's learning intelligent behavior rather than specific pieces of information by rote memorization." And, the students outperformed the other students on tests of sound-symbol relationships, vocabulary words, word identification, math calculations, spelling, punctuation, capitalization, word usage, paragraph comprehension,mathematics problem-solving , knowledge of math principles and relationships, and the affective measures. There was no evidence that the DI students engaged in test preparation as alluded to by Kohn.

PFT demonstrated, once again, that teaching these skills directly was more effective than teaching them obliquely which was what the other models believed would lead to superior performance. It turns out they were wrong and they continue to be wrong to this day.

For those keeping track at home, Kohn has now failed to establish the first two prongs of his argument. He has one more prong left which I'll take up in my next post.