Statistical Significance in Education Research

Recently, Alexander Russo posted about education research and was thinking about:

getting away from the whole notion of statistical significance that's been drummed into us, which apparently isn't the gold standard we think it is.

What we really need to do is get away from the layman's confusion with the term "statistical significance." From Wikipedia:

In statistics, a result is called statistically significant if it is unlikely to have occurred by chance. "A statistically significant difference" simply means there is statistical evidence that there is a difference; it does not mean the difference is necessarily large, important or significant in the common meaning of the word.

To put it in layman's terms, when we say something is statistically significant, we merely mean that we are at least 95% certain (typically) that the observed experimental difference is not the result of chance. But, here's where the confusion comes in.

A common misconception is that a statistically significant result is always of practical significance, or demonstrates a large effect in the population. Unfortunately, this problem is commonly encountered in scientific writing. Given a sufficiently large sample, extremely small and non-notable differences can be found to be statistically significant, and statistical significance says nothing about the practical significance of a difference.

Round up enough students in your experiment and even tiny differences in academic performance between the experimental group and the control group pass the test of statistical significance.

That's why we look to the effect size of the experiment, which is the magnitude of the observed effect of the intervention being tested. In education research, an effect size less than 0.25 of a standard deviation is not typically considered to be educationally significant. Here are the rules of thumb that are typically used for classifiying ffect size in education research: small effect size (> 0.25), medium effect size (>0.5), and large effect size (> 0.8).

So, when it comes to education research we want to see studies that conform to the standards of the behavioral sciences (this eliminates 90% of all "research" in education). Once we've culled the herd, we throw out all the research whose results are neither statistically significant nor educationally significant.

What is left? Not much at all. Such is the sad state of educational research.

For more on the common scams you'll find in education research see this post and the linked paper by Slavin.

Meet the New Boss, Same as the Old Boss

Carol Johnson took over as superintendent of Boston's public school system five months ago and has cleaned house by firing all her underlings. Her goal is to "close the achievement gap." Specifically, her plan is to close the achievement gap by doing the following:

• Ensuring graduation for all by creating and expanding programs to prevent students from dropping out of school and recovering those who have. Currently, only 58 percent of students graduate in four years. (rainbows)



• Strengthening support for students with special needs. Foster inclusion classrooms and reduce over-referral to special education, particularly among boys of color. (lollipops)



• Creating a district-wide literacy curriculum that prepares children to read and write proficiently. (unicorns)



• Expanding enrichment programs during and after school, including the arts and music, recreation and athletics, and advanced placement for students excelling academically. (happy thoughts)



• Exploring new school models, including public Montessori schools, early/middle colleges with dual enrollment, gender-specific programming, school/business partnerships and international baccalaureate programs. Appoint a new, senior-level position to oversee and monitor the performance of pilot schools. (magic)

Gee, no one's ever tried those bromides before.

Prediction: achievement gap will remain undisturbed in Boston.

WaPo is Confused or Maybe I am

Update: It's Wapo, not me.

Theola Labbé of the Washington Post reports on the $53 Million increase sought by the the DC school system.

Mayor Adrian M. Fenty said he will seek a 5 percent increase in per-pupil funding for the more than 70,000 students enrolled in the D.C. system and charter schools, a $52.9 million effort to improve city education.

Doing the math (52 mill/5%/70,000), I get per pupil expenditures of $14,857 which probably represents operating expenses since school matters reports that the DC school system spent $13,187 per pupil on operating expenses and $16,595 per pupil for total expenditures back in 2002. Perhaps my back of he envelope calculation is off, but get a load of Wapo's.

The recommendation to spend $8,770 a child when the school year begins in August came from State Superintendent Deborah A. Gist. Now, $8,322 is spent for each student.

Thee DC school system hasn't spent that little since Watergate. What gives?

You can bet when the military spends $600 for a toilet seat, it'll be reported in all its glory. But, when it comes to reporting to the public how much is spent on public education, obfuscation is the order of the day. Now why do you suppose that is so?

Garrison Keillor: Mugged by Reality

This Garrison Keillor article is making the rounds in the edusphere, thanks to Alexander Russo.

Keillor, the good Democrat that he is, has finally noticed that our public schools aren't exactly doing such a swell job educating students, especially the poor, brown, and black ones. And, what caused this sudden realization? The money quote:

Liberal dogma says that each child is inherently gifted and will read if only he is read to. This was true of my grandson; it is demonstrably not true of many kids, including my sandy-haired, gap-toothed daughter.

All bets are off when the effects of that "liberal dogma" start hitting too close to home.

Paying for Student Performance

Sherman Dorn rounds up the commentary on the recent plans to "pay" students for performing well at school.

None of the commenters, including Dorn, appear to understand "student motivation" and how "rewards" affect motivation or lack thereof. The result is a lot of ill-informed opinion, i.e., the typical state of affairs in education opinion journalism.

Here's a forty year old excerpt on the topic from Your Child Can Succeed:

A child who is told to do his arithmetic worksheet has choices. He can either do the worksheet, look out the window, draw a picture, or belt the little girl next to him. Operant Psychology would hold that if you want the child to choose one of these actions over the others, you have to make that one more rewarding (or less punishing) than the others. The value in making the desired activity rewarding (rather than less punishing) is that if the child learns that working arithmetic problems is "rewarding" he will tend to work on arithmetic problems even when he is not rewarded. If he is taught that every time he doesn't do his arithmetic problems he gets clobbered, he will learn a great deal about what happens when he doesn't do arithmetic, but very little about the rewards that may be associated with doing arithmetic.

Give the kid a reason for doing what you want him to do. Set up a contingency so that if he performs he receives something that he wants. The only way he can get the payoff is to do what you want him to do.

Some children do not work for the joy of doing arithmetic. By using payoffs to get them started, the teacher can systematically build up "motivation." At first the child is interested only in the specific payoff--the candy or the extra recess. As he learns, he receives other payoffs, such as praise for good work. After a while he learns to treat the payoff more as a symbol of his competence than as an end in itself. And he learns that the work itself was perhaps less than fun but certainly not punishment. Finally the child will be willing to work for nothing more than the praise and sense of achievement associated with performing well.

There is, of course, much more to student motivation than is presented in this short excerpt. This issue is far more complicated than the commentary would lead you to believe. Check out this series of posts on How To Effectively Manage a Classroom from an actual classroom teacher who's been successful motivating recalitrant students.

Dan Brown has been socially promoted

Dan Brown, author of “The Great Expectations School: A Rookie Year in the New Blackboard Jungle,” is guest blogging at eduwonkette today on mayor Bloomberg's new policy aimed at ending social promotion.

Let's see what a year in the trenches has taught Dan.

Bloomberg is proposing retaining eighth graders who fail one of the state tests or a core class. Brown thinks he knows the reason why these kids are failing:

Students come to school with low academic skills for a variety of reasons. In New York, many are faultless victims of the ever-present crush of poverty and its far-reaching tentacles. The school system’s obsession with high-stakes testing— a game struggling students are poorly equipped to play— exacerbates their frustration. Their self-esteem levels are rock bottom and oppositional behavior often takes root. Can you blame them?

Underlying this argument, which is masterfully hidden by Brown's use of the passive voice, is the premise: New York City schools don't know how to educate kids who come to school with low academic skills. Ultimately that is the root problem. Fixing poverty or eliminating NCLB, as Brown implies, won't fix that condition.

Poverty correlates with low academic skills, but no one has been able to prove that poverty causes low academic skills. And, No one has been able to show, outside of Kozol's and Rothstein's overactive imaginations, that eliminating poverty will improve these low academic skills.

Students who aren't learning in school receive a constant stream of negative feedback on a daily basis telling them that they are failures. Failing a NCLB test is just one drop in this tidal wave of negative feedback. Blaming NCLB for this condition is a large, disingenuous stretch.

Blindly pushing struggling students forward (social promotion) is not the answer, but neither is holding them back for another lap around a failed track. Retaining low-achieving students does not improve their academic future; in fact it often does quite the opposite.

The struggling student conundrum can’t be solved with false choices like the ones offered in the social promotion political debate, but with serious assessments of the short-term and long-term needs of students.

Oddly enough, Brown is complaining about false choices while giving us his own false choices. Why does moving a student to the next grade have to be done "blindly" and why does retaining a student have to be for "another lap around a failed track." These are false choice too.

Brown correctly identifies that these kids need remediation, but the underlying question is how best to deliver that needed remediation (short term solution) and how to prevent it from happening in the future (long term solution). Unfortunately, Brown flubs both answers, as you'll soon see.

The short-term answer for failing students is a major investment in remediation and individualized support. Clearly, the traditional classroom set-up isn’t working for these students.

Brown's short term solution relies on magic, specifically the magic of the "major investment" platitude. According to School Matters, New York City was spending $15,455 per student back in 2005. Dan, that is a major investment already. Spending even more, didn't help Kansas City, and it isn't going to help NYC either.

Disappointingly, Brown's long term solution also relies on the same magic.

Many of Bloomberg and Klein’s school reforms are dynamic and exciting, but the ones that they have not yet made are essential. A more substantial up-front investment in supporting all students will pay manifold dividends.

No doubt because it's worked so well in the past.

At least Brown gives us an idea how he'd spend the money this time: on hackneyed bromides naturally.

• preschool -- Brought to you by the same clowns running every other grade.
• reduced class size -- Another expensive bromide with dubious results.
• more skills tutoring -- Brought to you by the same people who didn't know how to do correctly it the first time around.
• more counseling -- In what? Coping with failure?
• supervisors more concerned with the student's real needs, not test scores -- Except that the the student's real needs are learning all the fundamental skills being measured by the test scores.

Brown, naturally, blames NCLB for denying these things to students. That's like blaming the thermometer for cold weather and thinking that breaking the thermometer will bring warmer weather.

Brown channels Marx for his big conclusion:

Bloomberg is an expert of the business sphere, but bottom-line-driven business models are an ill fit for the education of young human beings. Focusing on holding struggling students back rather than intensively attending to their academic needs is tantamount to blaming the victims. Many socially promoted students have unwittingly suffered the collateral damage of suffocating poverty at home and a depersonalized, test-obsessed regime at school. It’s time they had some doors opened for them, not slammed in their faces.

Too bad half-baked Marxism and circular reason haven't quite had the successful track record that Brown is hoping for.

If these arguments are any indication of the writing in his Brown's book, my advice to you is not to waste your money.

Technology that might be useful

I'm not a fan of technology, for the sake of technology.

Most technology doesn't improve what goes on in the classroom. Typically, the technology merely allows the teacher to do the same old stuff in a new way. Big deal.

However, the class room remote control system highlighted in this NYT article has potential. If used correctly, it could help increase student motivation and increase student feedback. Of course, it could also be used as a glorified Jeopardy game which is not exactly an improvement.

Go read the story.

Same as it ever was

The WaPo reports that there's a new science textbook in town:

The "Story of Science" series by Joy Hakim tells the history of science with wit, narrative depth and research, all vetted by specialists at the Massachusetts Institute of Technology.

The series, which has drawn acclaim, chronicles not only great discoveries but also the scientists who made them.

...

"These books humanize science," Pavlekovich said.

"We teach students this equation and this theory or this topic and that idea, but we never discuss the scientist behind it or how that scientist made the discovery," he said. "It helps students to understand how they struggled and overcame great obstacles to do what they did."

That's lovely. But does it work?

The answer is: nobody knows. That's because in education they don't test their new products before putting them on the market. The publishers didn't set-up a controlled experiment in which they tested this brave new textbook series with a group of students and a competent teacher to see if the students learned as much or more than a typical science textbook. There's a name for this conduct: educational malpractice.

Here's the testing the publishers did:

Scientists and educators say that there are many ways to teach science but that Hakim's approach makes sense.

...

Hakim said MIT scientists, including senior research scientist emeritus Edwin Taylor, checked each chapter of the Einstein book.

That's nice, but does it work?

Nobody knows and nobody cares.

It's Time to Play Spot the Looney

This Week's Education Looney is Philly Inquirer staff writer Susan Snyder for her article in entitled A Phila. school that really works. (hat-tip to commenter Kathy for finding this gem.)

Alexander Russo can stop writing about education research now because Susan Snyder has found what "really works."

Let's see if we can reconstruct the scene of the crime. Get out your luminol.

No doubt Snyder was looking for the Holy Grail of education, a public school, preferably serving a underprivileged minority population, that is performing well. Snyder's victim: The Samuel Powel Elementary School in West Philadelphia. At the Powel school, 70.3% of third graders and 71.2% of fourth graders passed last year's PSSA test. That compares favorably to the statewide average -- 72.8% pass rate for third graders and 70.1% for fourth graders and is far above the average score for Philadelphia Schools -- 46.6% pass rate for third graders and 42.8% for fourth graders. (These numbers come right from the Pennsylvania Department of Education)

Snyder was probably thinking that this one writes itself. All she had to do was visit the school, make a few expert observation, and interview a few expert teachers, and declare a solution to Philadelphia's education woes.

What she wound up with was what all the doe-eyed journalists wind up with when they write about education -- an article full of cliches.

Here are the reasons Snyder gives for Powel's success.

Whatever it was that one of the teachers, Joe Alberti, was doing with his class when Snyder visited:

Using the classroom's Internet-connected smart board, Alberti looked up "ball hawk" and showed students the definition. Throughout the rest of the hour-long lesson, he alternated between reading to students and asking them what different words meant, what they thought would happen next in the story and how they would feel if that happened to them.

His approach works.

Had Snyder visited a few more schools, she would have seen that this is what goes on in 99% of all elementary school classroom nowadays, even the poorly performing ones.

There's something "special" about Powel's environment that makes the students so darn creative:

"It's just the special environment of Powel that encourages kids to read, to be creative, to just be interested in their education," principal Marguerite Holliday said.

...

"In a way, it feels like a little private school in that parents are very, very active and have very high expectations of us," said Labov, who lives a block from the school and has taught at Powel for 11 years.

Maybe there's something in the water.

Smaller Class Size:

Powel has smaller class sizes. While Philadelphia elementary schools have a limit of 30 students in primary grades and 33 in upper grades, at Powel no class exceeds 26. The school's administration has used federal dollars and other discretionary funds to keep classes small. Parents of children at Powel also have pressed city officials to preserve funding.

News alert for Snyder: a classroom size of 26 is not small. And, small class sizes don't really work all that way anyway.

Stability:

Its teaching staff is stable, cohesive and 100 percent certified. Its student body is very stable, unlike at some schools in Philadelphia, where turnover is a third or more a year.

This is more an effect than a cause. More on this later.

Strong Parental Involvement:

Parental involvement is strong. Parents often reimburse teachers for supplies bought out of pocket, communicate regularly by e-mail with staff and hold staff-appreciation luncheons.

Again, this is an effect, rather than a cause.

An, emphasis on literacy:

In the classrooms, educators say they emphasize literacy and nurture students to care about reading early on.

...

"We have had a very powerful library program," third-grade teacher Sarah Labov said. "Parents have raised money for books. And teachers read stories to kids all the way up through the grades, continuing that kind of passion."

...

Powel also conducts school-wide projects, rich in literacy and focused on a theme. For the third- and fourth-grade biography project, students research a famous person, dress up in character and make a presentation.

All this educational nonsense goes on every day in almost every school in Commonwealth, even the poorly performing ones.

Basically, Powel isn't doing anything special. So why does it perform so much better than the typical Philadelphia school? Snyder actually glosses over the real reason:

The school also has the benefit of drawing students from around the city. About 50 percent come from the immediate Powelton-West Philadelphia area, Holliday estimates. The percentage of students from low-income families is also lower than the citywide average (59 percent versus 74 percent).

This may be news to Snyder, but there does exist a black middle class. And, Powell appears to be drawings primarily from the black middle class. And, no doubt, more than a few parents have some association with the two universities, Penn and Drexel, that Powel is sandwiched between.

And that 59% poverty rate of Powel is more than a little suspect. According to these 2007 PSSA results, only 42 of the 106 students taking the PSSA at Powell were classified as being economically disadvantaged, That's a "poverty rate" of only 39.6%. This also comports with the 2006 demographic data from School Matters which gives a poverty rate of 43.4%. And, let's put that poverty into perspective, here 's the distribution of household income for the Powel School:

Seems like there are not to many black Tiny Tim's attending the Powel school. The data shows that Powel serves a predominantly middle class, not the black "underclass" as Snyder would have us believe. This is consistent with Snyder's observations: lots of concerned parents, strong parental support, stable student body, stable staffing, and kids interested in their education.

Powel seems to be a slight overperformer based on its demographics. And, that slight overperformance might be attributable to Powel's close proximity to two universities and that it has an undisclosed admission policy to weed out the riff-raff.

The most likely explanation is that Powel does well because of its students. The students don't necessarily do well because of Powel. Snyder has the causation exactly backwards. It's as if she's saying wet streets cause rain.

Of course doing well is all relative. According to PSSA results, only about 70% of Powell students are reading at grade level. That means 30% aren't at grade level. Those results are hardly anything to cheer about coming from a middle-class school. That's about the average statewide on a test that's been juiced to comply with NCLB. On the federal NAEP test only about 40% of fourth graders were reading at grade level. I'd say that the NAEP results are much closer to reality.

And the black-white race back is alive and well at Powel. Every single white kid passed the math portion of the PSSA. Only 61% of the black kids passed. Things were better in reading with 75% of the white kids passing, compared to 70% of black students. But even this masks a largish gap. All 75% of the white kids who passed the PSSA scored at the advanced level. Only 23% of the black kids scored at that level.

Powel's teaching methods clearly favor middle-class kids and white kids at the expense of poor kids and black kids. The teaching methods used in Powel are being used nearly every other elementary school in the state. It is these poor teaching methods that aren't working with a large segment of students, especially the poor ones, the black ones, and the Hispanic ones.

Ironically, the very teaching methods Snyder is praising are the root cause of student academic failure.

Employers want new way to judge graduates

USA Today reports that employers want a new way to judge graduates beyond tests and grades.

Colleges have been scrambling over the past year to respond to recommendations from a national commission that they be clearer to the public about what students have learned by the time they graduate.

...

The survey of 301 business leaders nationwide suggests that colleges find ways to assess a student's ability to apply college learning to real-world settings.

Colleges have been dumbing down their curricula for some time now in order to deal with all the academically unprepared students coming their way, not to mention the various bogus fields of study they've invented (interdisciplinary studies, anyone?) now that society thinks that everyone should go to college


As W. S. Gilbert famously put it: "When every one is somebody, then no one’s anybody!"

Seven Random Things

Joanne Jacobs tagged me to write seven random things about myself, so here goes:



1. I am severely allergic to cats. I can tell, and by tell I mean sneeze uncontrollably, if you own a cat by the clothes you wear, especially if you are fond of sweaters. I can also tell if there was once a cat living in your house, especially if your house has wall to wall carpeting, even if that cat died or moved away many years ago.



2. I was once hit and partially run over by a car (which ran a stop light, thank you very much) and managed to walk away from the accident with only scratch (more like a raspberry on my knee), even though I was thrown about fifteen feet. And, it is true what they say about your life flashing before your eyes.



3. I've been online for about fifteen years now. When I started (with CompuServe) it cost $22 an hour to get online. That didn't last too long, soon it dropped to $11 an hour.



4. During this time, I once bought a state-of-the-art 500 MB hard drive, which I thought would be impossible to fill-up, for the princely sum of $400. I thought that was quite the bargain at the time.



5. I am the proud owner of 441 board games, mostly coming from Germany.



6. I'm at least partially responsible for the issuance of over 400 U.S. Patents, including this gem. It's a toothpick dispenser.




7. I used to read Joanne Jacobs blog back when it was called ReadJacobs, in fact that's what my bookmark still reads. Back then that was pretty much the only game in town when it came to education blogs worth reading.




Only took me a month to get around to this. Better late than never I always say.

Unfunded Mandate

Name an unfunded mandate enacted by President Bush.


NCLB? wrong.

The Americans with Disabilities Act. Enacted by Bush 41.

And guess who some of the biggest non-compliers are. Right, the government itself.

For example, the Richmond School District in Virginia.

Nearly every school in Richmond remains out of compliance with the federal Americans with Disabilities Act, two years after the Richmond public school system settled a multimillion-dollar lawsuit by parents with a promise to fix the problem.

So why do I hear so many complaints about NCLB being an unfunded mandate, when it really isn't, and none about the ADA, which is? They're both noble pieces of legislation which are perhaps misguided. What's the difference?

NYC Puts Cart in Front of Horse

The NYT reports:

New York City has embarked on an ambitious experiment, yet to be announced, in which some 2,500 teachers are being measured on how much their students improve on annual standardized tests.

This is a bad idea, at least today and at least in New York City.

In theory evaluating teachers based on student performance is a good idea. But only if the schools are functioning properly in the first place. Imagine this.

Company A and Company B manufacture automobiles. Each company has several factories where cars are assembled.

Company A has perfected its assembly line such that 99% of the automobiles coming off the assembly line function properly and pass all the quality control tests. If the cars rolling off of factories A, B, and C are rolling off at a 99% pass rate, we have good reason to believe that all the workers, including the managers, are doing their job correctly based on the performance of the products. If factory D only has a 75% pass rate, there's good reason to believe that something has gone awry in this factory based on student performance. Furthermore, if the failing cars were tested and it was discovered that most of the failures were the result of the engine not functioning properly, then there's good reason to believe that the guy who assembles the engine isn't doing his job properly. The solution is for manager to can the employee.

Company B has perfected its assembly line. Only 25% of the automobiles coming off its assembly line do not function properly and fail one or more quality control tests. If the cars rolling off the assembly lines of factories A, B, and C are mostly defective, what do we know about the workers in these factories? Nothing. The assembly lines of Company B are defective and they could just as likely be the cause of the failure as the workers. Examining the products rolling off the assembly lines of Company B doesn't tell us much about the ability of the workers. The failures do tell us something about the ability of Company B though. Company B is the failure. The managers of Company B are responsible for the failure, not necessarily the workers. so who cans the managers? The market, assuming Company B was operating in a properly functioning market.

The public schools in NYC are company B operating under the delusion they are company A. That's how we get silly policies like measuring student performance to evaluate teachers. Using this same thinking, why doesn't NYC measure the performance of the schools to evaluate the performance of the management of the NYC school district, starting with the chancellor. You can bet this won't happen because the public schools are a political beast far removed from the free market. The only losers in this situation are the consumers, i.e., the parents and students.

A Year and A half in the Making

While I've been studiously neglecting the blog, the page hits for D-Ed Reckoning have finally surpassed the 100,000 mark.

Woo hoo.

For a point of reference some blogs get that much traffic in a single day. Those would be the ones that post every day, I assume.

I see Joanne Jacobs has tagged me for seven random things. Maybe that will give me an incentive to sit down and write.

Today's Quote

Teaching is very easy if you don't care about doing it right and very hard if you do.

~Thomas Sowell