November 3, 2009

A Tale of Two Cities

Now that Canada's largest school district, Toronto, has taken the bold step of reporting its testing data for racial and economic subgroups, we are able to compare its achievement gaps with the achievement gaps of other large cities that also report achievement data for subgroups.

Let's compare Toronto to Philadelphia and see who has the larger achievement gaps.  Let's see how Toronto with it's compassionate and generous Canadian-style social policies compares to Philadelphia with it's backward and stingy U.S.-style social policies.

Better yet, I'm not going to tell you which city is which, see if you can guess from the achievement gaps which I've given as a fraction of a standard deviation.*  Negative numbers indicate that the subgroup performed worse than whites; positive numbers indicate that the subgroup performed better than whites.

City #1
Black-White Achievement Gap: -0.85
Hispanic-White Achievement Gap: -0.73
Asian-White Achievement Gap: +0.28

City #2
Black-White Achievement Gap: -0.73
Hispanic-White Achievement Gap: -0.70
Asian-White Achievement Gap: +0.14


Which city is Philadelphia and which is Toronto?

Bonus Question:  What does this analysis suggest for improving results and policies drawn on international comparisons between diverse countries like the U.S. and more racially homogeneous countries like those in Europe and Northeast Asia.

*What I did was to convert the percentile scores reported into z-scores (which is a more accurate way of analyzing normally distributed data) for each subgroup for math and reading combined  (sixth grade for Toronto and fifth grade for Philadelphia).  Then I calculated the difference between the z-scores of Asian (East), Black, and Hispanic (Latin) subgroup and the white subgroup. A difference of 0.25 is considered to be educationally significant.  A difference of 0.75 is considered to be a large difference in the social sciences; most interventions are not capable of remedying an effect size of this magnitude.

16 comments:

Dick Schutz said...

Ken, the ethnic populations of Toronto and Philly are very different. One can obviously run the stat analysis, but it's a dog's dinner. Neither city is reliably teaching kids to read and do math, and the tests used in both cities are flawed. Trying to figure out which city is worse is hardly worth the time. Particularly when you could be telling us the dinosaur story.

You overlooked the really interesting news in the Toronto report.

More than 80% of all parents are happy with the school their kid is attending. There is little difference across ethnic groups.

87% of parents expect their kid to go on to University! There is some difference across ethnic groups but not all that much.

The findings about preschool attendance and how much parents help kids with homework are also "provocative."

What is actually going on could be teased out pretty straightforwardly. But no attention is given to the instruction the kids are receiving.

It's in the instruction. Throwing insensitive tests against a black box is closer to a tail than a tale.

KDeRosa said...

Neither city is reliably teaching kids to read and do math, and the tests used in both cities are flawed.

That's the point, Dick. Neither city is doing a good job teaching and the tests are largely measuring SES/IQ. See Northof49th's comment in the previous post.

Sorry about the Dinosaurs, the second insure post is nearly done.

North of 49th said...

Actually you got it backwards, Ken. I can't speak to the Philly tests, which may well measure SES and/or IQ. But the researchers found that most of the variance in the EQAO tests was due to school (instructional) factors, *not* IQ or SES.

That's how a school with children of lawyers, doctors and engineers can score much lower than a school full of new immigrants.

Not all that surprising when you consider that the test measures mostly what is taught in school, no background knowledge, general vocabulary and so on.

KDeRosa said...

Northof49th, in your previous comment in the last post, you indicated that the EQAO test was sensitive to SES when the instruction was poor and that the instruction was mostly poor in TSD. I just followed this out to it's logical conclusion.

I'd suspect that some schools do a decent job teaching in Toronto. Some schools in Philadelphia also do a decent job teaching. But for many of the schools the instruction is likely to be haphazard and under such conditions, SES/IQ differences will differentiate student achievement.

Northof49th said...

The difference is that even high IQ or SES won't get you far if you have to construct stem-and-leaf plots and you don't know how to do it because you weren't taught. The EQAO tests measure what is taught in the school curriculum. Some high-SES schools do rather badly. They do lots of exciting project learning or what-have-you but do not actually teach the curriculum very well and the tests reflect this.

Their students would probably do well on a standardized achievement test, because those *do* tap into background knowledge, vocabulary and general ability to a large degree.

KDeRosa said...

Then I would think that math tests, which should be less reliant on background knowledge (let's ignore the reading comprehension issues for now), would be a better predictor of instructional efficacy. Reading tests generally get infected with background knowledge issues. But, the Toronto data shows more between-group variance in math than reading performance. Whereas math performance should have had less variance because the SES/IQ issues should have been at least partially neutralized.

Downes said...

Hm... a commenter called "Northof49th" (or "North of 49th") that has (at least, according to Google) never commented before, that we're supposed to think is Canadian, but who agrees implicitly with DeRosa, and who writes in deRosa's exact style...

KDeRosa said...

Stephen, you need to adjust your tinfoil cap.

Do you like the way my sockpuppet disagreed with me a few comments back? That was pure genius on my part.

All I know is that "North of 49th" seems to know a lot more about the specifics of Canadian education than either one of us.

KDeRosa said...

One more thing, Stephen: thanks for unwittingly serving as my strawman sockpuppet.

Northo 49th said...

As a matter of fact, I have commented on this blog before (not often), and on a few others as well, using Northof49th as my handle. I can't answer for why it doesn't show up on Google.

I can claim to know more about Canadian elementary education than either of you. I have taught in two different provinces (one in the Maritimes, where Downes now lives), and did student teaching for the Toronto Catholic DSB, so I have some familiarity with TO. Currently I teach two sections each of Grade 6 math and science at a 6-8 middle school in a southern Ontario district. In addition I have been involved in EQAO grading and development of anchor papers and rubrics as well as Ministry of Ontario curriculum "Exemplars." I've written some science and math curriculum materials used in Ontario schools and also some technology initiatives.

I've also taught lower grades, ESL and Gifted withdrawal programs.

And I don't look like Ken De Rosa, or deo gratias, Stephen Downes either. But I will take it as a compliment that I write like De Rosa. He is a smart guy even if I disagree with him about specific things. He also occasionally shows flashes of modesty, something I doubt anyone has accused Downes of doing.

Northof49th said...

Ken, as far as the math variability goes, it's hard to interpret. The students' scores do not depend on their math ability or proficiency so much as on their writing ability. To score a 3 or 4, the kid must write extensive essays to explain how he solved the problem and what his thinking was. Little arithmetic mastery is required or rewarded; calculators are used throughout, even in Grade 3.

It does not matter if the student's answer is correct, either. A student who solves a problem correctly, but does not "explain" ad nauseam will get no higher than a 2. However, the student who waxes prolix can get a 4 provided he or she uses a lot of the words required: vertices, attributes, variable, isosceles, commutative and such like.

Students can be taught techniques for scoring higher on the EQAO tests, but EQAO does not want that fact bruited about. One school where all their Grade3's got a Level 4 were investigated and it was discovered that there was no cheating, the teacher had taught the students a type of paragraph frame to use in solving and "explaining" math problems. I only know this because my colleague in one of the EQAO sessions I participated in was the investigator on the case.

Students with very strong math skills were marked lower for failure to "explain," and if we found a student who wrote things like "this is stupid -- everybody knows you multiply to find the area" you were supposed to flag the book to be handed over to the "EQAO police," as we (half-in-jest) called them.


I had two lieu days off this week so I had time to join this thread, but will shortly revert to read-only mode.

North of 49th said...

I found one of my previous comments here

Lapsus keyboardii, I left out the "th". Better register for a Google account, at least it will bring up the same username each time.

KDeRosa said...

To score a 3 or 4, the kid must write extensive essays to explain how he solved the problem and what his thinking was.

There goes your problem right there. I'd like to know how "show your work" got twisted into "explain how you solved."

found one of my previous comments here

Clearly, I was playing the long con all along.

Downes said...

> I can claim to know more about Canadian elementary education than either of you.

So long as you remain anonymous, you cannot claim any credential whatsoever (or, perhaps more accurately, you can claim whatever credential you want, who's to know?).

Dick Schutz said...

Hey, Stephen. What if north of's real name is "Anonymous." Anyone on the Internet could be a dog, for tht matter and the posts wouldn't be any different. Do you believe that all names have to be discovered and that there are higher order names?

I don't know anything about the Toronto schools, but the Philly schools are a mess--as are all "great city" schools, Philly is introducing Reading Mastey and Corrective Reading, but it's too little, too late, and being porrly implemented.

What steps is Toronto taking on the basis of the report? Nothing that will make any difference. The tests don't provide any basis for that. But we just keep on tesing and reporting.

palisadesk said...

I don’t think the comparison between Toronto and Philadelphia (or any other US city) is particularly instructive, since so many underlying factors are not comparable. What would be more illuminating would be some way to quantify how and why the long-term outcomes for minority kids in Toronto (and other Canadian cities) are so much better than for similar demographics in the US. For whatever reason, Canadian cities do not have anything like the established underclass that most US cities do.

Others interested in these issues may find some books I read recently as thought-provoking as I did. You can get them on Amazon, but there isn't much info there; I found the author's website here

Adams is with the Environics polling organization and has spent several decades studying different issues and people's opinions and values on both sides of the border. Much of what he reports surprised me and all of it was worth reading and reflecting on. He writes in a lucid, engaging manner and supports his theses with data and references to other sources for further exploration.

For a compelling comparison of the U.S. and Canadian values and attitudes, read Fire an Ice: The United States,Canada, and the Myth of Converging Values

(a review is here )

and also Unlikely Utopia: The Surprising Triumph of Canadian Pluralism . Both books were riveting reads. I passed them around at work and got similar reactions from several others. While not about "education," per se, they do flesh out some contextual factors which bear on the quest to improve schooling and outcomes for kids, especially at-risk kids.

Another Adams volume, which was not about Canada but did expound on an issue related to school reform, is American Backlash. Here he develops the theme that disengagement from political processes and public life generally is a threat to our democratic heritage, and that the difference bwteen "left" and "right" is much less than the difference between the engaged and the disengaged.

Read it, you won't be sorry.