tag:blogger.com,1999:blog-25541994.post114969038016413030..comments2024-03-26T14:44:37.985-04:00Comments on D-Ed Reckoning: Not So Stinky Research Part IIKDeRosahttp://www.blogger.com/profile/06853211164976890091noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-25541994.post-1152511017060320652006-07-10T01:56:00.000-04:002006-07-10T01:56:00.000-04:00Sorry Catherine, realised I hadn't made a reply to...Sorry Catherine, realised I hadn't made a reply to this point. <BR/><BR/><I>Medicine spent thousands of years doing things that often, once properly tested, turned out to make patients worse.<BR/><BR/>Actually, I think it's fair to say that medical research is quite weak. Every med student is taught that something like....is it 50%? of all studies published in JAMA in any given year are wrong. </I><BR/><BR/>How do they know that 50% of all published studies are wrong?<BR/><BR/>Either the 50% figure is a made-up number, or it is the result of research itself. <BR/><BR/>I suspect the 50% figure arose from research, and refers to initial results that are reported in medical journals. <BR/><BR/><I>From where I sit, that's very bad. Our family has already suffered from medications prescribed on the basis of peer-revieweed research instead of physician experience.</I><BR/><BR/>I've suffered from medications not being prescribed in the first place due to physician experience being wrong. <BR/><BR/><I>Individuals are constantly having wonky reactions to drugs the peer-reviewed literature say works great.</I><BR/><BR/>Medicine can only work with averages. <BR/><BR/>And not all the wonky reactions are the fault of research. My dad had a bad reaction to some medication and then got a call from the pharmacist saying he'd just realised he'd given Dad the wrong medicine.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1150909956394763372006-06-21T13:12:00.000-04:002006-06-21T13:12:00.000-04:00Hey Ken!I have the whole Distar Arithmetic series ...Hey Ken!<BR/><BR/>I have the whole Distar Arithmetic series to look at today!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1150311404438973042006-06-14T14:56:00.000-04:002006-06-14T14:56:00.000-04:00Skimming this thread, I'm inclined to feel that, y...Skimming this thread, I'm inclined to feel that, yet again, the ed schools are the problem. <BR/><BR/>Common sense simply isn't a stock in trade for ed schools. Ideology, politics, social justice - anything but common sense.<BR/><BR/>Medicine is radically diffferent.<BR/><BR/>In medicine, the doctor and the patient have one goal and one goal, only: make the patient better.<BR/><BR/>That's an oversimplification, of course. Doctors and patients can have conflicts over values and goals, too.<BR/><BR/>Doctors also have treatment paradigms that can outlast their usefulness by years.<BR/><BR/>Still, doctors are focused on the patient and what happens to the patient - and they treat patients as individuals, not as classes. <BR/><BR/>Individuals are constantly having wonky reactions to drugs the peer-reviewed literature say works great.<BR/><BR/>So doctors are continually seeing the "exceptions to the rule."<BR/><BR/>Doctors are probably almost forced to become Bayesians, if they weren't going in.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1150311046870528362006-06-14T14:50:00.000-04:002006-06-14T14:50:00.000-04:00TracyMedicine spent thousands of years doing thing...<B>Tracy</B><BR/><BR/><I>Medicine spent thousands of years doing things that often, once properly tested, turned out to make patients worse.<BR/></I><BR/><BR/>Actually, I think it's fair to say that medical research is quite weak. Every med student is taught that something like....is it 50%? of all studies published in JAMA in any given year are wrong. <BR/><BR/>Medicine is frequently said to be an art by "insiders" like me who, unfortunately, spend a huge amount of time with doctors. Doctors say the same thing.<BR/><BR/>Now that I know the name for Bayesian reasoning, I realize that's what I've always looked for in a physician; I look for a person who has "figured it out" through brains & years of experience.<BR/><BR/>Basically, I'm looking for the Alan Greenspan of medicine. (We've found him, btw. Eric Hollander. The Alan Greenspan of OCD.)<BR/><BR/>I think I tried to find the WSJ op-ed talking about Alan Greenspan's brain for you once, didn't I?<BR/><BR/>The new thing in medicine is exactly what we've seen happening for YEARS in education, which is the tyrrany of frequentism.....doctors are now supposed to do "evidence-based treatment." Period. <BR/><BR/>If a doctor's Bayesian brain tells him to do X instead of Y, and Y is what JAMA has been promoting for the past few years, he's supposed to ignore his experience and wisdom and defer to peer-reviewed papers.<BR/><BR/>From where I sit, that's very bad. Our family has already suffered from medications prescribed on the basis of peer-revieweed research instead of physician experience.<BR/><BR/>I'm in favor of research.....but I'm not clear on the limits of frequentist research, and I don't understand Bayesian, case history, or field research well enough to see which kinds of research we should trust to tell us what.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1150306708438198362006-06-14T13:38:00.000-04:002006-06-14T13:38:00.000-04:00Relative error is usually smaller than absolute er...Relative error is usually smaller than absolute error. It's easier to select between choice A versus choice B, but that says nothing about absolute accuracy or whether both choice A and B are bad. One can always limit research to these sorts of choices, but the results, however accurate, may be worthless once you remove the limitations or assumptions.<BR/><BR/>"For example, you're starting a charter school, brand new curriculum, and you have to pick a textbook."<BR/><BR/>Although I would be hard pressed to call this research, the results are all relative.<BR/><BR/><BR/>"Example #2, you're trying to hire a teacher and you have interviewed two candidates, like them equally well and they are identical in all respects except one has an advantage in years of experience and the other has an advantage in education or credentials."<BR/><BR/>This sounds more like assumptions, common sense, and judgment to me, rather than research - the research only being used to justify a value judgment. Imagine writing down an equation defining the merit function for this decision. What are your weighting factors? Whatever will give you the decision you want - all quite scientifically done.<BR/><BR/>"Example #3, you won an unrestricted grant for your school or district and you want to use it for reducing class size, buying a whole school reform package, or teacher training."<BR/><BR/>Choices A versus B versus C. All relative.<BR/><BR/>Research can be done scientifically but still not be worth the powder to ... well, you know. I don't want schools holding up "scientific" research as proof to do whatever they want.<BR/><BR/>"You don't care about the magnitude of difference, you just want a tie-breaker."<BR/><BR/>Maybe schools don't care about external or absolute magnitudes, but, as a parent, I do.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1150292483775005922006-06-14T09:41:00.000-04:002006-06-14T09:41:00.000-04:00ed researcher, your examples are starting to shed ...ed researcher, your examples are starting to shed some light now, but I still disagree with your underlying premise.<BR/><BR/>In education we have three main types of programs:<BR/><BR/>1. Stuff that works well and which usually has large effect sizes associated with its research.<BR/><BR/>2. Stuff that performs as well as the status quo. This stuff usually has small but positive effect sizes.<BR/><BR/>3. Stuff that doesn't work well. Typicially, this is the stuff with no or little research base. This stuff can linger on if it comports with the Ed fads.<BR/><BR/>What you're saying is that it's OK to pick ed programs that fall into group 2 if those are the only programs being considered.<BR/><BR/>What I'm saying is that if there are group I programs available, group 2 programs shouldn't even be considered. If we had such a thing as educational malpractice, picking a group 2 program would be per se negligence.<BR/><BR/>All things being equal, if a school is only selecting programs in group 2, it really doesn't matter which one they pick or what small effect sizes exist between them, in the real world student performance will be about the same or largely determined by extrinsic factors.KDeRosahttps://www.blogger.com/profile/06853211164976890091noreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1150166810956528022006-06-12T22:46:00.000-04:002006-06-12T22:46:00.000-04:00"... and research can only help by things like ind..."... and research can only help by things like indicating whether or not kids' brains are capable of learning alegbra in the 8th grade."<BR/><BR/>What can research tell us when the answer to this question is obviously yes. What percent of kids? Does it matter? This is my point. Assumptions. Is research going to be used to give kids the best lowest-common-denominator education for all, or the best educational opportunity for each individual child? My comments have absolutely nothing to do with whether research should or should not be done.<BR/><BR/>The problem is that some people think that proper ressearch is now the answer to all sorts of educational questions. Rather than discuss assumptions and expectations, some feel that all we need are proper research-based educational programs. This is about control, not research. I'm all for proper research, but not if it is used to continue to keep parents out of the loop. Proper research is no substitute for choice.<BR/><BR/>As for my own personal opinion, I think that most educational problems can be solved by common sense, not research. Define a proper curriculum, set very clear year-to-year standards, and expect both the teachers and kids to get down to work - not play learning.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1150157957999733542006-06-12T20:19:00.000-04:002006-06-12T20:19:00.000-04:00My point is that as a parent, "I don't need no sti...<I>My point is that as a parent, "I don't need no stinking research." </I><BR/><BR/>The downside being that if we (as a society as whole) don't do any research we don't know what works.<BR/><BR/>Medicine spent thousands of years doing things that often, once properly tested, turned out to make patients worse.<BR/><BR/><I>I don't want to wait until someone else decides (for me) what constitutes a good basic education.</I><BR/><BR/>I think it would be useful to make a distinction between two different things here.<BR/><BR/>One is <B>what</B> kids should be taught. <BR/><BR/>The other is <B>how</B> kids should be taught.<BR/><BR/>Research has a lot to do with the "how", but is much more limited in what it say about the "what". <BR/><BR/>So whether schools should be aiming at algebra in the 8th grade is a value judgement sort of thing, and research can only help by things like indicating whether or not kids' brains are capable of learning alegbra in the 8th grade.<BR/><BR/>But assuming that society has settled on a goal of algebra in the 8th grade, research can tell us a lot of useful stuff about how to most kids from 1st grade to passing algebra in 8th grade.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1149950672115912952006-06-10T10:44:00.000-04:002006-06-10T10:44:00.000-04:00KDeRosa, thanks for alerting me to this report. Ye...KDeRosa, thanks for alerting me to this report. Yes, there are other interventions that produce large effects, as I've noted on <A HREF="http://johnl.edschool.virginia.edu/blogs/TeachEffectively/2006/06/08/effectiveness-research/" REL="nofollow">Teach Effectively!</A>, but these are not comprehensive school reform models. <BR/><BR/>I'd rather see an ES > than .25 as the threshhold. Researchers should probably pay attention when there's an average (over dozens of studies) ES of .25, but for practice I'd like to see something double that standard. <BR/><BR/>The Accelerate Schools ES is a little suspect to me, given that it's based on only 3 studies. <BR/><BR/>ed researcher, I agree that cost is an important factor in the reform equation. And, given the choice between a reform model that costs X and produces a negligible effect and one that costs 2X and produces a .5 effect, I'll go for the latter. In most cases, X doesn't equal $0. Communities are already spending lots of education, so adding $200K over three years is a small percentage increase, no?JohnLhttps://www.blogger.com/profile/02180934387309782275noreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1149735294523621102006-06-07T22:54:00.000-04:002006-06-07T22:54:00.000-04:00Affluent parents get to do their own research base...Affluent parents get to do their own research based on a set size of one. The rest have to wait for someone else to do some valid research that will have a significant effect for all.<BR/><BR/>My point is that as a parent, "I don't need no stinking research." I don't want to wait until someone else decides (for me) what constitutes a good basic education.<BR/><BR/>Don't get me wrong, Ken. Reliable educational research is very important in determining what works best. But, "best" for what? Relative improvement for all, or best for each individual child, right now?<BR/><BR/>Actually, I have no issue unless "research" (good or bad) is being used to tell me what to do. My problem is that "research" is used to avoid the fundamental issues of educational assumptions and parental choice. A teaching methodology could have a statistically significant effect, but still not lead to algebra in 8th grade.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-25541994.post-1149731695722144932006-06-07T21:54:00.000-04:002006-06-07T21:54:00.000-04:00it keeps perpetuating the myth that effect size is...<I>it keeps perpetuating the myth that effect size is a unit-free measure that does not depend on ... the reference population [and] the nature of the contrast</I><BR/><BR/>No, I indicated that the research is only valid for the conditions under which the study was conducted. You can't necessarily extrapolate.<BR/><BR/><I>If you compare Curriculum A vs. Curriculum B and they cost the same, but use different philosophies (e.g. whole language vs. phonics; "traditional" vs. "new" math; etc.), then any statistically significant difference, no matter how small the effect size, is meaningful.</I><BR/><BR/>I disagree. Implementing a new program with a small effect size usually results in a real-world effect size of zero. That's why we have the educationally signficant cut-off of 1/4 sd. Chasing smaller effect sizes aren't worth the effort no matter what the cost.<BR/><BR/>Once we get over the 1/4 sd hump, then you're points are valid. All things being equal, you'd rather chose the intervention that cost less per unit of increase. But, interventions having effect sizes significantly greater than 1/4 sd ar few and far between. We're in a "get what you can get" position.<BR/><BR/><I>People often (over-)use class size reduction (using the TN STAR experiment) as their benchmark.</I><BR/><BR/>STAR is a double failure. Low effect size and high cost.KDeRosahttps://www.blogger.com/profile/06853211164976890091noreply@blogger.com