Lots of Data, Set Free
The most important aspect of the plan is data. Useful data must be made freely available to all parents and students so that parents can make sense of the various educational offerings. Parents must be able to answer the following questions:
- What is the predicted performance of my child in this education program based on the past performance of children, like my child, in this program?
- How much better (or worse) is the predicted performance of my child in education program A compared to educational programs B, C, ... , n.
The disaggregation of data made available has to go far beyond that provided under NCLB. For example, a male student with two Hispanic parents - one with a high school degree and the other with a college STEM degree, a family income of $ x, from neighborhood z, with various scores on prior tests taken should be able to drill down through the data and find out similar students like him and see how they fared in a particular course.
Such a system is predicated on accurate demographic data being collected, initial placement exams being taken when the student enters formal education, and all subsequent testing results.
Furthermore, the data should be made publicly available (while respecting FERPA regs) so that organizations, like S& P, can provide services for the public to use to evaluate the various educational offerings.
Maintain the Default System, Allowing Educators to Experiment with Their Top-Down Schemes
Under my plan, in which teachers take responsibility for student learning within the agreed upon obligations agreed upon by the students, there will remain some students that no teachers believe thay are capable of educating. These students must still be provided with a free appropriate education, so an educational system of last resort must be maintained. In all liklihood, states will maintain a scaled-down version of their present public education system for these children (and those parenst who favor the present system and wish to keep their children in it). These children will remain a political problem and states and the federal government will continue to employ top-down edicts and experiments in an attempt to solve the problem. They'll also continue to pump money into the system. None of it will work because the nature of the present precludes such a system working. Nonetheless, eventually some educators will be enticed by the per pupil funding available (the reward) and agree to the risks inherent in educating this kids.
Funding
I've left this one for last because I have no ecific ideas how to employ an equitable funding system. Basically, some kids are much easier to educate than others and, as such, require more effort to educate. This effort is likely to cost more and you'd think there should be additional compensation for those endeavoring to educate these kids.
There's also an argument that the present funding levels are more than adequate to educate most kids.
I don't have an answer for this problem. Nor does anyone else.
I do know how you might go about solving the problem. First you gather all the educators who think they know how to educate these kids. Then have all of these educators bid to take as many schools worth of children that they think they can handle. The losing educators can then be offered to participate in a small scale experiment and take a random sample of children at the winning bid price and see what they can do. The winning bidder will be paid only for the children they are capable of successfully educating. If the winning bidder (and any of the successful losing bidders) is sufficinetly successful, they should be provided any needed funding to scale up.
****
That's it. The minimal framework needed for a bottom-up evolution of the present system within a top-down regulatory framework which provides for a fair environment for professional educators to work and permits for top-down efforts for the difficult to educate children.
I'll discuss how a system might work in practice in my next post.
30 comments:
Looking forward to the "in practice," as I can't even fathom it from the description.
So, to determine what method works you conduct educational experiments on children?
Do you have any idea how unethical that sounds?
Today, schools rarely have a firm scientific basis for what is going on in most of their classrooms. technically, what they do is mostly experimentation, they just don't call it that.
At least I'm honest enough to call it what it is and give parents an option not to participate in the experiment.
Try opting out of your district's new untested education program and see where it gets you.
So, in this proposed system, the feds have set national standards and require all the various educational institutions to test all of their students regularly? I think that's sort of implicit in that it seems required to make the rest of your plan work out, but I just want to make sure I'm understanding you right.
I suppose if you are taking the Fed's money, they'll be able to impose whatever standards they want.
As will many states.
Personally, I'd prefer a system of voluntary standards. Let each group set their own standards and see who wants to adopt them. Think ISO 9000 standards. Or UL stndards. I's like to see math standards set by the "National Society of Math, Science, and Engineering College Professors."
I guess I don't see how the "lots of data, set free" part is supposed to work if you don't have consistent assessments across schools. And consistent assessments imply consistent standards.
I think all the testing in the world, and all the analysis of the results, will lead us on our continuous race toward change for the sake of change. Until teachers are allowed to teach, we will remain in this predicament.
Like I said (in the link you will surely read) double the pay and see who shows up.
Then, once you have great teachers, we can get to the real problem of poverty.
http://thefrustratedteacher.blogspot.com/2008/06/teaching-science-or-art.html"
How about tests like CTBS, ITBS, SAT-10? Even curriculum based assessment have some value. I also suspect that most states would continue to use their existing tests.
You're advocating a lot of unnecessary trouble to accomplish your intent, Ken. (Although, each time you do another reboot, I'm less sure of the intent).
What's your referent for "program."
The "prediction" isn't relevant. The anticipated transparent instructional accomplishment is what is relevant.
You're presuming an interaction between the demographic variables and the instructional accomplishment. With well-developed instructional product/protocol the demographic variables wash out. If a student has the prerequisites for the specified instruction, the instruction should deliver the specified transparent instructional accomplishment. If it doesn't there are "bugs" in the instruction.
Jim Pelligrino has what seems to me a better resolution. He views it as a future,but it's immediately feasible.
www.skillscommission.org/pdf/commissioned_papers/Rethinking and Redesigning.pdf
"We can therefore imagine a future in which the audit function of assessments external to the classroom would be significantly reduced or even unnecessary because the information needed to assess students at the levels of description appropriate for various monitoring purposes, could be derived from the data streams generated by students in and out of their classrooms.
"A metaphor for such a radical shift in how one “does the business of educational assessment” exists in the world of retail outlets, ranging from small businesses to supermarkets to department stores. No Longer do these businesses have to close down once or twice a year to take inventory of their stock. Rather , with the advent of automated checkouts and barcodes for all items, these enterprises have access to a continuous stream of information that can be used to monitor inventory and the flow of items. Not only can business continue without interruption, but the information obtained is far richer, enabling stores to monitor trends and aggregate the data into various kinds of summaries. Similarly, with new assessment technologies, schools would no longer have to interrupt the normal instructional process at various times during the year to administer external tests to students. Nor would they have to spend significant amounts of time preparing for specific external tests peripheral to the ongoing activities of teaching and learning. (p 13)…
"This is exactly the kind of information we now lack, making it difficult to use assessment to truly support learning. The major issue in not whether this type of data collection is feasible in the future. Rather the issue is how the world of education anticipates and embraces this possibility, and how it will explore the resulting options for effectively using assessment information to enhance student learning." (13-14)
TFT, doubling teacher won't accomplish much of anything. Especially when everyone else raises their compensation to retain their employees.
Dick, I'm not assuming that everyone will adopt well designed instruction.
I don't see your point in listing off various tests. Your scheme, as described, depends on students (or, rather, instructional quality) being judged against the same criteria regardless of school. It's all well and good that you're aware of the ITBS, but it is, as it stands, an opt-in assessment. Your scheme, as far as I can tell, depends on a mandatory assessment, which only makes sense in association with mandatory standards.
I get the impression this is a conclusion you're trying to avoid.
Why is an opt-in test test inferior to a mandated test?
The problem is that we don't yet have the tests we want. We haven't been able to get good instruction by mandate and I don't think we're going to get good test by mandate either.
Yes, it would be ideal if we had a set of great tests that every student would take at the completion of each course. But we don't.
There are other ways to skin a cat besides a mandate.
"Dick, I'm not assuming that everyone will adopt well designed instruction."
Who are you assuming will adopt your "reboots," Ken?
That's not a very convincing reply on standards/assessments. It's just hand-waving.
Your argument is, essentially, that it will just turn out that every school teaches to, and assesses against, the same standards - but not by federal mandate! Just because.
Dick, it's one big reboot. And, it has the advantage of giving everyone all the freedom they want provided they can attract students. This lessens the amount of resistance. Having said that, however, it stands virtually no chance of being adopted because it wrests control, and hence power, from those who currently have it.
Paul, there are millions of students in the system. That's a lot of data. Not everyone has to adopt the same test/standard. Eventually some tests/standards will gain popularity and respect from parents/students and schools that don't follow these standards will be at a competitive disadvantage.
"there are millions of students in the system. That's a lot of data. Not everyone has to adopt the same test/standard."
The logic of "Standards/Tests" has failed. It has morphed over 100 years but it has never yielded the anticipated consequences.
The "number of students" in the system is irrelevant. It's the substance and organization of the "datums" that is central.
The popular standardized achievement tests are not "fit for use." When "proficiency" is treated in terms of arbitrarily set "cut scores" the mother of all reboots won't do anything for kids, parents, or citizenry.
It's entirely feasible to have the kind of freedom you're seeing and to also have open Educational Intelligence regarding the operations, costs, and benefits of the el-hi enterprise. The "reboot" involved is at the top, not at the bottom of the enterprise.
"The popular standardized achievement tests are not "fit for use." When "proficiency" is treated in terms of arbitrarily set "cut scores" the mother of all reboots won't do anything for kids, parents, or citizenry."
I still don't understand any of the arguments against standardized testing. Of course, the results need to be handled with care, as does any statistical data. But I yet to have anyone propose a schema that makes more sense, in terms of efficiency, ease of use, etc.
I understand the argument against "cut scores". But you can use distributions. Also, the test data will tell you where things are working, and where they aren't.
Those who argue against test scores won't change their mind because of this post, just as I haven't changed mine. but perhaps we can plant a seed that says "standardized testing may not be the ultimate, but it's better in terms of ease of use and effectiveness than other alternatives . . . so let's just train people on HOW to use standardized tests."
But maybe not . . .
Hey, RMD, I'm not "opposed to standardized achievement test scores." I worked on the construction of several and I've administered thousands.
For prediction purposes, the tests are quite "fit for use."
For RELATIVE comparisons of educational administrative units in terms of general ability" they aren't very sensitive, but they aren't "unfit for use."
Standardized personality tests, interest tests, and other tests that measure "latent traits" ARE fit for use.
But academic expertise in reading, math, and other school subjects is simply not a "latent trait" any more than transparent performance in other areas of life is. At one point, these matters were considered to be "faculties of the mind," but that notion was abandoned decades ago.
"standardized testing may not be the ultimate, but it's better in terms of ease of use and effectiveness than other alternatives . . ."
Yeah, that's been the standard argument, but it doesn't hold water.
There ARE alternatives with equally sound psychometric foundations. Their application has been demonstrated to yield transparent results that are easier to apply and that yield much more productive information.
If we want "Educational change we can believe in," it's simply not in the standardized achievement test cards. Reliance on the tests as the basis for Federal educational policy has been entirely without success. The latest policy failure is documented in the Non-Impact of the $6 billion investment in Reading First. The findings of the $100 million evaluation were quietly buried in the last days of the Bush Administration and have not yet been discovered by the Obama Administration:
http://ssrn.com/author=1199505
Where's "responsibility" and "transparency" when we could really use them?
One option to solve the "funding" issue: Implement early age intervention programs that lower the lifetime costs of educating struggling learners. One of the biggest problems with the drive toward testing and outcome based school ratings is that it limits the vision or willingness of schools to implement programs that don't align specifically with the tests they are forced to deliver.
i dont know when you guyz are going to stop the experiments on education and students. its a BIG question for all of us that when we will be mature enough in education that we don't need further experiments.
KD:
Are you familiar with the Lorenz Curve?
Did you know test scores of 13 year olds have the same distribution as wage inequality later in life?
Trying to get my head around this. Maybe you can read up and do a post on it?
http://www.econ.queensu.ca/working_papers/papers/qed_wp_947.pdf
Here's another datum for the voucher'ers and charterist's to process.
www.jsonline.com/news/education/41868652.html
Study of Milwaukee voucher schools reports essentially no difference in test scores and equal parent satisfaction in both government voucher and government public schools.
How much evidence will it take to convince people that school vouchers and charters are metaphorical snipe?
(anonymous): "So, to determine what method works you conduct educational experiments on children? Do you have any idea how unethical that sounds?"
(Character education): "i dont know when you guyz are going to stop the experiments on education and students. its a BIG question for all of us that when we will be mature enough in education that we don't need further experiments."
Education technology evolves. Curriculum changes. Society at large evolves, and the possible careers into which children will move shift. Unless some dictator has access to divine inspiration, and can tell which technology and which curriculum will work to prapare which children for which careers, experiments will always be necessary. What's unethical is to compel participation in any particular experiment. Vouchers reduce compulsion by giving to parents the power to select the experiment in which their children will participate.
But academic expertise in reading, math, and other school subjects is simply not a "latent trait" any more than transparent performance in other areas of life is.
Well transparent performance in other areas of life are normally a "latent trait". For example, if you observe me driving a car safely and competently on a motorway in fine weather you don't know whether I can drive a truck in the snow. If you observe me reading and comprehending "The Lion, The Witch and the Wardrobe" you don't know whether I can read and comprehend "The Open Society and Its Enemies". You need to test my abilities to do those different tasks separately.
But if you know my height is 5'2" you instantly know that I'm shorter than 6'0" and taller than 4'5". I can't think of any area of performance that isn't a latent trait.
At one point, these matters were considered to be "faculties of the mind," but that notion was abandoned decades ago.
What bit of the body do they (whoever they are) now consider these matters to be? And if they don't consider them to be faculties of the mind, how do they explain the varying effects of brain injuries?
There ARE alternatives with equally sound psychometric foundations. Their application has been demonstrated to yield transparent results that are easier to apply and that yield much more productive information.
So far the only alternative method you have ever mentioned in my reading does not meet the criteria of transparency or yielding more productive information. The method you have given of testing reading achievement is to give the child a passage to read and ask them questions to check their comprehension. The problems with this method are:
- it does not control for variations on the part of the tester. Different testers can give different test passages, and have different standards of toughness for what it means to comprehend a passage.
- it doesn't control for biases on the part of the tester. Biased observation is a serious problem in medical research and even in physics research.
- it doesn't scale. If you live in a country where the population is in the millions it is just impractical for every individual who is interested in educational performance to go around testing every single kid themselves.
The Government-Academic-Testing complex have made achievement testing a virtual mountain out of a transparent molehill. Take "driving" for example. There is no reason to be concerned with "driving ability" as a "trait." It's transparently obvious that it's task dependent--type of vehicle, weather, "road" and so on. And the driver must have some minimal prerequisites--age (as an indication of "responsibility," vision, and background information about driving conventions and laws.
These are readily "tested" in a transparent way in few minutes. If you don't "pass," for whatever reason, you can come back and take the test again.
Certainly, drivers can be scaled from Novice to NASCAR, but there is little reason to do so.
They can also be categorized in terms of "risk." Insurance companies do so, but again the indicators are transparent rather than latent.
Exactly the same logic is applicable to reading and to other foundational academic matters of interest to individuals and citizenry.
Fancy psychometrics do have a place.
And there is a requirement for aggregate information re instructional condition and status. But this need not be "full census" information, relying on a couple of different tests administered at widely separated grade 3 times cross-sectionally, rather than longitudinally to cover the entire el-hi instructional span.
One would think that the notions of sampling and verification by
replication had not been invented.
These "tests" are not even available for public inspection. That's about as un-transparent and ir-responsible as one can get. And the results of the ungrounded measures is "proficiency" defined in terms of arbitrarily-set cut scores.
The WiseOnes say, "That's the best that can be done." "Everyone" says, "OK."
It's kids' individual well-being and the future of the citizenry that's riding on which multiple-choice bubbles are marked. That's probably not worth getting concerned about.
Where is "rebooting" when it would be feasible and highly beneficial?
There is no reason to be concerned with "driving ability" as a "trait." It's transparently obvious that it's task dependent--type of vehicle, weather, "road" and so on. And the driver must have some minimal prerequisites--age (as an indication of "responsibility," vision, and background information about driving conventions and laws.
Dick, the reasons we are concerned about driving ability is that bad drivers disrupt traffic flows and impose a threat to the cars/bikes/buses etc and lives of other people on the roads.
Also, is it really the case that where you live someone can get a full driving licence without actually having to demonstrate an ability to drive a vehicle of the type the licence is for? Where do you live? In NZ, you need to pass some minimal prerequisties of the type you discuss to learn how to drive (which means that you can drive a car/truck/etc with an experienced driver in the front passenger seat, obviously this doesn't apply to motorbikes but there are other restrictuions). After that you sit a second test where you show that you can drive a car/truck/motorbike/etc.
Exactly the same logic is applicable to reading and to other foundational academic matters of interest to individuals and citizenry.
Of course. This is why we want standardised achievement tests for reading and other academic skills like we have for driving.
And there is a requirement for aggregate information re instructional condition and status. But this need not be "full census" information, relying on a couple of different tests administered at widely separated grade 3 times cross-sectionally, rather than longitudinally to cover the entire el-hi instructional span.
Regardless of whether you measure longitudinally or every 3 grades, you need standardised achievement tests to do so reliably if you live in a country which is too big for every interested person to test every student personally. And if you want to determine the relative effectiveness of different programmes you really need to control for observor bias.
One would think that the notions of sampling and verification by replication had not been invented.
And the relevance of this to standardised achievement tests is.
These "tests" are not even available for public inspection. That's about as un-transparent and ir-responsible as one can get.
Actually no. Your proposed method of trotting over to the school and just asking kids to read is more untransparent and irresponsible, for the reasons I pointed out earlier.
I will also note that this idea that tests are not even available for public inspection is a country-specific claim. In New Zealand the test questions are publicly released after the exam and the answer papers are sent back to the relevant student. There is nothing stopping the USA from adopting such a method. The flaws in your claimed alternative are far more inherent.
And the results of the ungrounded measures is "proficiency" defined in terms of arbitrarily-set cut scores.
This depends on the test. The NAEP, as I pointed out to you earlier, gives reasons for their setting of the scores.
And it is very very weird that you keep going on about "ungrounded measures" and "arbitrarily-set cut scores" when your proposed method has absolutely nothing to tell observers about what questions they should ask, and what answers they should accept.
Sorry, Tracy. You're misinterpreting everything I said.
Scratch, my contention we had reached agreement on a point. We have no agreement on any point that I can identify. I see nothing to be gained by rehashing your statements which you have not supported with anything but your opinions.
Sorry, Tracy. You're misinterpreting everything I said.
Thank you for the apology Dick, but can you now explain what you were trying to say? I am still curious.
I see nothing to be gained by rehashing your statements which you have not supported with anything but your opinions.
I am rather surprised that given your attitude here, you didn't bother to provide any sources for your earlier statements. If I am to be ignored because I didn't provide sources for my assertions, why do you expect anyone to listen to you?
I asserted that the NAEP gives reasons for their setting of their scores. I did not bother repeating the source for this as I had already provided this to you before, my apologies for assuming that you would remember being shown evidence contrary to your assertions. Link: http://nagb.org/publications/frameworks.htm
The evidence that driver licensing tests require actually demonstrating driving skills in the countries I have lived in is available at
http://www.landtransport.govt.nz/factsheets/03.html (it takes a while to cover all the material)
United Kingdom:
http://www.direct.gov.uk/en/Motoring/LearnerAndNewDrivers/PracticalTest/index.htm
Massachusetts USA
http://www.mass.gov/rmv/license/5classd.htm
Publication of NZ's standardised achievement tests for the National Qualifications are available at http://www.nzqa.govt.nz/ncea/search/types.html
I didn't bother sourcing this before because I assumed that on such a factual matters my word would be sufficient, and also you hadn't queried my assertions about NZ practices before.
Research on the value of driving skill (as oppposed to merely the aptitude characteristics you listed such as age, eyesight, knowledge of driving rules) is a bit harder to come by as everywhere I know of in rich countries requires practical driving licence tests and poor countries don't do much social science research. But see page 4 of this pdf http://www.dft.gov.uk/pgr/roadsafety/research/rsrr/theme2/thedifferentialeffectsofform.pdf from the UK - they report their finding that errors fall with driving experience.
See http://skepdic.com/control.html for a description of why we control for observor bias. The observer bias problem I referred to earlier I did not provide a source for because I assumed that it was common knowledge, especially for someone like yourself who claims to know something about standardised achievement tests. This is my mistake, I keep overestimating your knowledge of testing. I think it's the confidence with which you make your pronouncements that keeps misleading me. I will try to do better next time, but my past failures make me skeptical about the future.
As for my assertion that it is not practical in a big country for an interested individual to test every single student's reading ability, I will provide some calculations. In New Zealand, which is not a big country by world standards, the statistics department estimates that there are 287,680 people between the ages of 5 and 9. See http://www.stats.govt.nz/NR/rdonlyres/64350493-1DDF-47D6-BCCA-95899516ABCA/40602/npedec08qtralltables1.xls
Assume that it takes 6 minutes to assess each one's reading skill (remember that you will need to travel around the country to each school). Working a 24 hour day, it would take you 1,198 days to test every child in the 5-9 agegroup, or over 3 years. Practically of course you can't work a 24 hour day - sleep is neccessary and it is possible that a number of parents would not let you wake their child up at 3 in the morning for a reading test anyway. By the time you had tested every 5 to 9 year old it would be time to start over again. This is why I say your proposed method is impractical.
Now you could use a sampling method, but you would still face all the travel time between the different schools.
Now Dick, I've provided my sources, you can provide yours. Where do you live where driving skill, as opposed to aptitute matters like eyesight and age, isn't tested?
What are those alternatives with equally sound psychometric foundations?
You asserted that matters wer econsidered to be faculties of the mind, but that notion was abandoned decades ago. Can you provide some empirical support for this assertion? Can you answer my earlier questions "What bit of the body do they (whoever they are) now consider these matters to be? And if they don't consider them to be faculties of the mind, how do they explain the varying effects of brain injuries? "
http://ies.ed.gov/ncee/pubs/20094050/pdf/20094050.pdf
New US Institute of Education Sciences study of Wash DC vouchers (called "Scholarships) after 3 years.
Not much to there to support the claims of voucherists and free marketeers. A hefty portion of those awarded vouchers never used them and a larger number weren't using them after three years.
A puny difference in Stanford9 reading scores. None in Math.
Voucher parents were more satisfied with schools, but no differences in kid satisfaction.
More than half the vouchers went to Catholic and "faith-based" schools.
The report says nothing about what went on in either "treatment" "control" --one black box compared to another black box.
The mindboggling finding to me, which the report overlooks is that 70% of parents in both groups rated their kid's school as A or B.
Yet in neither treatment or control in neither reading or math was the average %ile rank more than the upper 30s or so. There are lots of kids being taught very little. But parents aren't told so and researchers and the media don't notice.
Post a Comment