November 10, 2010

The Instructional Black Box

I'm in the process of wading through the 100 or so comments generated in my five part Economics for Edupundits to address the issues raised. But, in the meantime I wanted to address the following issue raised by Dick.

Who is examining the nuts and bolts of instruction—which are the determinants of formal instructional accomplishments—or the lack thereof? No one. Well, almost no one. But that’s a whole nother story.

By and large in the US, instruction remains a black box [scratch black, make it a white box] between Standards and Standardized Test.
Dick is, of course, right.  Instruction largely remains a black box.  But why does it remain a block box?

The answer is that there is little incentive or reward in our present system for cracking open the black box.  And, here's why.

Students can be divided into the easily educable and the difficult to educate.  Let's deal with the former group first.

For the easily educable, there's no real need to crack open the instructional box.  These students are going to learn regardless of the instruction being delivered. There's no need to understand why your instruction seems to be working with this group.  Just accept that whatever you do, provided it's not too loony, will work pretty good and pick something and a style you find personally enjoyable.  If you're likable and organized, the students will like you and life will be good.  Sit back and collect your paycheck and enjoy your summers off content with the knowledge that the measuring stick used to determine what the students have learned is a glorified IQ test, the SAT or ACT.

If you run a private school, charter school, magnet school, or any other school with selective admissions, the situation is largely the same.  Your main goal is to weed out all the difficult to educate students and only admit  the easily educable students, preferably the easiest to educate and hence the ones most likely to perform well on the SAT/ACT.  The better you can separate the wheat from the chaff will largely determine your reputation and therefore the tuition you can charge.  If you need to differentiate yourself from your competitors, there are many non-instructional areas that can be attacked much more easily than the instructional areas.

There is so little competition in the current system, schools have largely adopted the carve up the market approach, establishing their little minimal competition, non-profit fiefdoms and enabling them to leased a comfortable existence.

Don't expect innovation to come from this segment of the market.

Now for the difficult to educate students, this is where cracking the instructional black box would bring the most benefits.  But where is the incentive to actually do so?

There is no incentive at the political level.  Political careers are too short.  Long-term accomplishments (like instructional reform) don't matter.  The game is to appear to care, enact some nebulous reform, throw some money at the problem, and hope you are out of office (or have moved onto a better office) when the chickens come home to roost.

And the reforms aren't going to come at a lower level because below the political level everyone is a wage slave.  The wage slave's only incentive is to collect the next paycheck.

Instructional reform requires entrepreneurship.   But what incentive does a Vanderbilt have to enter the market to compete with all Fulton and Livingston monopolies?

Reform won't happen at the curriculum level because there is no incentive for schools to adopt a better curriculum or for schools to not subvert the better curriculum if forced to adopt it.  Many good curriculums are languishing in obscurity due to this very reason.

So, tell me all you Kenysian market steering types, I'm looking at you Dick, how do you propose steering the educational ship to not only unlock the secrets of the instructional black box, but also to assure that schools follow the revealed secrets instead of, you know, sticking with the status quo?

25 comments:

Dick Schutz said...

I wouldn't claim to be either a animal-spirited Keynesian or a freedom-wishing free marketeer. I don't know enough econ to join either side, and I'm not learning much from your lectures. But from what I see, the Keynesians seem to be winning in the US.

Incidentally. I concur with a third of your analysis, Ken. Some kids learn despite mal-instruction. We need to be concerned about these kids because we (should) want all kids to be what they can be. The schools don't deserve any credit for their accomplishments to date, but they do take credit. This is akin to the medical trade taking credit for the healthiest.

A second group limps along through the mal-instruction and does well enough to score about a low cut-score on a test. They are labeled "proficient" and the schools take credit for this also.

The third group is shuffled through "interventions" through a trial by ordeal until they can be shunted into "special education."
The instructional failure is attributed to the kids and/or their genes. Couldn't possibly have anything to do with the instruction the kids received.

But opening the white box of instruction won't be done by manipulating incentives. It will be done in the same way that improvements are effected in every other sector of life--by comparing the differences in reliability of achieving a specified aspiration and the time and cost of the achievement.

It's quite feasible to do this.
The basic logic was developed in the early 1960's with investigations of First Grade Reading and Follow Through. Planned Variations. The methodology was dropped for political reasons that Engelmann has been complaining about on his trips to the bank ever since.

The methodology can be updated and implemented today as Natural Educational Experimentation.

There are only a few requirements to play the game.

One. There has to be a transparently observable means of demonstrating the accomplishment. No stinkin standardized test, but something everyone can see the kid has been enabled to do. If instructional competitor can agree on this, there is a foundation for competition. If they can't agree on any such all the talk is empty rhetoric.

Two. Each competitor has to have an instructional product/protocol. This can be anything from necromancy and witchcraft to a script that the instructor is to follow to the letter.

Three. Each competitor has to have between a set of between 5 and 11 Key Performance Indicators to mark the progress of the instruction from beginning to end.

These three elements provide the basis for Instructional Intelligence analogous to Business Intelligence--a hot item in the Corporate world.

The Experimentation is natural because teachers and schools have only to specify which "model" they're adopting. The rest is transparently observable.

The reliability of effects can be analyzed for the usual demographics of interest and competing models/programs can be compared in terms of reliability, time and cost.

The methodology knocks in the head the common belief that "the teacher is the most important variable. The instructional task is the most important variable. Basket weaving is more easily taught than is teaching kids how to handle the conventions of written English or the number system.

The next most important variable is the tools/artifacts=product protocols to get the job done.

Teachers are not unimportant, but with few exceptions all teachers who have gone through 4 or 5 years of college are "qualified" to teach elementary school kids, and it doesn't take that much more specialized training to teach all pre-collegiate kids.

That's how I'd go about it. There may be better ways. If so we can have a competition among methodologies. The logic here will certainly beat the current logic of Standards and Standardized Tests.

kieran@coredogs said...

I'm betting time and money that there are people who will reward someone who opens the box. I've created CoreDogs at http://coredogs.com (plug), an online text for intro Web tech courses. It uses lessons from learning science to help learners use their time efficiently.

Yes, it includes an instructional protocol - I agree that that is important. CoreDogs (plug) relies on frequent feedback, something we know works. It has a streamlined workflow to reduce the time (and hence cost) instructors spend giving the large amount of feedback that the CoreDogs' instructional model requires.

BTW, the black box I opened includes not only the learning process, but the administrative apparatus surrounding it, and the time and money constraints all parties have. Real solutions need to honor all real constraints.

Given how inefficient much teaching is (I work at the higher ed level, BTW), it's possible to improve instruction and reduce cost at the same time. But only if you're willing to open the pesky black box, look at the details of learning, and create content and tech support for efficient learning processes.

The devil is in the details. You need to look the devil in the eye, and fight him on his own turf.

Right now, CoreDogs (plug plug, pluggity plug) is free. I hope to make $ with it someday. My value proposition is basic: more learning at lower cost. That's achieved by studying learning in depth, and creating tools and processes that support it.

There's a short page on the CoreDogs Way at:

http://coredogs.com/article/on-line-textbook-large-classes

Kieran
http://coredogs.com
(Plug)

Dick Schutz said...

Sorry, Kieran. Good try, but no cigar.

What you have is a product/protocol for teaching a course related to techniques of website creation. Personally, I like the notion, but I don't think this is the matter Ken is addressing, and it's nowhere near what I'm addressing.

Roger Sweeny said...

I'm not sure where this fits in but it seems to me that there are foundational skills that are constantly used, like reading and writing, and then there is subject matter knowledge, which may never be seen again after the course is over.

Acquiring skills means that the student has to be able to do something. Acquiring subject matter knowledge means that the student has to know something. Actually, it means that the student has to know something at the time he is being assessed but can forget it thereafter. Most do, though we like to pretend that they do not.

Dick Schutz said...

Actually, it means that the student has to know something at the time he is being assessed but can forget it thereafter. Most do, though we like to pretend that they do not.

Yep. But the Internet is a game changer here. There is nothing that you can't find more about than you care to know. Just as written language extends our collective memory, the Inter net extends it a quantum notch more.

This doesn't that we don't have to have any information in our head, but it means that we need to have more and different academic procedural knowledge and less and different declarative knowledge.

The Standardistas and Coreists are addressing a pre-Internet age. Re math, they regress further and address a pre-calculator age.

There is some "math" and some "substantive information" that should be "in kids heads as a matter of personal independence and social responsibility. But it's anachronistic to attempt to teach these matters encyclopedic- inclusive.

Parry Graham said...

Ken (and others),

Do you see the increasing focus on student growth, as opposed to proficiency/mastery, as having any impact on this? In my school, I am far less interested in how many of my students are proficient than I am in what percentage of students demonstrated academic growth over the course of the year.

Might this create increased interest in describing the black box for both the higher achieving and lower achieving kids?

Parry

Dick Schutz said...

I am far less interested in how many of my students are proficient than I am in what percentage of students demonstrated academic growth over the course of the year.

Hmm. There is less to "proficiency" and "growth" than meets the eye.

"Proficiency" has been trivialized to an arbitrarily-set cut score on an ungrounded statistical scale.

"Growth" is in terms of difference scores on these ungrounded numbers--adjusted for SES.

These practices fog the fact that although "gains" are being reported, the kids haven't been taught to read or do math and will likely never be taught.

The practices shed no light on the instruction the kids received. That remains the "black box" of Ken's post.

Parry Graham said...

Dick,

Some growth measures do factor in SES, others do not. In North Carolina, SAS’s EVAAS tool, which was built by Bill Sanders, uses an algorithm that factors in a student’s prior standardized test scores to predict future performance, but SES is not a factor in the equation. The extent to which actual performance varies from predicted performance creates a growth number. At the individual student level, there are clearly problems with trying to draw valid conclusions from these scores. As larger groups of students are used as the data pool, however, it is possible to draw more valid conclusions.

For individual teachers, EVAAS produces a “value-added” report. It is difficult to draw much in the way of conclusions from one year’s worth of data, but trends over time can paint an interesting picture.

I understand what Ken means by the black box. My post was in response to his point that there is little incentive to cracking open that box. If value-added scores become a part of teacher evaluations (as is happening in some districts and states), that might create some incentive. In other words, if demonstrating student growth on standardized tests becomes a part of my evaluation (whether I am a teacher, a principal, a superintendent, etc.), then I have an incentive to figure out what types of instructional practices are most likely to lead to growth.

All of this is of course predicated on the assumption that standardized tests can reliably provide insight into what students know and are able to do, and that the tests are reliable enough to be able to document true changes in student learning over time, but I wonder if the focus on “value-added” scores might work as an incentive to figure out what leads some teachers to get better scores. Whether you like standardized tests or not, I think it is hard to make an argument that a teacher who consistently helps kids improve their performance on a standardized test from the start of the year to the end of the year (i.e., growth), especially relative to peers, isn’t doing something right.

Parry

mmazenko said...

Having taught for years in Asia, I would argue that the same black box is in place in most systems around the world - but to much greater effectiveness.

The reason may be a free market answer. Those school systems have serious student accountability and a system based on merit and competition to progress to higher levels.

While I find other systems to be too draconian in their restrictions on higher ed, I often worry about our permissiveness, not to mention our single-minded focus on bachelor degrees for all.

Dick Schutz said...

All of this is of course predicated on the assumption that standardized tests can reliably provide insight into what students know and are able to do, and that the tests are reliable enough to be able to document true changes in student learning over time

Yep, And that's a set of black bubbles within the black box. The test items are "secret," but there are enough of them floating around to indicate that each item is designed to trick a student into filling in the wrong bubble.

Then the items are massaged statistically to construct an ungrounded statistical "scale." The same scale holds for all, irrespective of the instruction that has been delivered.

Value added=statistical witchcraft. It sells big.

Unknown said...

This is a very compelling topic which is worthy of examining. Separating educable students from uneducable ones is, by and large, giving up on the ones who are not easy to teach. However, the parents send the best children they have to schools and expect nothing less than excellence. If anything, they pay taxes which are used to run these public schools and teacher salaries. Please note that I’m not in any way attempting to suggest that parents don’t have a role to play in their children’s learning. That being said, I do know that there are many good teachers and school administrators who are doing their best on a daily basis to enhance student’s learning experiences (their hearts are in the right place).

Perhaps, the problem is the will power to pay teachers what they are worth. Think about it, most teachers leave teaching after five years. Consequently, if we can handsomely compensate teachers, we will not only reduce the attrition rate, but attract the best and brightest as well. If anything, teaching as a craft is like wine, it gets better with time.

Your thoughts

Roger Sweeny said...

The same scale holds for all, irrespective of the instruction that has been delivered.

Why is that bad? If I wanted to know if someone could fly a plane, I would want to know if they could fly a plane. I wouldn't care how they were taught.

Dick Schutz said...

If I wanted to know if someone could fly a plane, I would want to know if they could fly a plane. I wouldn't care how they were taught.

True. But the measures at issue are not direct observations; they're ungrounded statistical scales. If pilots-to-be were crashing, one should care about "how they were taught."

Roger Sweeny said...

Dick Schutz,

From your November 16, 11:37 AM post, I gathered that you were talking about bubble-in-the-right-answer multiple choice tests.

Those tests purport to measure something students have learned. If the tests correctly measure what students have learned, and if lots of kids are failing, I would definitely want to know how the kids were taught.

But the purpose of the tests is to measure what has been learned, not how it was taught.

Dr. Kwame M. Brown said...

I find it interesting here that no one mentioned once as a factor - whether children LIKE school or not! This is an undeniable part of this "black box" you speak of. What about the actual target of the learning? What is being to address target affinity?

Furthermore, how can not discuss teacher pay? In a capitalistic society, EVERYTHING is affected by how much money is put into it.

What say you?

Malcolm Kirkpatrick said...

(Dr. Brown): "I find it interesting here that no one mentioned once as a factor - whether children LIKE school or not! This is an undeniable part of this "black box" you speak of. What about the actual target of the learning? What is being to address target affinity?

Furthermore, how can not discuss teacher pay? In a capitalistic society, EVERYTHING is affected by how much money is put into it.

What say you?
"

To your first point: compulsory, unpaid labor is slavery, black or white, male or female, young or old. We justify compulsory attendance statutes and taxation in support of school in terms of benefit to society and long-term benefits to children, and ignore the mismatch between these adult reasons to impose school, on the one hand, and individual students' reasons to do what we require.
To your second point: the US K-12 school system is one of the Earth's last command economies. North Korea in the US. It fails for the same reasons that the people of North Korea eat tree bark.

Tracy W said...

Dr. Kwame M. Brown - whether children like school is not exogenous to how the school operates.

Furthermore, how can not discuss teacher pay? In a capitalistic society, EVERYTHING is affected by how much money is put into it.

Because your claim here is wrong. Educational quality is not affected by education spending, at least within the range of spending covered by OECD countries. See http://d-edreckoning.blogspot.com/2008/03/data-shows-you-are-wrong.html

Dr. Kwame M. Brown said...

@Tracy W:

I am not quite sure what you mean by "not exogenous"? Did you mean "not endogenous", to mean a factor contained within? If so, you would be wrong. This is an unavoidable truth of any activity involving children. If they enjoy it they do it, and do it well. Look up some of the TED talks by Ken Robinson. Furthermore, I have worked with children from infancy through 18, and at pretty much every SES level, and advise the federal government on play, so this would be something I am stating from a pretty solid background in development, not a musing.

The teacher pay thing, and your reference to the blog post - this was a look at overall school resource spending. Furthermore, as one of the commenters pointed out, higher paid teachers are usually associated with higher SES school districts, where more often than not, the kids DO perform better! There are a variety of factors in education. Teacher pay just happens to be one of them - how else in a capitalistic society do you attract talent en masse?

KDeRosa said...

Children do tend to enjoy what they do well, so the causation attributed to your correlation observation may be backward. We are concerned with the kids who aren't doing well and therefore tend not to like school.

Also, you might have missed this paragraph toward the end of the post re instructional expenditures (i.e., largely teacher pay.)

"In case you were wondering, I also ran the regression with instructional expenditures instead of total expenditures to see if perhaps these districts with lots of low-SES students were squandering their resources on non-instructional expenses. Apparently, they aren't. The correlation between instructional expenditures and low-SES student achievement was even weaker than that for total expenditures (R = 0.03, R2 = 0.0012, and P = 0.47)."

Tracy W said...

Dr. Kwame M. Brown - whether children like school is not exogenous to how the school operates.

Furthermore, how can not discuss teacher pay? In a capitalistic society, EVERYTHING is affected by how much money is put into it.

Because your claim here is wrong. Educational quality is not affected by education spending, at least within the range of spending covered by OECD countries. See http://d-edreckoning.blogspot.com/2008/03/data-shows-you-are-wrong.html

Tracy W said...

I am not quite sure what you mean by "not exogenous"?

I meant that kids enjoyment of school is endogenous to the schooling system. What the school does and how it does it can affect, to some extent, how much children enjoy school and their school work. You say that if kids enjoy their schoolwork then they can do it, and do it well. I think that some of the casuality may be the other way around. Let's imagine two extremes, (1) a school that set impossibly high standards, eg expecting ordinary 6 year olds to perform at PhD levels, or (2) a school that introduced no new work, and just had kids doing drills all day of things they already knew. At either school, can you imagine many kids enjoying their school work after a few months, no matter what attitude they started with?

To enjoy learning, you need tasks that are doable, but not too easily-doable, this is why sports are generally split-up by skill level or a proxy-thereof (eg premier-league, second-tier, elite), or handicaps are introduced to artificially level matters between players (eg golf) - most people don't find it that fun to be on either side of a walk-over. In schools, the difficulty level of the tasks is controllable by the school. So are other factors, like switching between activities (eg splitting up maths time between things like solo-work on worksheets, maths games involving all the students, learning a new skill, etc.)

Teacher pay just happens to be one of them - how else in a capitalistic society do you attract talent en masse?

Talented at what? Teaching difficult students or passing job interviews, or passing teacher qualification tests, or getting along with senior management? What talent are the people who hire teachers looking for? I think the biggest problem with our education system is that the people in charge of it, the politicans and the senior management, are way too separated from feedback on how it's doing.
I'll also note that other ways of attracting talent en masse in a capitalistic society is to offer better working conditions. Which can mean physical facilities, and which can mean a job where you feel like you're accomplishing something.

Dr. Kwame M. Brown said...

@Tracy:

Some of your logic escapes me, especially where you write "some of the causality may be the other way around" and then proceed to give the example of two extremes, neither one of which is enjoyable, for different reasons. I never defined any one approach (especially not an extreme) as what WAS enjoyable.

Yes, we agree that the balance between challenge and success is one of the roots of enjoyment - but it's also parent and teacher engagement. Does the teacher enjoy what they are teaching? Are they into it?

You ask what talents do we want to attract? yes, that is something we must figure out. The major talent I am looking for in anyone working directly with children is the ability to interact with a child on the CHILD's level. Everything else you mention is important, but secondary. Now when we talk about talent, many assume that this means "born with". This couldn't be further from the truth. Most talents are learned. So that speaks to a reform in the way teachers are educated. So, yes, I believe teacher pay is one factor, but acknowledge that it is not the only factor. My original point was, how can we have a conversation where we EXCLUDE that, not that it is the only thing.

Have you looked up Ken Robinson yet? By no means is he some sort of education messiah, but he puts his finger on some very interesting points, especially about the way we structure education. Part of what we suffer from is outdated models.

When you speak of work environment, the data you are looking at, I believe, comes from studies on work SATISFACTION, not the factors involved in initial attraction to a field. Students shy away from the education field every day because of the perception and reality of low salaries. Especially students who are from low income backgrounds themselves and saddled with educational debt.

Please be careful when considering and quoting scientific data. Research must be read very thoroughly to be properly and authoritatively applied in the real world (or a debate). This means reading the original research, not just abstracts or requotes, as well as reading and considering the original authors' conclusions. Twice hear, you have made invalid projections of data from one question onto another.

But let's keep it going, we can continue to learn from each other.

Roger Sweeny said...

I'll also note that other ways of attracting talent en masse in a capitalistic society is to offer better working conditions. Which can mean physical facilities, and which can mean a job where you feel like you're accomplishing something.

Amen.

c-u-r-m-u-d-g-e-o-n said...

Charter schools should not be grouped with private and parochial schools as able to select easy-to-educate students. In return for the student's share of the school district's funds, the charter school accepts and educates any student who lives in the district whether or not that is easy to do. The mission of the school district is not just to educate the educable but to protect the community against the harm poorly educated people do.

Since people move about much more now than when public schooling began, the fruits of this effort are greatly diluted. Nonetheless, state law requires districts to persist in it whether or not they directly manage the schools. Charter schools are privately managed public schools.

Keuter said...

This post is written by Jason Keuter.

The top group is not really learning. They are just getting SAT, PSAT, and State test scores that justify the system. Thus, we can't pretend that instruction doesn't matter for them and they'll do well no matter what. They will master mediocre and mindless curriculum, yes, but the top students are actually the ones showing the steepest declines.

To give but one example, among top students who take AP US History, they enter the class, generally, unable to write anything approximating a coherent essay. They are unable to really make an argument, much less defend one. They don't really know anything at all about the most elemental facts of US, European, or World History.

No, I don't buy it. Measured against objective curriculum standards of what a top achieving student should be able to do, what top achieving students can do is pathetic - really just enough to allow schools to say they are successful. And don't forget, the schools, state education departments and test makers all work hand in hand to determine what it is advanced students will succeed at - usually doing well on tests crafted not to assess but to advertise "success".