September 7, 2006

This Scares Me

The story so far...

Jay Matthews argues in favor of national standards using the wide disparity between students labeled as proficient in the NAEP and in various state tests:

Maryland recently reported that 82 percent of fourth-graders scored proficient or better in reading on the state's test. The latest data from the National Assessment of Educational Progress, known as "the nation's report card," show 32 percent of Maryland fourth-graders at or above proficiency in reading.

Virginia announced last week that 86 percent of fourth-graders reached that level on its reading test, but the NAEP data show 37 percent at or above proficiency.

Sherman Dorn helpfully reminds Matthews that cut-score setting is not the same thing as standards setting. And further adds:
[A]ny attempt to use a test to "set standards" is getting things backwards. Don't we first decide what we want students to do?
So if there were a national test every child takes, I predict that there would be a yawning gap between the test and any sense of real standards or expectations.
This is, of course, a sucker's bet.

The current NAEP already reflects a yawning gap between the test and any sense of real standards or expectations.

Tom Loveless of the Brookings Institute recently took a look at the math portion of the NAEP. He coded each released arithmetic item according to Singapore's (who leads the world in math achievement) scope and sequence for its math program and determined how well kids answered the problems in the 8th grade NAEP. Here's the results:

Grade Level% of Questions% Answered Correctly

So, for example, in the 8th grade NAEP 16.3% of the test questions were at a 1st grade level and yet only 54% of the 8th graders taking the exam answered the question correctly.

This is what Loveless concludes based on his analysis of both the 4th and 8th grade NAEP tests:

A couple of things stand out in the fourth grade portion of Table 1-3. First, the problem solving items on NAEP are not very challenging—at least not in the arithmetic required to answer them. Content taught in first and second grades is at least two years below grade level for fourth graders. But that is the level of difficulty of more than four out of ten (43.6%) problem solving items on NAEP. The second surprising finding is that even though the NAEP items are so easy,
fourth graders do not do very well on them. The first and second grade items demand nothing more than being able to add and subtract whole numbers and knowing basic multiplication facts. Yet a majority of the nationÂ’s fourth graders miss the average item pitched at this level.

Even more dramatic findings are evident at eighth grade. The eighth grade items are only slightly more difficult than those on the fourth grade test (3.4 mean grade level). Almost four out of ten items (39.6%) address arithmetic skills taught in first and second grade—six years below the grade level of eighth graders taking the test. Indeed, more than three-fourths of the items (33/43) are at least four years below grade level, taught in the fourth grade or lower. Yet the percentage of eighth graders answering problem solving items correctly is an unimpressive 41.4%. Problem solving items on the eighth grade NAEP only require knowledge of very simple arithmetic. Despite this, eighth graders have trouble getting them right.
Loveless then analyzes the "algebra" strand of problems and finds them also lacking.
Despite the simpler arithmetic on algebra items, fewer students answer the algebra questions correctly than the number sense questions, suggesting that some of what has been discovered here may be because of test design. It is clear that the algebra items are assessing something other than arithmetic. One assumes that the something else is algebraic—indeed it seems to be quite challenging to most eighth graders. Nevertheless, really knowing algebra means being able to solve equations that contain more sophisticated forms of numbers than whole numbers. Anything less challenging is appropriating the term “algebra” to convey a false sense of rigor to a pool of test items.
It would appear that Matthews is barking up the wrong tree if he thinks that the NAEP is the gold standard or that national testing is the answer to anything.

What scares me though is not where the Board of Governors has set the cut scores for NAEP, but where the states are setting their cut scores (some 50 points higher). At least the NAEP cut scores accurately show that most students can't still can't add their way out of a paper bag.

It also makes me wonder why teachers and public education apologists raise such a big fuss over standardized testing. Just look how low the standards really are. Imagine if we raised the standards to accurately reflect what students really need to know to succeed academically.


Anonymous said...

I'm going to disagree. I don't think national standards or testing is the issue; I think having the government do it is the issue. After all, for years we've had international testing bodies writing, administering, analysing and reporting test results, and doing so remarkably efficiently, particularly compared to the government. I think national exit exams are a great idea, provided that some institution like ETS or College Board is completely in charge of it.

KDeRosa said...

I agree. National standards merely define what we want kids to learn. But to do national standards right you have to know how much kids can learn and how it is to be presented to effect such learning. For the most part schools don't know how to even do this with the lower half of the population. Testing, assuming the tests are properly aligned with the standards, will merely indicate whether the standards have been achieved or not. There is still the confounding variable of what and how the material is being taught. And, as you point out, with the government running the show I don't expect themto do it efficiently or correctly.

Anonymous said...

On my wish list is a reliable standardized test - normed to international standards - posted online for parents to use on their own recognizance.

Our school district now does no standardized testing whatsoever apart from the New York state tests, which are brand new, not normed to anyone but NY state kids, and meaningless to parents. (We have no idea what the scores mean.)

I'm spending a huge amount of time getting myself certified as an officiall ITBS test-giver just so I can test my child.

Somewhere along the way homeschoolers started giving their kids standardized tests "defensively"; they did it so they could brandish their kids' scores and get people off their backs.

Parents of kids in public schools could do the same.