This Scares Me

The story so far...

Jay Matthews argues in favor of national standards using the wide disparity between students labeled as proficient in the NAEP and in various state tests:

Maryland recently reported that 82 percent of fourth-graders scored proficient or better in reading on the state's test. The latest data from the National Assessment of Educational Progress, known as "the nation's report card," show 32 percent of Maryland fourth-graders at or above proficiency in reading.

Virginia announced last week that 86 percent of fourth-graders reached that level on its reading test, but the NAEP data show 37 percent at or above proficiency.

Sherman Dorn helpfully reminds Matthews that cut-score setting is not the same thing as standards setting. And further adds:

[A]ny attempt to use a test to "set standards" is getting things backwards. Don't we first decide what we want students to do?

and

So if there were a national test every child takes, I predict that there would be a yawning gap between the test and any sense of real standards or expectations.

This is, of course, a sucker's bet.

The current NAEP already reflects a yawning gap between the test and any sense of real standards or expectations.

Tom Loveless of the Brookings Institute recently took a look at the math portion of the NAEP. He coded each released arithmetic item according to Singapore's (who leads the world in math achievement) scope and sequence for its math program and determined how well kids answered the problems in the 8th grade NAEP. Here's the results:

Grade Level	% of Questions	% Answered Correctly
1st	16.3%	54.0%
2nd	23.3%	45.4%
3rd	18.6%	41.4%
4th	18.6%	32.6%
5th	13.9%	38.5%
6th	2.3%	NA
7th	7.0%	27.7%

So, for example, in the 8th grade NAEP 16.3% of the test questions were at a 1st grade level and yet only 54% of the 8th graders taking the exam answered the question correctly.

This is what Loveless concludes based on his analysis of both the 4th and 8th grade NAEP tests:

A couple of things stand out in the fourth grade portion of Table 1-3. First, the problem solving items on NAEP are not very challenging—at least not in the arithmetic required to answer them. Content taught in first and second grades is at least two years below grade level for fourth graders. But that is the level of difficulty of more than four out of ten (43.6%) problem solving items on NAEP. The second surprising finding is that even though the NAEP items are so easy,
fourth graders do not do very well on them. The first and second grade items demand nothing more than being able to add and subtract whole numbers and knowing basic multiplication facts. Yet a majority of the nationÂ’s fourth graders miss the average item pitched at this level.

Even more dramatic findings are evident at eighth grade. The eighth grade items are only slightly more difficult than those on the fourth grade test (3.4 mean grade level). Almost four out of ten items (39.6%) address arithmetic skills taught in first and second grade—six years below the grade level of eighth graders taking the test. Indeed, more than three-fourths of the items (33/43) are at least four years below grade level, taught in the fourth grade or lower. Yet the percentage of eighth graders answering problem solving items correctly is an unimpressive 41.4%. Problem solving items on the eighth grade NAEP only require knowledge of very simple arithmetic. Despite this, eighth graders have trouble getting them right.

Loveless then analyzes the "algebra" strand of problems and finds them also lacking.

Despite the simpler arithmetic on algebra items, fewer students answer the algebra questions correctly than the number sense questions, suggesting that some of what has been discovered here may be because of test design. It is clear that the algebra items are assessing something other than arithmetic. One assumes that the something else is algebraic—indeed it seems to be quite challenging to most eighth graders. Nevertheless, really knowing algebra means being able to solve equations that contain more sophisticated forms of numbers than whole numbers. Anything less challenging is appropriating the term “algebra” to convey a false sense of rigor to a pool of test items.

It would appear that Matthews is barking up the wrong tree if he thinks that the NAEP is the gold standard or that national testing is the answer to anything.

What scares me though is not where the Board of Governors has set the cut scores for NAEP, but where the states are setting their cut scores (some 50 points higher). At least the NAEP cut scores accurately show that most students can't still can't add their way out of a paper bag.

It also makes me wonder why teachers and public education apologists raise such a big fuss over standardized testing. Just look how low the standards really are. Imagine if we raised the standards to accurately reflect what students really need to know to succeed academically.