Grading the Tests

There's been a lot of back and forth in the comments about the difficulty level of the tests that states have created in response to NCLB. Educators seems to uniformly agree that they are unable to get most kids to the proficiency level by 2014. Parents don't seem to have any idea what's even on the tests. So why don't we take a look at the testing regime of a state to see what's going on.

I'm going to pick Pennsylvania because a) that's the state I'm most familiar with and b) they make a lot of information available on their website, a poorly laid out website for sure, but you can usually find what you're looking for if you poke around long enough.

The Tests

Pennsylvania does not release actual test questions from its PSSA test; however, they offer the next best thing -- lots of sample questions from each test. Good enough for our purposes. Here are representative problems (pdf) for the upcoming 2006-2007 tests for 5th and 8th grades and from the 2005-2006 exams for 11th grade. (all in pdf form.). The tests are given at the end of March.

Math	Reading
5th Grade	5th Grade
8th Grade	8th Grade
11th Grade	11th Grade

Before you look at the tests, let me tell you what it means to be proficient. To be "proficient" on the tests here's the percentage of questions that the students would have to answer correctly.

	Math	Reading
5th Grade	65%	67%
8th Grade	62%	75%
11th Grade	62%	73%

These proficiency percentages were calculated from the cut scaled scores given on p. 117 and the raw score to scaled score conversion tables given in appendix II-9 of this monster 2005 technical report (pdf).

So in math, a student could get 1 out of every three problems wrong and still be proficient. In reading, the student could get one out of every four problems wrong and still be proficient.

Here's a few more notes before you jump into the sample questions. In the math portion, students are permitted to use calculators on all but a tiny fraction of calculation questions on the fifth and eighth grade exams. In addition, a formula sheet and ruler are provided, as indicated in the scoring sampler.

Now go ahead and take a look at the sample problems.

In math, you'll see that many of the problems at each grade level are simplistic. Many of the problems can be solved easily by plugging the multiple choice answers into the calculator. Few, if any problems, require multiple steps to answer. Few of the problems contained in each test are at grade level; many questions test skills that should have been taught years before. Manipulation of fractions, a critical skill needed for success in algebra, is barely tested at all.

In reading, the passages are about at grade level (using the lexile calculator); however, the comprehension questions are simplistic, frequently requiring that the student find the requested fact in the text.

Here are the disaggreagated scores from the 2005-2006 PSSA exams. The table shows the percentage of students who scored at or above the proficient level. Math scores are given first followed by reading scores (math/reading).

	All	Asian	White	Hispanic	Black	Low SES	IEP
5th	67/61	83/73	74/68	47/36	40/34	49/40	33/24
8th	62/71	82/81	70/78	39/45	33/44	42/50	20/27
11th	52/65	72/70	58/71	25/35	22/35	29/41	11/19

The scores break down almost exactly along IQ distribution lines. There is an achievement gap of roughly a standard deviation between whites and blacks/hispanics in both math and reading. And, there is an achievement gap of 0.4 between asians and whites. This indicates that about 4/10 of asians in PA are from northeast asia and 6/10 are from southest asia which seems about right to me based on my observations from the Philadelphia area.

Here's another basis of comparison.

Here's the placement test given for the middle of fouth grade for Singapore Math. Compare the questions to the 5th grade math PSSA questions.

Here's the placement test given for the middle of sixth grade for Singapore Math. Compare the questions to the 8th grade math PSSA questions.

Here's the placement test given for the second year of algebra (8th grade) for Singapore Math. Compare the questions to the 11th grade math PSSA questions.

Singapore is tops in the world for math achievement. One reason for that achievement is the rigorous Singapore Math curriculum the country uses. Another reason is that Singapore is full of brainy northeast asians. That's why I selected the placement tests that were one to three years behind the PSSA tests, giving less brainy U.S. students an extra one to three years to catch-up.

Here's the post test for Connecting Math Concepts (CMC) level F, a fifth grade math program. The curriculum has been extensively field tested so that almost every student, including the ones with IEP, who has completed the six years in the probram will be able to answer at least 80% of the post test questions correctly. Compare the questions on the post test to the PSSA questions. Now look at the high percentages of black,hispanic, low SES, and IEP students who are not proficient in Pennsylvania (the 11th grade IEP students are performing at the chance level, i.e., randomly filling in the bubbkles will yield the same score). Now ask yourself why less than 2% of schools use CMC or a math program like it.

The comments section is open.