October 26, 2006

Grading the Tests

There's been a lot of back and forth in the comments about the difficulty level of the tests that states have created in response to NCLB. Educators seems to uniformly agree that they are unable to get most kids to the proficiency level by 2014. Parents don't seem to have any idea what's even on the tests. So why don't we take a look at the testing regime of a state to see what's going on.

I'm going to pick Pennsylvania because a) that's the state I'm most familiar with and b) they make a lot of information available on their website, a poorly laid out website for sure, but you can usually find what you're looking for if you poke around long enough.

The Tests

Pennsylvania does not release actual test questions from its PSSA test; however, they offer the next best thing -- lots of sample questions from each test. Good enough for our purposes. Here are representative problems (pdf) for the upcoming 2006-2007 tests for 5th and 8th grades and from the 2005-2006 exams for 11th grade. (all in pdf form.). The tests are given at the end of March.


















Math
Reading
5th Grade
5th Grade
8th Grade
8th Grade
11th Grade 11th Grade

Before you look at the tests, let me tell you what it means to be proficient. To be "proficient" on the tests here's the percentage of questions that the students would have to answer correctly.























Math Reading
5th Grade
65% 67%
8th Grade
62% 75%
11th Grade
62% 73%

These proficiency percentages were calculated from the cut scaled scores given on p. 117 and the raw score to scaled score conversion tables given in appendix II-9 of this monster 2005 technical report (pdf).

So in math, a student could get 1 out of every three problems wrong and still be proficient. In reading, the student could get one out of every four problems wrong and still be proficient.

Here's a few more notes before you jump into the sample questions. In the math portion, students are permitted to use calculators on all but a tiny fraction of calculation questions on the fifth and eighth grade exams. In addition, a formula sheet and ruler are provided, as indicated in the scoring sampler.

Now go ahead and take a look at the sample problems.

In math, you'll see that many of the problems at each grade level are simplistic. Many of the problems can be solved easily by plugging the multiple choice answers into the calculator. Few, if any problems, require multiple steps to answer. Few of the problems contained in each test are at grade level; many questions test skills that should have been taught years before. Manipulation of fractions, a critical skill needed for success in algebra, is barely tested at all.

In reading, the passages are about at grade level (using the lexile calculator); however, the comprehension questions are simplistic, frequently requiring that the student find the requested fact in the text.

Here are the disaggreagated scores from the 2005-2006 PSSA exams. The table shows the percentage of students who scored at or above the proficient level. Math scores are given first followed by reading scores (math/reading).









































All Asian
White
Hispanic Black Low SES
IEP
5th 67/61 83/73 74/68 47/36 40/34 49/40 33/24
8th
62/71 82/81 70/78 39/45 33/44 42/50 20/27
11th
52/65 72/70 58/71 25/35 22/35 29/41 11/19
The scores break down almost exactly along IQ distribution lines. There is an achievement gap of roughly a standard deviation between whites and blacks/hispanics in both math and reading. And, there is an achievement gap of 0.4 between asians and whites. This indicates that about 4/10 of asians in PA are from northeast asia and 6/10 are from southest asia which seems about right to me based on my observations from the Philadelphia area.

Here's another basis of comparison.

Here's the placement test given for the middle of fouth grade for Singapore Math. Compare the questions to the 5th grade math PSSA questions.

Here's the placement test given for the middle of sixth grade for Singapore Math. Compare the questions to the 8th grade math PSSA questions.

Here's the placement test given for the second year of algebra (8th grade) for Singapore Math. Compare the questions to the 11th grade math PSSA questions.

Singapore is tops in the world for math achievement. One reason for that achievement is the rigorous Singapore Math curriculum the country uses. Another reason is that Singapore is full of brainy northeast asians. That's why I selected the placement tests that were one to three years behind the PSSA tests, giving less brainy U.S. students an extra one to three years to catch-up.

Here's the post test for Connecting Math Concepts (CMC) level F, a fifth grade math program. The curriculum has been extensively field tested so that almost every student, including the ones with IEP, who has completed the six years in the probram will be able to answer at least 80% of the post test questions correctly. Compare the questions on the post test to the PSSA questions. Now look at the high percentages of black,hispanic, low SES, and IEP students who are not proficient in Pennsylvania (the 11th grade IEP students are performing at the chance level, i.e., randomly filling in the bubbkles will yield the same score). Now ask yourself why less than 2% of schools use CMC or a math program like it.

The comments section is open.

17 comments:

Anonymous said...

This isn't totally relevant to the thread, but I want to complain about at least one question:

"Kareem put a fence around the perimeter of a baseball field. Which units are reasonable for measuring the perimeter?"

Kilometers is offered, but is wrong.

Huh? Maybe not as good as yards, but what is unreasonable about using Km for perimeter? Because you'll need a decimal point?

-Mark Roulo

rightwingprof said...

"In the math portion, students are permitted to use calculators on all but a tiny fraction of calculation questions on the fifth and eighth grade exams."

Since I've administered more exams than I'd like to think about, I don't understand how you can allow a calculator (or anything else) on part of the exam but not the rest. How do you enforce that?

And why would anybody need a ruler on the math section? Because they can't remember how many inches there are in a foot?

I haven't looked at the questions yet. I hate pdf files. Tomorrow, probabaly.

Anonymous said...

Most of the educators (teachers, administrators) I know don't complain that they won't have most students proficient by 2014; they say they won't have all students proficient by 2014, which is the actual target.

Michael

KDeRosa said...

That's because most educators don't know how to read a statute.

Even as the law is currently written, "all" students will not have to be proficient on the state sssessment under NCLB. Currently 1% of students are exempted from taking the state assessment and may take an alternate assessment commensurate with their needs (not to be confused with alternate accomodations). The 1% number is derived from the number of students with legitimate cognitive impairments.

Go through the 2005 report I linked to in the post and you'll see that in PA the number of kids who are autistic and/or mentally retarded and/or emotionally disturbed only amount to at beast 2%. Nearly 2/3 of IEP students are learning disabled which really means teaching disabled. The rest have non-cognitive handicaps, like speech problems, orthiopedic problems, deafness, and blindness.

SteveH said...

"...they say they won't have all students proficient by 2014, which is the actual target."

Look at the questions. And the problem is????

Would you complain that schools have to get ALL kids to learn to tie their shoes by 2014 in fifth grade? How about learning to add and subtract to 20 by fifth grade? What's your cut-off?

The real question is why are these kids in fifth grade if they can't do these problems?

SteveH said...

"The rest have non-cognitive handicaps, like speech problems, orthiopedic problems, deafness, and blindness. "

For many cases, an IEP status is the only way kids can get the state money to address their non-cognitive problems. In our state, this is a large percentage.

SteveH said...

Great post Ken.

I've always thought that parents really need to see the details. When I went to an open house about our state's testing years ago, I was the only one looking at the actual tests. Everyone else was listenting to how our schools are "High Performing" and talking about relative performance, not absolute, or world-norm referenced performance.


As for the tests, the proficiency levels are lower for multiple choice tests. You can get 25 percent correct by just guessing. Many questions have obviously wrong answers, so you might up that to 33 percent correct by using just an ounce of brain power.

rightwingprof said...

"The real question is why are these kids in fifth grade if they can't do these problems?"

Bingo. And that is exactly why we can't take GPAs seriously.

MassParent said...

Back when I was in 7th grade, we had a couple of 9th graders who'd been held back in our classes.

Total chaos in those classrooms. They shouldn't have been there. The only things I learned from them was that it was cool to smoke and do drugs, and that cool kids sometimes brought guns to school.

Holding kids back is one of the few reliable ways to make AYP gains, so we will undoubtedly see a lot of it in the next few years. It is unlikely that will benefit most of the students who are either held back or who have to attend classes with other students who have been held back.

SteveH said...

"Total chaos in those classrooms. They shouldn't have been there."

That's the school's fault, not the idea of holding kids back. The school doesn't have to put those kids in the same classroom as the rest. You can also make the same argument about passing these kids along to the next grade. They are not magically going to be model students. Holding them back is not going to cause smoking or drug use.

However, you can't use this broad brush explanation for all kids and all grades. The problems exhibited by this student probably started years ago. They could even be related to the student being socially promoted to a grade where they couldn't handle the work.

MassParent said...

You ever been in a rural school?

Get five towns together, and you've got just enough to be able to offer one classroom for basic subject matter. If you have to pull five kids out and set up a separate classroom, you'll need to add mandatory study-hall time. Presumably with a separate room for those five, when they have mandatory study hall.

SteveH said...

"You ever been in a rural school?"

What on earth are you arguing here?

... that since rural schools don't have the resources to deal with diruptive students (whether they are held back or socially promoted) then what? What's your point? State testing is bad?

rightwingprof said...

"Huh? Maybe not as good as yards, but what is unreasonable about using Km for perimeter? Because you'll need a decimal point?"

One is usually asked to find the BEST answer, not necessarily the ONLY CORRECT answer, on these tests. And yards is a better answer than kilometers, precisely because of the size of a baseball field.

rightwingprof said...

"It is unlikely that will benefit most of the students who are either held back"

Irrelevant. Either the grade means something or it does not. If you socially promote students it is meaningless, and we must have tests to determine proficiency, since the grade is meaningless.

Catherine Johnson said...

I mentioned taking the 2006 Regents Math A exam in one of the Comments at Kitchen Table Math. (I Passed with Distinction, thank heavens.)

Regents A is, I believe, the only exit exam high school students have to pass in math in order to earn a Regents diploma.

After going through the scoring rubric I found that a student could pass through various configurations of correct or partially correct answers lying somewhere between these two:

* getting 17 questions out of 39 correct (this option includes getting all of the Part 1 two-point multiple choice questions correct)

OR

* 14 out of 39 correct (under this option the student gets full credit for both 4-point questions in Part IV, full credit for both 3-point questions in Part III, and full credit on 10 2-point (test has 30 multiple choice 2-point questions in Part I; 5 full response 2-point questions in Part II (Students can earn partial credit - 1 point - on problems in Part II)

PASSING SCALE SCORE 65%BR%
PASSING RAW SCORE 34 out of 84 possible points (40%)

DISTINCTION SCALE SCORE 85%BR%
DISTINCTION RAW SCORE 63 OUT OF 84 possible points (75%)

+++++

In short, a student could pass Regents Math A with a percentage correct as low as 36%.

And, of course, quite a few of these answers can be solved by the process of elimination.

Catherine Johnson said...

We're going to push our district to pilot some sections of a class using the Keller method.

Then when they say NO we'll carry on pushing them to pilot some sections of a class using the Keller method.

And in the meantime we'll get more-more parents asking why our district does not teach to mastery or even express mimimal curiosity as to whether any particular child has mastered anything.

Catherine Johnson said...

URLs for the Regents Math A (Aug 2006) test I took:

Regents Math A August 2006 (pdf file)

scoring key (pdf file)

conversion charts


Archived Regents exams 1957 - present