I am studying education policy, so I have to comment when big things happen in the education policy arena.

The big thing in the last week was the release of the 2007 NAEP (National Assessment of Educational Progress) reading and  math scores.  The media reports are enough to make any one’s head spin.

The white house is claiming credit for No Child Left Behind (NCLB ), saying “No Child Left Behind Is Helping Students Achieve Record Success“.  The New York Times noted that the reading results were mixed, but otherwise reported the party line. 

These are the types of sources that most Americans hear about.  They don’t hear from groups like fairtest.org who pointed out that scores were rising faster BEFORE NCLB and have slowed down since.

More important, however, is the argument that isn’t happening outside of certain academic circles.  NAEP is often called “the Nation’s Report Card” and is used to beat on public schools for their inability to get most students to “proficient”.  Yet when the definitions of proficiency were originally set, they were set to unreasonably high levels.  Reviews by the US Government General Accounting Office, National Academy of Education and National Academy of Science have all stated that the levels set were at best meaningless and more likely highly misleading.  To quote the GAO report “We find that the empirical validity of the levels results — whether the levels measure what they claim to measure — has not been demonstrated.”  These agencies and many other researchers have been ignored.  Even the governing board that controls the NAEP levels admits that all is not right.  In a 2001 report the governing board said: 

Nor is performance at the Proficient level synonymous with “proficiency” in the subject. That is, students who may be considered proficient in a subject, given the common usage of the term, might not satisfy the requirements for performance at the NAEP achievement level. Further, Basic achievement is more than minimal competency. Basic achievement is less than mastery but more than the lowest level of performance on NAEP. Finally, even the best students you know may not meet the requirements for Advanced performance on NAEP.

What that paragraph amounts to is an admission that, though they used words that we all think we understand (proficient, basic, advanced) those words really mean something else in the NAEP world.  Not that the media or the white house appear to have gotten that memo.  Thus reports of most students not being “proficient” ends up in the news, providing fuel for whatever new reform someone wants to try. 

So, the test is inaccurate to what most people would expect, and the white house is telling us that because scores have gone up since NCLB, NCLB must be a good thing.  Which has led to a terrifying trend toward using the NAEP levels for setting NCLB performance levels.  An excellent paper on why this is a really REALLY bad idea is here.  All of this smacks of dishonesty, but is far too disbursed to accuse any one person. 

What is clear is that each group uses the numbers in their own ways.  Bush et al want us all to think that U.S. schools are failing because students aren’t making NAEP proficiency levels, yet NCLB works because NAEP scores are up.  Lies, damn lies, and statistics indeed.