'The Phony NAEP Crisis' by Thomas Ultican
The recent data release by the National Assessment of Education Performance (NAEP) for mathematics has inspired balderdash. Jeb Bush called it “Alarming.” A Chalkbeat headline characterized it as a “massive drop.” Harvard’s Tom Kane wrote that it signaled “ enormous learning losses.” The Heritage Foundation’s Lindsey Burke was able to place her article in many outlets with the subtle headline, “New NAEP Test Scores Are a Disaster. Blame Teachers Unions.”
In reality, the score drops were not massive and learning loss which probably isn’t actually a thing was not enormous. However, if the purveyors of doom can convince enough people it is a crisis, then they can advance their own pet agendas such as ending public education.
NAEP (pronounced nape) testing which is known as the nation’s report card was originally implemented in 1969. The tests use a combination of standardized testing and sampling. The Washington Post reports that this year 224,000 fourth-graders from 5,700 schools and 222,000 eighth-graders from 5,100 schools were sampled. Sampling certainly makes more sense than states paying testing companies to test every student but standardized testing is still not a capable tool for measuring learning.
It is not just me saying it. Unlike the scientifically well behaved data associated with genetics study, standardized testing data is extremely noisy. The famed Australian researcher Noel Wilson wrote a seminal work in 1998 called “Educational Standards and the Problem of Error.” His peer reviewed paper which has never been credibly refuted says error in standardized testing is so large that meaningful inferences are impossible. Unfortunately, the paper has been ignored.
Wilson’s paper was followed a year later by a paper from UCLA’s Education Professor James Popham which stated, “Although educators need to produce valid evidence regarding their effectiveness, standardized achievement tests are the wrong tools for the task.”
It does seem that with all of the tests taken, data gathered and arithmetic performed, the tests must be telling us something but what? We know that the one thing this kind of test correlates to is the student’s family wealth. Education researcher Linda Darling-Hammond puts that correlation at an r-value of 0.9. An r-value of 1 on the 0-1 scale says it is a dead certainty like men not becoming pregnant. No other variable studied has a strong influence meaning they mainly input noise into the data.
So what caused the downward turn in 2022 NAEP math data for 4th and 8th graders? Is it really related to learning and should it be a large concern?
Let’s Go to the Scoreboard
Everyone has the right to access the NAEP Data Explorer and create their own data reports and charts. The tested years, the jurisdictions, data types, the subject, etcetera may be manipulated to shape a report. It is possible to compare states, public schools and private schools, districts, etc. There are limitations such as charter school data being lumped with public school data.
In the following charts, I chose mathematics either 4th or 8th grade in tested years 2003, 2019, 2022. I selected the average scale scores which are based on a 500 point scale.
Data Explorer Graphed Fourth Grade State Comparisons
One of the first observations to make is that the 500 point scale scores are plotted on a 60-point graph scale which visually magnifies any differences by more than 8 times. The national average scores go from 235 in 2003 to 241 in 2019 and then 236 in 2022. If we use the lowest data point for a denominator that five point drop from 2019 represents a 2% drop, but if we use the 500 point scale as the denominator which we should that purported enormous drop is just 1%.
Of the five large states queried, only New York had a larger than 5 point drop. Its 10-point drop calculates to a 2% decline.
The Walton Family financed publication The 74 is known to support libertarian positions on education policy. Some people claim they are biased against public schools. The 74 recently claimed in a headline, “Strong Link in Big City Districts’ 4th-Grade Math Scores to School Closures.” Under the previous president, the political right railed against health care policies like masking, vaccination and closing schools. By September 2020 there were loud sometimes violent open-schools-now protests at school boards meetings in many states and jurisdictions. The 74 article looks like an attempt to say “see we were right” but the data does not support their specious claim.
For evidence, they turned to the Koch addled economist Emily Oster. She is the Brown University professor that argued in the summer of 2020 that children should be back in school. At the same time, she cast doubt on masking. With the new NAEP results, she again supports the libertarian cause stating, “The districts with more remote learning have larger test score losses.” This appears to be something she just said with no evidence.
If we look at the states graphed above, the only outlier is New York with its 10-point drop, but California, Texas, Florida and Massachusetts all had 5-point drops.
District Comparisons of Fourth Grade Mathematics Scale Scores
Oster’s claim was about big city districts. If we look at these big city data sets there does not appear to be real differences. All of the big city districts had an 8- to 9-point drops in their fourth grade test between 2019 and 2022. Whether they opened early or stayed closed longer.
Education reporters note that test score drops in eighth grade were worse than those in fourth grade. On the 500-point scale the average drop in fourth grade was five points while in eighth grade it was eight points or 1% and 1.6% respectively.
Eighth Grade Mathematics by District
It is true that the national math data for eighth graders showed an average 8-point drop in 2022. However, the declines were not uniform between districts. The country’s second largest school district in Los Angeles actually returned a positive result and the 4-point decline in the nation’s largest school district was relatively modest.
There is no way the eighth grade testing result for the nation’s two largest school districts could fairly be characterized as a crisis. It is also noteworthy that these two districts were closed longer than most others in the nation.
The Roots of the Down Turn
America’s students like everyone else suffered through a two year pandemic-inspired nightmare. Did anyone really expect that on average they would perform at par?
One of the difficult pandemic related student manifestations was increased violence. As schools were reopening, Homeland Security notified them, “The reduced access to services coupled with the exposure to additional risk factors suggests schools — and the communities in which they are located — will need to increase support services to help students adjust to in-person learning as they cope with the potential trauma associated with the pandemic response.” Schools around the country saw a dramatic increase in fighting and insolent behavior.
This past July, the Washington Post reported, “The data, collected as the 2021-2022 school year was winding down, also showed that more than 70 percent of schools saw increases in chronic student absenteeism since the onset of the pandemic and about half of the schools reported increased acts of disrespect toward teachers and staff.”
Many school districts started experiencing crippling staff shortages and the NAEP testing came at a particularly inconvenient time. During the January to March, 2022 testing window, the nation experienced the omicron variant infection explosion. CDC data shows that during the testing window infection rates grew to more than 200 people out of every 100,000 in population becoming infected daily.
Disaster Capitalism Needs a Crisis
Amway Billionaire and dominion supporter Betsy DeVos said the NAEP data showed that children should no longer be “hostages” in a “one-size-fits-none system that isn’t meeting their needs.” She has been spending for decades to get rid of the secular public schools she sees as an evil.
Like every education crisis since 1983’s “A Nation at Risk” this is another manufactured crisis. The crisis rhetoric used to justify incessant accountability layered onto a constant process of new standards and new tests is, as Berliner and Biddle documented, a manufactured lie.
In writing about the pandemic effects on schools, John Merrow reported, “Secretary of Education Miguel Cardona, who called the results ‘appalling and unacceptable,’ told a group of reporters that the results are ‘a moment of truth for education,’ adding ‘How we respond to this will determine not only our recovery, but our nation’s standing in the world.”’
And even more over the top than Cardona, Harvard’s Tom Kane wrote in the Atlantic,
“[S]tudents at low-poverty schools that stayed remote had lost the equivalent of 13 weeks of in-person instruction. At high-poverty schools that stayed remote, students lost the equivalent of 22 weeks. Racial gaps widened too: In the districts that stayed remote for most of last year, the outcome was as if Black and Hispanic students had lost four to five more weeks of instruction than white students had.”
When people start using the sham CREDO days of learning metric, I am pretty sure they are dissembling. This is the kind of stuff that caused Professor Paul Thomas to declare, “But mostly, I hate the lies, political, media, and commercial interests that are eager to shout “crisis!” because in the spirit of the good ol’ U.S. of A., there is money to made in all that bullshit.”
Cardona’s Department of Education is known to embrace at least three methods for helping struggling students raise their test scores: 1) extend school day and year, 2) mandatory summer school and 3) ‘high-dosage tutoring,’ where one trained tutor works with no more than four students, three times a week for an entire year. In other words our Education Secretary who is a former New Leaders Fellow embraces a method that may raise test scores but promises to undermine engagement and the joy of learning. It is a corporate solution, not an educator’s solution.
The NAEP test scores are not a crisis but bad education leadership, suspect scholarship and billionaire meddling are. It is time to get out of the road of educators and let them do their job. No high-dosage tutoring, no extended days and no forced summer schools.
The children are not broken. If they missed some lessons over the past two years, unfettered educators will quickly resolve the issues. Students who have not been convinced that education and learning are onerous and hateful will be fine. Cardona, Kane and DeVos are the crisis.