You probably think you know which states have the best and worst education systems in the country. If you regularly dip into rankings such as those published by U.S. News and World Report, you likely believe schools in the Northeast and Upper Midwest are thriving while schools in the Deep South lag. It's an understandable conclusion to draw from those ubiquitous "Best Schools!" lists. It's also wrong.
The general consensus on education, retold every few news cycles, is that fiscally conservative states are populated by cheapskates. In those necks of the woods, people are too ignorant to vote in favor of helping their illiterate and innumerate children. Intelligent people understand that high taxes and generous pensions for public school teachers are the recipe for an efficient and smoothly functioning education system. If skinflint voters would just lighten up, the story goes, they too could become erudite and sophisticated.
Paul Krugman rehashes this narrative regularly in his New York Times column, frequently bemoaning the country's purportedly miserly education budgets. Increasingly, he perceives libertarian barbarians at the gates of state governments, brandishing axes for dreaded spending cuts. In April, he wrote that "we're left with a nation in which teachers, the people we count on to prepare our children for the future, are starting to feel like members of the working poor.…One way to think about what's currently happening in a number of states is that the anti-Obama backlash, combined with the growing tribalism of American politics, delivered a number of state governments into the hands of extreme right-wing ideologues. These ideologues really believed that they could usher in a low-tax, small-government, libertarian utopia."
In Krugman's view, which reflects the education establishment's view as well, those attempting to keep the size of government in check are a danger to your child. To support this claim, education wonks and activists point to state rankings in U.S. News, Education Week, or WalletHub—outlets that grade states according to a few key measures, such as graduation rates, education spending, and test scores. When education is discussed in the news, these rankings are often cited to illustrate the havoc that restrained budget growth and right-to-work laws can wreak.
Indeed, such rankings do seem to show that the highest-quality state educational systems tend to be in big-spending states in the Northeast or Upper Midwest. These places apparently honor and respect teachers, while Southern states inexplicably abhor them. But the cheapskates in cheap states get their just deserts: Sophisticated northern jurisdictions grow ever smarter, while stingy conservative backwaters sink into ever-lower depths of ignorance. The solution is obvious: Pay up or your kids will suffer.
There's just one problem with this narrative: Traditional rankings are riddled with methodological flaws.
Flipping the Script
To better capture the real state of play, we recently conducted a detailed investigation of state education rankings. We built a new set of rankings based on students' performance on the National Assessment of Education Progress (NAEP), a battery of standardized tests sometimes called "The Nation's Report Card." These tests are given to fourth and eighth graders as well as some high school seniors.
U.S. News and the others also generally use NAEP scores as an element of their rankings, but they use them in a misleading manner. When appropriately analyzed, the evaluation has a very different tale to tell.
We fixed two serious problems common to traditional rankings. First, we removed factors that do not measure K–12 student performance or teaching effectiveness, such as spending per student (intentions to raise performance are not the same as raising performance), graduation rates (which often indicate nothing about learning, since 38 states do not have graduation proficiency exams), and pre-K enrollment.
Rankings that include these factors distract from true student performance. For example, under traditional rankings, states with inferior test scores sometimes outrank states with better ones simply because they spend more. A June article in the Tampa Bay Times highlighted the role of spending in the state's position in one lineup: "Critics of Florida's public education funding system got another piece of ammunition Wednesday, as Education Week rated the state's school spending an F alongside 25 other states."
As recently as 2011, Education Week placed Florida fifth in the nation. Then the publication altered its methodology to put more weight on raw expenditures. Despite high test scores, the state dropped to 29th place—not because teaching effectiveness fell, but because the state supposedly spent too little!
Such a dramatic drop in an influential state ranking has the power to sway public opinion. Yet taxpayers and parents should be appalled by the pervasive and perverse notion that spending, by itself, is a positive factor. (If anything, burning more money without concomitant gains in outcomes should detract from a state's reputation.) In our rankings, merely spending more on education does not increase a ranking. To receive high marks, states must actually impart learning to their students.
Our second and more important change was to disaggregate student performance data so that we could compare likes with likes. Traditional rankings effectively reward states for not having many minority students. States do well simply because they are populated by families from more socioeconomically successful ethnic categories—not because they are actually doing a good job educating their various categories of students.
Student heterogeneity is precisely why traditional state education rankings can be so misleading. Small, largely white states in New England, such as Maine and Vermont, do very well in these rankings, but that status merely reflects their small black and Hispanic populations.
This is starkly illustrated by comparing Texas and Iowa. According to U.S. News and World Report, Texas, which ranks 33rd, is far surpassed in educational quality by Iowa, which ranks eighth. When only the test scores are examined at an aggregate level, the ranks shift somewhat but their relative positions don't: Texas moves to 35th and Iowa to 17th. But when we disaggregate student performance scores by racial categories (white, black, Hispanic, and Asian), the rankings change dramatically.
By looking at test scores for students in fourth and eighth grade in math, reading, and science, and by separating students by racial category, we get 24 different possible bases of comparison. This allows us to measure how well states do for each specific student type—Asian fourth-grade math students, for instance. (We have adjusted our rankings to compensate for the fact that not all states report scores for every student group.) Giving each type equal weight, Texas comes in fifth and Iowa 31st—a remarkable reversal.
Iowa, it turns out, falls so far because it does a below-average job of educating white students (30th in the country), black students (36th), and Asian students (40th), although it is slightly above average with Hispanic students (20th). Because Iowa has a disproportionately large share of white students, who as a group score higher than blacks and Hispanics, rankings that use aggregated test scores place Iowa's education system as above average and superior to that of Texas. Yet Texas students score higher than Iowa students in all but one of the 20 possible bases of comparison between these two states.
Think about that: White students do better in Texas than in Iowa. Black students do better in Texas. Hispanic students do better in Texas. Asian students do better in Texas. Given these facts, it is absurd for U.S. News to rank Iowa higher than Texas in terms of educational performance. And this example is no fluke. Many other state comparisons similarly reverse if you account for student heterogeneity.
Taken together, these methodological problems should disqualify mainstream rankings from use in our national discourse.
Keys to Success
It's perhaps understandable that news outlets have treated these mainstream ranking numbers as credible—they've been virtually the only game in town. Hopefully, that ends now.
Our study corrects the errors discussed above: We excluded metrics not directly related to learning and looked solely at NAEP scores; we disaggregated students by age, subject, and racial category to reflect a state's student heterogeneity; we gave equal weight to each category to produce a new average quality score for each state; and then we ranked the states by this new metric to produce the "quality" rankings in the table.
When this more appropriate method is used, the results are vastly different from the dominant narrative. Only two of the U.S. News top 10 states, Massachusetts and New Jersey, show up in our top 10 based on the quality of state education.
There are other major changes in the rankings as well, particularly in New England. Maine drops from sixth to 48th; Rhode Island from ninth to 39th; Vermont from fourth to 27th. Going in the other direction, Texas, Georgia, and Florida jump from 33rd, 35th, and 40th to fifth, seventh, and third, respectively. The Northern monopoly on top rankings disappears.
We also address another question that traditional rankings altogether ignore: How much are states spending to achieve their levels of success? New Jersey, which ranks fourth under our methodology, spends more than twice as much per student as third-ranked Florida. New York, the highest spender, ranks 30th in our lineup, only slightly below 29th-ranked Tennessee—but the Empire State spends about two and a half times as much as the Volunteer State. Surely, Florida achieves its student outcomes much more efficiently than New Jersey and Tennessee much more efficiently than New York.
Since many of the states with high expenditures are also more expensive places to live in general, we adjusted the annual per-student spending values by the cost of living to make comparisons more appropriate. We then produced an "efficiency" ranking by dividing each state's quality measure by (adjusted) per-student expenditures. All of the top five states according to this metric—Florida, Texas, Virginia, Arizona, and Georgia—are right-to-work states in the South and Southwest. They're getting the most bang for their education buck. The only state from the U.S. News top 10 that makes it into the top 10 for efficiency is Massachusetts, in 10th place.
The scatterplot illustrates the two components that go into these efficiency rankings: A state's quality score is on the vertical axis and its per-student expenditures (controlling for cost of living) is on the horizontal axis. This allows readers to see how their states compare in terms of both educational performance and educational expenditures. The upper-left corner is most desirable: good teaching at low cost. The lower-right is the worst of both worlds: poor teaching and high costs. Notice that the chart reveals no clear relationship between spending more on education and achieving better outcomes.
In order to examine the relationship between expenditures and quality more precisely, we ran multiple regression analyses on our data, which included several other variables. The regression results support the view that expenditures are not linked to student performance. It turns out that throwing more money at something isn't guaranteed to yield improvement—as Kansas City demonstrated when, under court control from 1985 to 1997, it became the highest-spending school district in the country. It also failed to increase its performance.
Our regression results revealed other findings as well. The most interesting is that union strength has a powerful negative effect on student performance. It's well-known that teachers unions aim to increase wages, which might lead to better teachers and increased test scores. But apparently, other union goals that are harmful to student performance—such as protecting poor teachers from being fired or blocking merit-based pay—have a greater impact. This may come as a shock to those who think teachers unions are a recipe for educational success.
Waste Not
Overall, our results demonstrate that existing state education rankings aren't to be trusted. When those scores are corrected, the conventional narrative is turned on its head. Students in fiscally conservative right-to-work states often perform better than their counterparts in high-tax, high-spending progressive utopias. And they don't have to break the bank to achieve their success.
Journalists and public officials should stop referencing the flawed traditional rankings. Just last April, New Jersey Gov. Phil Murphy cited Education Week to claim the Garden State is superior in education to Texas. Murphy was responding to an April op-ed by Texas Gov. Greg Abbott inviting Jersey residents fed up with high taxes to move to Texas. Instead, Murphy should consider taking a page from the Lone Star State's playbook. Importing the policies that make Texas second on the list for "efficiency" could help maintain New Jersey's high performance while discarding its current punishing taxes
Unfortunately, mainstream rankings confirm the biases of many media outlets and the self-serving interests of education functionaries who only gain from higher spending—while also giving short shrift to minority students in predominantly white states. As a result, we suspect that the usual narrative based on those flawed state rankings will continue to predominate.
That's a shame. The time is ripe to re-evaluate education policy in this country. After all, minds and dollars are terrible things to waste.
This article summarizes results from a Cato Policy Analysis, which readers can view for more details.
Comments