Monday, January 3, 2011

The relationship between education spending and test scores

Good public policy is impossible without understanding causality. The most common logical mistakes in the policy debate is the tendency of confusing correlation with causality. One common reason correlation is confused with causality is the failure to make demographic adjustments.

A demographic adjustment without some form of exogenous variation can rarely tell us what the definitive truth is. But it can often be used to discard weak arguments. Usually, I focus on criticizing the left in this regard, but let's make an effort to be intellectually honest and scrutinize the right for once.

The argument of the right is that educational spending is wasteful, and doesn't produce outcomes. They base this on the fact that the United States already spends more on education than almost all other nations, and more than ever in history.

John Stossel for instance writes that:

"If money were the solution, the problem would already be solved"

The difficulty is that education spending goes up when social needs go up. This is the classical head-ache pill example: just because people with worse head-aches take more pills, it doesn't mean the pills are ineffective or even causing the pain. The same argument is also true of spending on the police. Cities with more crime have more police, not because the police are causing higher crime rates, but because they are a response to high crime.

So for instance the entity that spends the most on education and has the worse test scores in the U.S is the District of Columbia. This doesn't prove anything, because D.C also has much more poverty and many more disadvantaged students. The relevant policy question is what would happen to test scores if D.C spent less.

Let's start with a simple cross-country regression of the normal, unadjusted PISA-scores (the ones the media keeps reporting) and education spending per pupil, as measured by the OECD.

It turns out that the correlation is negative (-0.2). The United States spends the second most of any country, and has below average test-scores. Ethnically homogeneous Japan, South Korea and Finland spend at average rates and have the best test scores. Tiny, ethnically homogeneous and "hungry" Estonia spends less than half as much as the United States and Norway on education but has far better test scores.

Argument over. Doing the comparison in the New York Times way, Conservatives win, and Liberals lose.

But what if we do it my way instead? Let's do a crude demographic adjustment, and compare European-origin groups to other European-origin groups, just as I did before. The methodology is explained and defended in detail here.

For Europe, I remove first and second generation immigrants. For U.S, I remove Latinos, Asians and all other groups than non-Hispanic whites (two thirds of the U.S population is non-Hispanic white).

Now, prepare to witness the power of the simple demographic transformation.

The relationship between spending and outcomes is now reversed. More spending is indeed associated with better PISA scores. The United States does well, as predicted by how much she spends on education. Finland with her admired education system is still an outlier, which I guess indicates there really is something special about Finland and their school system.

The graph suddenly makes more intuitive sense, doesn't it?

But we have more variation in data we have not used. Why not break out each U.S state and treat it as a separate observation? We have comparable data on 8-grader test scores for each state from the NAEP which I have converted to PISA-equivalents (method explained here).

We also have access to education spending per pupil for each U.S state, courtesy of U.S Census.

Again first we do the standard, unadjusted PISA-scores for this pooled sample of countries and U.S states.

A flat, slightly downward sloping curve, countries and states that spend more don't appear to score any higher.

Notice that Utah, Finland and South Korea outscore California, even though they spend half as much (they don't need to spend more, because they lack California socially disadvantaged masses). Minnesota beats Sweden, but generally U.S states do poorly, even though they spend more than Europe.

Another victory for Republicans. Clearly doubling money on education has no effect, and may even - somehow - make your countries children score a little less.

Now, look at the exact same graph, only with my standard crude demographic adjustment (For Europe, first and second generation immigrants are removed. For U.S states, all demographic groups other than non-Hispanic whites are removed from the sample).

Again the results are reversed after a demographic adjustment. The advantage of the U.S and the advantage of rich U.S states that spend a lot on education suddenly becomes apparent. More public spending on schools is associated with better outcomes. The mystery of the missing correlation is solved.

Lastly, let's look at test scored of African Americans and state educational spending.

Of course these graphs alone do not prove that the driving factor is education spending itself. In theory, it could be because students in rich states are different, for example in terms of family income. But the relationship still represents powerful suggestive evidence in favor of public expenditure on education.

However the left in the United States doesn't use this argument, because they are ideologically averse to demographic adjustments having to do with race and ethnicity (most of them consider all statistical generalizations about race and ethnicity somehow offensive, regardless of why you are doing it).

The result of liberals political correctness is that they are depriving themselves of a very important argument in a very important debate.


  1. On some level, this had to be a measurement problem. Does anyone really believe that more money makes education worse?

    I'm surprised at how dear those few extra points of PISA come. Going from 495 to 535 costs $16 thousand a year. That's about $200 grand to purchase 40 points which is worth about two years of extra schooling ( Maybe we should have lousy and cheap schooling and simply have a massive graduation bonus.

  2. "Does anyone really believe that more money makes education worse?"

    No, but a lot of conservative believe that it has no effect after a certain point.

    You are completely right that it is very expensive, just like health care, and in both cases because of diminishing marginal effects in providing services.

    But conservatives defend higher American health care spending, even though we have diminishing effects there as well.

    Also, keep in mind that education does much more than raising test scores.

    Heckman has shown very convincingly that early intervention for disadvantaged children is worth the expenditure for society. This not because test scores go up much, but because of human capital and norm formation makes the children have better outcomes in life.

  3. But 40 points on PISA is a lot. That's more than half the majority/minority achievement gap.

    The Unites States spends about 3000$ more per pupil per year on primary education than Western Europe, and appears to score about 0.18 deviations higher than Western Europe.

    I for one am willing to pay the extra taxes for this advantage. I understand if others are not.

  4. Again the results are reversed after a demographic adjustment. The advantage of the U.S and the advantage of rich U.S states that spend a lot on education suddenly becomes apparent. More public spending on schools is associated with better outcomes. Puzzle solved.


    I'm generally very favorably disposed to your method of analysis, that is, isolating variables and correcting for confounds, but I think that you're being too quick on the "puzzle solved" conclusion.

    How do rich states become rich and what enables them to spend a lot on education and when you measure the education outcomes of the students are you actually measuring the influence of the educational process or is there a confound mixed in batch, a factor that is common to both educational outcomes in students and the socioeconomic status of their families?

    We've both seen simplistic models which posit that socioeconomic status ---> student outcome, but this model misidentifies the independent variable, which is intelligence. Intelligence helps determine socioeconomic status and it also helps determine student outcomes. If you really want to know what the effect of socioeconomic status is on student outcome you need to control for the intelligence of the parents and children.

    When this individual dynamic is scaled upwards to the state level, then we're still dealing with the same process but now it's more difficult to isolate and measure the independent variable, which is the aggregated metric for the intelligence of the state's residents.

    Secondly, when we compare state spending on education we need to address the cost of living disparities between states. The flipside to this question is "what is the extra education funding buying for the student?" If the extra funding is primarily going to a higher teacher salary or a nice facility, because the community is willing to spend more on nicer facilities because education is a superior good, then the black box of education process is not what you've isolated and measured.

  5. Heckman has shown very convincingly that early intervention for disadvantaged children is worth the expenditure for society. This not because test scores go up much, but because of human capital and norm formation makes the children have better outcomes in life.

    The Canadians and the Australians learned bitter lessons from discovering the same thing. The Stolen Generations sagas played out in both countries because they too tried to displace the "wrong" lessons taught by the students' parents and replace those lessons with better lessons taught by teachers and counselors.

  6. Any chance you could do the same with %GDP spending?

    Also, I think a lot of the people pushing for vouchers (like myself) think the analysis above is a bit simplistic. The latest voucher experiment in DC showed that: a) the same scores could be reached (not better as many had claimed), b) satisfaction rates were higher among parents, and c) all of that for half the cost per pupil of the public schools. Within that small experiment, it's implied that spending twice as much among the lottery winners did nothing for their achievement.

    This certainly doesn't discount the good work you've done here (should definitely give pause to US conservatives), but I think there are lots of details covered up by macro studies.

  7. Great work, but does it not still show that demographic factors dominate spending?

    If one looked at Hispanics and Asians, that might give a further sense of how demographic factors play out.

    I would also be very interested how much spending correlation was about resources, how much teacher salaries and how much class sizes.

    Of course, it is easy to suggest further work for you :)

  8. Tino,

    I was wondering how you would interpret this graph:

    What explains the flatlining of test scores despite surging federal education expenditures? Should federal expenditures be regarded as a different animal than state and local education spending?

  9. However the left in the United States doesn't use this argument, because they are ideologically averse to demographic adjustments having to do with race and ethnicity (most of them consider all statistical generalizations about race and ethnicity somehow offensive, regardless of why you are doing it).

    Yes and I find it is very amusing to watch in debates.

    BTW it reminds me of who drinks diet soda. If you take it as evidence that drinking diet soda helps on lose weight you might expect the thin people to select the diet on the other hand if you think the people who are motivated to reduce weight you expect fatter people to select the diet.

    This why we need randomized trials.

    BTW No random trial but my observation is that fatter people are more likely to drink diet soda.

  10. Colin the graph shows the relative rise in the number children from groups that score less well.

  11. This comment has been removed by the author.

  12. Tangoman:

    You are right that the correlation of between spending and test scores doesn't prove spending is causing higher test scores. It could be that states that can afford more spending have better educated parents.

    But now at least no one can say that there is no correlation at all between spending and test-results.

    More precise studies have shown that American schools raise test-scores by about 0.1 s.d per year of extra schooling. This effect size is probably local, which means that if you put them in school for 5 extra years you should expect diminishing returns.

    I have done some calculations on IQ-data by state, but I am not going to put it online.

    However one result is that differences in IQ between American states are smaller than you think, and smaller than test-score gaps.

    You are also right about the cost of living. But adjusting for cost of living whiten the United States is notoriously difficult. My international data is adjusted for the cost of living.

    Regarding the Canadian/Australian argument, I completely disagree about the analogy. That was about taking the children by force from the parents. This is about giving the parents something they like, which is in practice day-care and longer school days, and better behaved children.


    I have never seen any evidence that vouchers worsen results. On test scores, school choice either has small positive effects or no detectable effects.

    So I am not against vouchers. I simply don't expect vouchers to fix the achievement gap, because we already know the achievement gap comes from home, not mainly from the school. Extensive use of vouchers in Sweden have not closed the immigrant/Swede test-score gap.

    Libertarians shouldn't make promises to the American public that the free market cannot deliver on.

    One thing we know is that poorer and minority children in the United States do better in catholic school. So my personal view is we need more schools that focus on norm formation and discipline.

    Lorenzo: I will keep your suggestions in mind, for the future.

    The CATO graph uses not one but two libertarian cheat-arguments:

    They correct for inflation, but not for wage cost. Teacher and administrator salaries are higher in 2008 than 1970, in real terms.

    The schools have no choice but to pay for this increase in cost just to maintain output at previous levels. This problem that libertarians often ignores is referred to as Baumols disease by economists.

    , whiBut there is another, a little more interesting cheat going on, which is demographic composition change, as JW suggested.

    Let's look at NAEP test scores by ethnicity:

    White: 1975-2008: +11

    Hispanic: 1975-2008: +24

    African-American: +23

    Other (includes Asian): +29

    U.S average: 1975-2008: +10

    How is this possible? Because in the 1970s, children from affluent ethnicities (White and Asian) were well over 80% of children. In 2008, they are less than 60%. So the average score of the U.S is stagnating, while all American groups, particularly minorities, are doing better, simple because of the changing demographic composition of the nation.

  13. The NAEP figures above for 1975-2008 are for reading for 4th graders, by the way. The graphs are reading+math for 8th graders.

    Just to avoid confusion.

    1. The availability and reliance on these two data points are a problem in themselves. Our understanding of educational progress, or the lack of it, would benefit from annual testing on a national level.

  14. To illustrate my point about the cost of labor:

    Since 1970, real expenditure per pupil has gone up by about 250%.

    Since 1970, the teacher-pupil ratio has only increased by about 50%.

    In 1970 teachers: 2.3 million, pupils 51 million.

    In 2008 teachers 3.7 million, pupils 56 million.

  15. I agree that 40 points is significant,indeed worth almost 2 years of education. Nevertheless, I still maintain it is a bad value, even if the goal is maximizing education for the money spent. My point was that for far less than the cost of delivering more intensive education we could deliver more extensive education by say making community college free and paying people $50,000 to graduate high school and community college.

    The Baumol cost disease argument is a bit of a red herring. I don't object to your math, but I think the argument misleads. The government near-monopoly on education combined with teacher union dominance of education policy has lead to a focus on keeping class size small and employing a large number of teachers. Of course their wages we going to go up along with other college educated people and education costs would go up with productivity and not inflation. On the other hand, we don't have to keep this educational organization fixed. We could try computer or TV based education as well as experiment with larger class sizes.

    We don't really know that education has to be organized with constant labor inputs, so it is unfair to to say that education costs would naturally have to increase. If, for example, we still made cars with lots of manual labor, car costs would have gone up with wages, but raising wages forced the car companies to substitute in to less labor intensive forms of production.

    I'm not saying that we know there are good alternatives, but we really wouldn't know at this point because of the enormous dominance of labor intensive public teaching.

  16. Let me say something about libertarians and why I don't call myself a libertarian. This is not about all libertarians, just one particular (and very common) stream in America and Sweden:

    Libertarians suffer from ideological over-confidence.

    They have one principle: individual freedom, no government, and no tradeoffs between principles and other socially desirable outcomes such as equality. They don't need a lot of data or debate to make up their minds on each issue.

    They have the same position on all issues: minimize government.

    During the last half a century, the political debate centered around economics and central planning. In turns out that in that particular arena, libertarian dogma works extremely well.

    Because of the first welfare theorem, or the invisible hand or whatever you want to call it, it actually seems that maximal individual freedom works in reality and outperformed other systems, as witnessed in 19th century America or 20th century Hong-Kong.

    Libertarians don't seem to understand that the success of minimizing government is due to the specific nature of market economics, and not divine laws. Just because decentralized planning is the best mechanism to build computers doesn't mean reality will always follow your ideology.

    Schools are one example where libertarianism theory has low predictive power. Illegal immigration is another more important example, where "maximizing" one narrow principle (the government isn't allowed to determine who gets to move into your country) doesn't coincide with favorable social outcomes.

    But because libertarians always win the debate in economics without having detailed facts or stringent arguments, some of them are under the impression that they don't need detailed facts and stringent arguments in general.

  17. Any way we could obtain educational data controlling for the IQ (or decent proxy therof) of the Parents? Like an analysis of the 2nd generation from the study population used in The Bell Curve?

  18. Tino, thanks for the response.

  19. Jehu:

    You can control for the education of the parents with most data-sets (including PISA). That is sufficiently close to IQ.

    If you want parental IQ, NLSY has the IQ of the mothers in the original panel.

    Certainly in the future we are likely to get new technology that will allow us to escape from Baumol's cost disease.

    But between 1970-2009, we didn't. We instead got computers and cell phones and video games, which probably made teaching more difficult, especially outside the classroom.

  20. At the root of the spending issue is the question of what the additional spending is buying for the student. The best case for additional spending is when one can demonstrate that the spending is going to be directed to educational aids which the students currently lack and which have a demonstrated effect on improved learning. For instance, if students have to share textbooks, then additional funding to enable each student to have their own textbook runs a higher chance of improving student outcomes. The alternative of ineffective spending might be to spend a whack of money on putting LCD projectors in every classroom and hiring 2 computer support technicians to help teachers with their email programs. What we might see as a result is teacher efficiency increasing, in regards to intra-school communication and a lessening of their own prep time, and more colorful or entertaining material presented via the LCD projectors, but this is unlikely to increase student test scores.

    So my point is that I believe that there are diminishing marginal returns on increased spending, spending beyond providing fundamentals isn't very effective. Kansas City was a pretty clear demonstration of the latter point.

  21. It is expensive to translate money into test-scores. But it seems possible.

    Many readers keep asking about IQ. But IQ is not everything, and schools can improve test-scores conditional on a given IQ. Let me illustrate:

    Southern whites score no higher than Western-Europeans on IQ tests. According to the NLSY97, the score of southern whites (102) is one IQ point below Europe. Unlike Asian immigrants, no one can accuse southern whites of being a privileged, positively selected group on IQ or income or parental education.

    However schools in the American South spend about $9.100 per pupil, which is about 30% more than the average of Western Europe.

    Southern Whites 8-graders score 0.16 standard deviation above the U.S national average on reading+math,. This is equivalent to a PISA score of 513, compared to 506 as the average of Western Europe (excluding first and second generation immigrants).

    Maybe it's better norms, or less ideological damage to the education system from modern fads in education.

    Nevertheless my best explanation for this advantage is that this is because America is richer, and can spend more on public schools.

  22. Here are some of the things that concerns me with the Kansas-example:

    * There is no control group. Perhaps test-scores would have declined or declined more without the intervention? Impossible to tell without a control group.

    In some periods, test scores drop year after year, for reasons having to do nothing with education funding.

    The Kansas-city experiment started just as the crack-epidemic started.

    * The controlled experiments that we do have (from Tennessee) demonstrate a small increase in test-scores from more resources.

    *Also, quasi-experimental studies in economics show that an extra year in school also tend to slightly increased test scores.

    * Having more funds for education in normal times may be different than suddenly throwing billions at a problem in panic, like Kansas city did.

    * I would like to see the Kansas data myself.

  23. Many readers keep asking about IQ. But IQ is not everything, and schools can improve test-scores conditional on a given IQ. Let me illustrate:

    IQ most certainly isn't everything but it is the single largest identifiable factor at work.

  24. Tino,

    Thinking some more about your response and have a question about the applicability of Baumol's cost disease. According to this:

    Public school teachers are paid 61% more per hour than private school teachers, on average nationwide.

    If accurate, this would seem to suggest that the increase in teacher pay is far from inevitable, and is likely due more to the bargaining power of teacher unions than market conditions, no?

  25. Thanks.

    I would encourage you to look more into cost-of-living adjustments within the U.S. New Jersey, for example, is a an expensive state with some expensive people in it. Mississippi is not.

    One difficulty is that NAEP scores are aggregated by state while most cost of living data (such as ACCRA) is by metropolitan area. You are supposed to buy ACCRA data, but the state of Missouri puts up simple averages of cost of living for each state here:

    What I think Missouri is doing, however, is that they are weighting each metropolitan area in a state equally, rather than by population.

    That page claims that New Jersey's cost of living is 30% above the national average and Mississippi's is 8% below, mostly due to housing costs being more than twice as much in NJ. That's not implausible, even if Missouri's web page is simplistic.

    I believe ACCRA, which is intended for use by firms relocating employees, looks at the cost of buying a home as well as renting a home. But that's reasonable when looking at the cost of schools, since a big chunk is the cost of schoolteachers, most of whom are supposed to be middle class college graduates (i.e., the kind of people who expect to, eventually, buy rather than rent.

  26. The other issue is whether the kind of cost per pupil numbers we normally see (e.g., in the $10k range for the U.S.) fully reflect all the costs. For example, I have heard, but can't say for sure, that the cost per pupil in California that's usually cited does NOT include the enormous capital expenditures of building new schools. LAUSD, for example, typically spends several billion dollars per year building schools, and that doesn't necessarily get included in the annual operating costs. The capital cost of the new Robert F. Kennedy Schools on Wilshire Blvd. is over $1.4 million per student.

    Another question is the cost of defined-benefit pensions. If the stock market goes up 10% per year, employee pensions are pretty cheap. If it goes up 2% per year, they aren't.

  27. An alternative to cost of living adjustments is school spending as a percent of GDP by State. For example, the nominal per capita GDP of NJ in 2009 was $50k and in Mississippi was $30k.

  28. Tino, you said:

    ... So I am not against vouchers. I simply don't expect vouchers to fix the achievement gap, because we already know the achievement gap comes from home, not mainly from the school. Extensive use of vouchers in Sweden have not closed the immigrant/Swede test-score gap.

    Libertarians shouldn't make promises to the American public that the free market cannot deliver on.


    I agree with that, as per my DC example. I too think the push for privitization has overpromised results. However, I think that experiment calls into question the meaning of your results above.

    Namely, can one say "increased spending per pupil produces increased scores"? Well, there's clearly a correlation that implies that in your work above.

    But bringing up the DC example (or any other voucher experiment), one can see that decreasing spending per pupil by up to 50% does not decrease scores.

    So if reducing spending per pupil (via vouchers) has not decreased scores, what confidence can one have in increasing scores by increasing spending?

    This has direct political implications in the US: Is any given area simply underfunding education? Increased spending can improve scores! Or is something beyond simple spending the problem? Vouchers allow the same results at half the cost!

    Basically I guess my point it that your macro correlations here are in conflict with the few voucher experiments (slightly more controlled but also biased).

  29. My graph only shows correlation. Not causation. Let me be emphatic about this.

    However while correlation alone doesn’t prove causation, it is often a necessary first step.

    If you cannot convince people the variables are positively related, how can you hope to convince them that one is causing the other?

    Liberals have been stock on step one, and have no good answer to conservatives who cite the lack of a positive correlation between spending and results.

    (I cite one paper in the comments that does include experimental evidence linking smaller classrooms to slightly better test scores, so some casual evidence exists out there, just not in the graph).

    I will do the do-diligence the readers have asked, but not anyday soon.

  30. Eric Hanushek who takes a gimlet eyed view of these issues estimates that moving the U.S. near the top of international math and science rankings would have a present value of $100 trillion.

    Hanushek, Eric A. 2010 The Economic Value of Higher Teacher Quality. National Bureau of Economic Research Working Paper 16606 (December). Available at:

    By the way the Tennessee class size experiments clearly show the analytic payoff to randomized trials. In this case both students AND teachers were randomly assigned to treatment and control groups. It now seems that previous studies were confounded by two factors: OTBE reducing class size dilutes teacher quality and the rational propensity of principals to assign the best teachers to larger classes.

  31. Richard Vedder's research leads him to believe that at current levels additional education spending lowers economic output. He does say that his research is inconclusive.

  32. Then there is the problem of comparing European descendants in the USA to Europeans in Europe - couldn't an argument be made that those here are of a different sort? Maybe more ambitious, willing to take risks, and harder workers? All things that you would think would make you do better on tests.

  33. Anyone want to take a stab at explaining why PISA 10th grade reading scores for Hispanic students in America are in the same neighborhood as those in Mexico, despite the fact that the USA spends four times as much per pupil?

  34. Thanks for the nice information,as a Testking Expert it is really helpful for me with research point of view...

  35. Nice discussion above which will helpful to grab and explore the relationship between education spending and test scores,As a Testking Expert I think these both have the close relation with each other.

  36. That's just so generous of you that you have provided us with this kind of worthy information.
    how to make hair grow faster

  37. This is a good common sense article. Very helpful to one who is just finding the resources about this part. It will certainly help educate me.

    graduate schemes

  38. Tino,
    Trying to reconcile the aggregate negative correlation in the US and by country, with the positive correlation with spending for European descent and African Americans: Aggregate populations show a negative correlation with spending. Demographic corrections, looking at first European descent, and separately, African Americans show a positive correlation.

    Question: To reconcile these differences with the overall negative correlation, shouldn't your data show that there is a negative correlation for the US Asian and/or Hispanic population subgroups?

  39. Tino,
    As an engineer who uses a lot of statistical methods in my occupation I see weak correlation in the graphs. Because of this it looks like you were able to pick a data set to corroborate your desired outcome. Drop a country here, add a different country there, cull the extremes for the US-states and I bet we could invert the slopes of every correlation line on your graphs.

    Just a thought.

    The real question that no one ever seems to ask: Why? I only get one brief time for my life here on earth so what if I do not want to spend it learning statistics and would prefer driving a truck around this beautiful world or working the high range herding some sheep? Why do we try so hard to make others into our own image? Or our own world-view?

    This is the rarely spoken but real thought of the true libertarian. Libertarian is one of the lesser-hypocritical political philosophies.

  40. Your blogs are truly awesome I have no words to praise for your blogs. Welcome to Geo Top News

  41. According to the problem stated above it needs to be investigated by secondary students in all countries that will allow me and my college make sense of personal studying in all ways possible. This company will introduce it in alternative meaning and other ways with general sense and facts. This kind of information can be fully presented at absolutely cheap term papers for sale made by smart writers.

  42. Great post guys, you have done the nice job, and solved my problem I was in trouble. Thanks a lot. this contact form

  43. Very nice website and articles. I will be realy very pleased to visit your website. Now I am found that we actually want. I check your website everyday and attempt to learn something from a blog. Thank you and looking forward to your new submit.

    software drivers download


Google Analytics Alternative