Saturday, January 1, 2011

How well do above average American States do compared to above average countries in test scores?

Over at National Review Online, Reihan Salam has some complaints about my PISA-comparison.

First, he notes that even if European-Americans and Asian-Americans are doing well, minorities are not, and they are going to be a larger and larger part of the U.S population. This is completely true.

However my analysis tells us two new things:

* The problem is not likely to be the American public school system itself. If it was the education system that constituted the problem, Asian- and European-descended students (70% of the U.S population) would not be performing so well relative to other nations. After all, they are in the same government controlled system. Studies that use more sophisticated methods to control for school quality come to the same conclusion: It's not the schools.

* Europe has a similar achievement test gap between majority/minority. It's just that the size of the European minority population is still small, so it doesn't affect the national average as much.

But if your aim is to isolate the treatment effect of a policy (in this case the design of the school system), the size of the group impacted is irrelevant. Turks in Sweden have approximately the same gap with Swedish natives as Mexicans in California to California natives. This should conclusively end one discussion: the Scandinavian school system has not discovered how to solve the majority/minority achievement gap.

If you want to explain why Sweden has less aggregate economic and social inequality, the relative size of the minority groups in question matter. If you want to compare the efficiency of school systems, it doesn't. It is immaterial for what we are doing, a comparison of how well the school system closes the achievement gap, that Turks are less than 1% of the Swedish population and Hispanics 30% of the Californian population. It is sufficient to show that they have the same gap. If anything, European schools should be doing better considering the smaller size of the disadvantaged groups.

Jumping up and down in excitement, and congratulating Swedish and other Social Democratic school systems for having "solved" the achievement gap while sending fact finding teams there to learn from their system is ridiculous. Their system hasn't solved anything, the same gap is there, they just have few minorities so the gap influences national averages less and is less apparent to outsiders.

So under closer scrutiny it turns out that the "secret" of Scandinavia's lack of a achievement gap is that until recently, they didn't let many minorities into their countries. Once they started to, the achievement gap among socially disadvantaged groups emerged, a little smaller in magnitude but likely due to the same causes.

In the 1960s, a naive liberal in the U.S could be excused for assuming that Sweden's success was due to their education or welfare policy, and that Sweden had "solved" societal problems such as ethnic achievement gaps. Other countries could just copy whatever Scandinavia was doing in order to deal with their social problems.

Now that we have all seen footage from Scandinavian ghettos, as well as national statistics from Sweden and the rest of Europe confirming an achievement gap with minorities, it should be clear that this theory was wrong. So why are we still spitting on American public schools and congratulating Social Democratic ones for producing roughly the same outcomes?

2. Salam believes that the U.S should be doing even better, and asks about the performance of Greenwich (Connecticut).

"The NEA, and Tino, would have you believe that it is very, very impressive that Greenwich is treading water with Yazoo...."the United States, in terms of GDP per capita, is closer to being the Greenwich [Connecticut] of the OECD than the Yazoo City [Mississippi]."

He has two points here. First, he believes that with America's resource advantage, the U.S should be outperforming other countries even more. I might write more on this in detail later.

The resource gap of the U.S vs. Western Europe is not the same as that between Connecticut and Mississippi. As I have written here, Connecticut has 110% higher per capita income than Mississippi. Meanwhile the United States has "only" 36% higher per capita income than Western Europe.

If we instead of Mississippi compare Connecticut with the U.S average, it fits better. Connecticut is 35% above the U.S national average in per capita income, and the U.S national average is about 35% above Europe. In this sense the United States is equivalent of a (albeit much larger) 'Connecticut of the OECD', which is what Salam wrote.

Nationally, Connecticut as a whole scores a little less than 0.2 standard deviations above the U.S mean. Well, my analysis suggested Americans of European-origin score a little less than 0.2 standard deviations above Western Europeans in Western Europe.

So if this is the criticism, it turns out that the U.S educational advantage is approximately what the U.S economic advantage would predict.

I can't fault Mr. Salam for wanting more output for the input American tax payers are spending. However my analysis suggests that unlike conventional wisdom on the political right, public education spending is not just wasted. It appears successful in buying America better test scores, for each given demographic group, and of course with diminishing marginal returns.

Second, Salam is not sufficiently "impressed" by the average American advantage. Let's see if we can impress him.

The data I presented was national averages. In the case of the U.S, this was the average of rich cities such as Greenwich Connecticut and poor cities such as Yazoo Mississippi. When you aggregate a large group (200 million people) the variance goes down, as Greenwich and Yazoo cancel each other out.

This is important to keep in mind when comparing with small outliers such as Hong-Kong and Finland. Even if the U.S beats Europe on average, the best performing tiny part of Europe will likely beat the average of the U.S. What I was impressed by was the fact that one large group (all Americans from Europe) beat another large group (all Western European native born) by a sizable margin, as well as beating most of the small, top performing outliers.

But I would be happy to disaggregate to defend my argument.

PISA doesn't report the score of U.S states, because the sample sizes are too small. However there is another similar international test of math and science of 8-graders that does have data on individual states, TIMSS. They also have data on 2 U.S states, Minnesota and Massachusetts (both are similar to Connecticut in terms of test scores nationally, according to other data).

Remember that the United States scores relatively worse on TIMMS than PISA, especially compared to Asian countries, because PISA includes reading (which Americans are good at), whereas TIMSS is only a test of math and science.

Massachusetts and Minnesota each score higher than every single European country, although Finland is not included.

The achievement gap between Minnesota and Sweden is as large as the gap between Sweden and Bulgaria!

Keep in mind that this is the average of Minnesota and Massachusetts, including immigrants and minority students. If you were for example to break out only Asian-Americans in one of those states, the results are going to be stratospheric.

For fun, I also used a third large test of 8-graders (NAEP) which allows me to break out Asian-Americans in Connecticut, but requires more work to make it comparable with PISA. If Asian-Americans in Connecticut score as high relative to other Americans on PISA as they do on the NAEP relative to other Americans, they would beat every single Asian country in PISA. So would Asian-Americans in Texas and New York and several other states.

Note: this second graph relies on translating scores on one standardized test of 8-graders (NAEP) to PISA-equivalent scores, assuming that your group scored 10% (in standard deviation terms) above the American national average on the NAEP means your group would also score 10% above the American national average on PISA. I know that those readers who are resistant to my argument are going to nit-pick and are going to to be reluctant to accept the comparison.

However regardless if you buy my second graph, what is very important for readers to understand is that the first graph with TIMSS is not based on me doing any sort of adjustment of the data. It's simply the average of the scores, as reported by the Department of Education, on a test that is internationally standardized.

This doesn't definitively settle the debate about the quality of education system, because of biased selection of Asian immigrants. But I hope this example illustrates that affluent groups in the United States destroy the international competition, and are competitive even when compared to small rich countries such as Hong-Kong.


  1. About graph 2:

    Conneticut is not a historical area of Asian-American settlement (unlike California). It's highly probable that the Asian-Americans there do not have socio-economic profiles representative of the national average (for Asian-Americans). Why not use the numbers for Hawaii and California, the Asian equivalents of Minnesota, instead? The NAEP figures for Hawaii are not particularly impressive, maybe about 3 to 4 percent above the national average, which puts them on par with the California Asian-American average.

  2. Fox:

    The purpose is not getting at "representative" groups. The national average already takes care of this.

    The purpose is to illustrate how small rich states do, when comparing with small rich states such as Singapore and Hong-Kong.

    Hawaii is a terrible example because they lump together North-East-Asians with a bunch of other groups, such as Hawaiian natives and Pacific Islanders.

  3. I see that the NYT and Niklas Kristof has fallen for charlatans Wilkinson and Picket and their unscientific snake-oil:

    I have previously explained that the "research" in Wilkinson and Picket was debunked years ago.

  4. I've always instinctively distrusted Kristoff. He seems like a swell guy trying to make a difference, nevertheless...

    Back in the nineties I used to pick up those free newsprint publications touting some combination of "Natural Healing/Self Help Spirituality Guides To Oneness and Pure Enlightenment Through Colonic Cleansing and Getting in Touch With Your Inner Spirit Animal Shamanic Dream Master" etc. Every head shot of every author/spiritual master, especially those offering their healing services in the back pages, featured a laughably uniform beatific facial expression, as though the photographer had captured the precise moment of transcendence in the subject. Either that or they had always had their portrait taken apres a very satisfying bowel movement. (sorry to be crude, but hey, I'm just keeping it real! hehe).

    Well, by now I think my point is obvious. Kristoff's head shot shows precisely the same smug, uber enlightened expression. And it's not just his portrait, I've seen the same look in other photos of him. I like that he gets into some pretty hairy places around the world and often writes with an earnest humanity which is admirable. Yet, as when he courageously reported from Egypt's Tahriri Square that the United States should DO SOMETHING, presumably militarily, he strikes me as mortally naive. Want to be a do-gooder, that's fantastic. Want to be an ideologically driven do-gooder? Everyone duck.

    Although I'm lacking in time, education, and perhaps brain cells to get to the bottom of this particular scrum, I sense that Tino and company are on to something. At the risk of thinking with my gut, (we all know how well that method has served our country in the last decade), I'll go ahead and raise my glass (of tap water) to the writers of this blog in thanks and cheer your efforts to debunk this foolish notion. The cheating scandals are just warming up, and we as a nation are going to have to face up to some very hard truths soon. It will take just this sort of dogged research to break through the bull chips passing for conventional wisdom.

    Good night and good luck.

  5. Tino,

    Given your proximity and expertise, I wonder if you could comment on the scandalous rift between "saltwater" and "freshwater" schools of economics, with your current institution, the University of Chicago, bearing the standard for the "freshwater" idea that (if I'm not totally confused) Keynes was wrong about the efficacy of government stimulus in averting/cushioning the effects of a recession. As I understand, the champion of the "saltwaters" would be Krugman, who has been scathing toward what he regards as the institutional mendacity of the opposition, so much so that he wonders whether in light of such a fundamental disagreement can the field of economics even be considered a science at all?

    I'm open to your opinion, although I admit I tend to side with Krugman, and share his frustration that at such a critical time, when we desperately need clear economic guidance, the field is degenerating into a circular firing squad of squabbling academics with so many dubious affiliations with industry that no wonder the product of their incestuous unions is a stillborn policy with three arms and four heads fit only for plopping into a jar of formaldehyde and charging the rubes to gawk at.

    Lord I need to be in bed I apologize for my ridiculous prose but still, if you can get past it, I'm seriously interested in your take.

    Thanks much.

  6. I'm a compulsive commenter who has no patience for signing in or making profiles etc...just wanted to make clear that Unknown above is "The Dude" above that

    I'm going to bed now but I've bookmarked this page and I'll check back from time to time to see if I get a reply. If you do reply I'll be as excited as Chris Crocker with a backstage pass at a Britney Spears concert so please if you can spare the time you'd make one middle-aged married dude very happy. Maybe not THAT happy, but still.

  7. As far as any student can see, smart thoughts as mental activity may involve individual skills needed at college. Young people are looking for unique samples of research proposals as we all hope experienced writers of such top quality possess creative thinking style no matter what the topic of the study would be. High school audience can make it best to make the educational process more or less simple.


Google Analytics Alternative