Economists Do It With Models

Warning: “graphic” content…

Bookmark and Share
On “The Least Surprising Correlation Of All Time”, SAT Scores Edition…

August 31st, 2009 · 22 Comments
Fun With Data

On Thursday, the NYT Economix Blog reported on the correlation between (self-reported) family income and SAT scores. The main thesis is summarized by the following graphic:

My first thought was “How on earth do these kids know how much their parents make?” since, when I was in high school, I really had very little clue how much dough my parents were pulling down. Actually, I still don’t really know. (Feel free to enlighten me, Mom.) From an economic perspective, my initial thought was “This is clearly a correlation rather than necessarily a causal relationship.” (For a primer on correlation vs. causation, click here.) I really don’t mean to be as snarky as this sounds (okay, maybe I do), but perhaps something that announces itself as an economics blog should be careful to be, well, either economically sound in its analysis or careful about pointing out the limitations of its interpretation. I am apparently not alone in this opinion, and I will outline for you the s**tstorm that ensued:

Step 1: Greg Mankiw’s blog gives the above graph the title “The Least Surprising Correlation of All Time”. (I would argue that this one showing the relationship between beers consumed and perceived employment outlook could be a contender for that title. But I digress.) He then subtly (and rightly, in my opinion) criticizes the Economix Blog for not specifically cautioning against inferring a causal relationship from this data. He goes on to detail the problem with the causal interpretation:

This graph is a good example of omitted variable bias, a statistical issue discussed in Chapter 2 of my favorite textbook. The key omitted variable here is parents’ IQ. Smart parents make more money and pass those good genes on to their offspring.

Suppose we were to graph average SAT scores by the number of bathrooms a student has in his or her family home. That curve would also likely slope upward. (After all, people with more money buy larger homes with more bathrooms.) But it would be a mistake to conclude that installing an extra toilet raises yours kids’ SAT scores.

It would be interesting to see the above graph reproduced for adopted children only. I bet that the curve would be a lot flatter.

I like his diving into the absurd bathroom analogy almost as much as I dislike him referring to his “favorite textbook.” (It’s his textbook, in case that wasn’t obvious. I feel like a dash more irony or self-mocking is needed to make statements like that work, but in a small way I commend the effort.) Now, he’s right in that it *should* be enlightening to see the graph reproduced for adopted children, since this would theoretically take parents’ intelligence as a genetic driver of childrens’ SAT scores out of the equation. But wait, there’s more…

Step 2: I read the following in my Twitter feed, via @mattyglesias (See here if you are not familiar): RT @conorjclarke: 15 seconds of googling 2 find problems with mankiw: high-income adoptees are 12 IQ points higher http://bit.ly/2RzLIp. Wow, people are quick to jump on the “Mankiw is wrong” bandwagon, though in checking my Google Reader I notice that Mankiw’s post went up at 5:24am. What in the hell is he doing up and writing economic rants that early? (The Twitter comment didn’t come until 10:52am, which I consider to be a much more reasonable hour.) Anyway, the link is (via Google Books) to a book entitled Intelligence and How to Get It: Why Schools and Cultures Count, and the relevant information is the following:

On average, the biological children of high-SES [socioeconomic status] parents had IQs that were 12 points higher than those of low-SES parents, regardless of whether they were raised by high-SES or low-SES parents…

The crucial finding is that children adopted by high-SES parents had IQs that averaged 12 points higher than those adopted by low-SES parents- and this was true whether the biological mothers of the children were of low or high SES.

The book goes on to give further evidence from a natural experiment to support the claim that environmental factors are significant determinants of IQ. So wait- high-SES parents can actually cause their kids to have higher IQ’s? If we believe that there is a positive relationship between IQ and SAT scores, this would imply that the relationship between SES and SAT scores would persist even if we looked exclusively at adopted children. Stay tuned…

Step 3: The s**tstorm continues with a post by Brad DeLong where he quotes what Mankiw wrote in his post and then starts commentary with:

But merely saying that correlation is not always causation and dropping the issue is, I think, profoundly unhelpful–and shows a… lack of work ethic as well.

For the record, that is about how civil academic economists generally are to each other. He follows up with a lot of words and math and then the following conclusion:

The rule of thumb, I think, is that half of the income-test score correlation is due to the correlation of your test scores with your parents’ IQ; and half of the income-test score correlation is [sic] coing purely from the advantages provided by that component of wealth uncorrelated with your parents’ (genetic and environmental!) IQ.

The curve is less steep, but there is definitely a “what” here to be thought about.

Sheesh. Okay, well, let me throw my brain into the ring here:

First, you can see the data from The College Board that everyone is using here. The chart data comes from the page labeled 4 in the report, which is actually page 8 of the pdf. You will notice that, in addition to the scores versus income category, there is a category for scores versus highest level of parental education. Now, one would expect that SES and education are fairly correlated, so it’s not surprising that we see the same pattern as above:

(Hey look, I can make random charts too!) There are plenty of charts like this that we could make but none of them would tell the whole story. The whole story looks something more like this:

To be able to see the causal impact of socioeconomic status on SAT score, we would need to control for all of those factors that affect SAT scores that are correlated with SES- namely parents’ IQ, parents’ level of education and parents’ focus on education. Otherwise you will mistakenly attribute differences in SAT scores to money in and of itself as opposed to those qualities that got the parents the money in the first place, among other things. That said, if you’re going to do some wishful thinking on omitted variables, why not ask to control for the student’s IQ directly in addition to using the parents’ IQs?

As for Matt’s critique, the presumption that IQ and SAT scores are related is a fair one. See here for some evidence. (Granted, the article is from 2004, and the test has been redesigned in recent years, but I doubt that the relationship would have disappeared entirely with the redesign.) I’m giving this part of the battle an “Conor J. Clarke:1; Mankiw:0″. (You will note that he even added his own analysis as to this point here as I was writing.) As for Brad’s critique, I’m not sure that I buy all of his math (nor do a lot of the commenters, so I feel like I am at least in good company), but the thought process seems qualitatively valid. I think the main important points are to a. not draw conclusions that the data doesn’t support, and b. think carefully about how one would go about answering the question of interest and then try to get as close to that as logistically possible.

Ha. Apparently over the weekend Tyler Cowen felt the need to weigh in on the issue, and then Greg gives a (unintentionally, I am guessing) humorous response to the criticism.

P.S. I find it funny that follow up NYT posts now explicitly say things like “As always, though, correlation doesn’t necessarily mean causation.” At least we now know that they’re paying attention…

Tags: Fun With Data

22 responses so far ↓

  • 1 Tony // Aug 31, 2009 at 5:23 pm

    Nice… Thanks for writing on this one.

    I hadn’t read the DeLong article, but I read Krugman’s (yes, he weighed in too). It seems that everyone wanted to weigh in on this one. And, of course it gave Krugman an excuse to say, “It’s comforting to think that we live in a meritocracy. But we don’t.”

    But, I have to admit… when I read Mankiw’s original post, my first thought was, “Hey, maybe high-IQ parents are better at coaching kids to do well on the SAT.”

    That is, even the low-IQ adopted kids would have a ready-made tutor, and therefore, I’d expect them to do better (i.e., I’d expect that the graph would be flatter, but not flat).

    In other words, Mankiw is right that there is an omitted variable, but other than kid-IQ, there may be omitted variables (as your nice hand-drawn flowchart shows).

  • 2 Ben // Aug 31, 2009 at 5:26 pm

    Actually I would be interested to see the correlation between self-reported family income and family income, but I believe the correlation is between self-reported income and SAT scores :)

  • 3 Christa Watson // Aug 31, 2009 at 5:38 pm

    Agreed! Typo- fix quickly before tens of tens of people comment about income on income instead of what an interesting article this is! :)

  • 4 PeterM // Aug 31, 2009 at 6:07 pm

    I want to throw in some odd facts. First, look at the reportage on how well Asian Americans did on the SAT. http://www.insidehighered.com/news/2009/08/26/sat A 587 math average for Asian-Americans? Yowza. I don’t know if there is a separate income/score correlation for Asian Americans but I suspect it is buried in the data.

  • 5 econgirl // Aug 31, 2009 at 6:11 pm

    @Christa: Wow, that was an impressively large oversight. Gah. :)

  • 6 econgirl // Aug 31, 2009 at 6:21 pm

    @PeterM: check this out:

    http://www.census.gov/compendia/statab/tables/09s0669.pdf

    This shows a median income for Asian households of $64,238 compared to an overall average of $48,201 (in 2006). So the chicken and egg problem remains…though I suppose in a dynamic sense it would be very frustrating/nonsensical to see Asian-Americans having consistently higher SAT scores and not higher incomes…

  • 7 Dan L // Aug 31, 2009 at 10:56 pm

    Why do people always say it’s disingenuous to allow simple facts to speak for themselves? If a reader needs to be reminded that “correlation does not imply causation,” then I dare say that the reader is ill-equipped to understand the issue, with or without a reminder. (This reminds me of that stupid xkcd million vs billion thing. I for one cannot decide whether 500 million or half a billion “sounds” bigger. Yes, I am repeating myself.)

    In particular, I think it’s silly for you to criticize the Economix blog. (To be fair, Mankiw’s post is only criticizing by insinuation.) Not only did that post report facts in a technically accurate manner (using the word “correlation” explicitly), but it does not even insinuate that the graph describes a causal relationship.

    Regarding IQ vs SAT, whatever you believe about nature vs nurture, SES *should* have a stronger causal effect on the SAT than on IQ, since IQ is *designed* to test “innate” intelligence. (The SAT much less so.)

    Also, you said, “To be able to see the causal impact of socioeconomic status on SAT score, we would need to control for all of those factors that affect SAT scores that are correlated with SES- namely parents’ IQ, parents’ level of education and parents’ focus on education.” But that’s just ridiculous. When most people talk about the causal relationship between SES and SAT scores (or similar issues), “SES” is really just a stand-in for all of the *environmental* factors that go along with SES. To take things to the extreme, if you control for enough factors (including, for example, how the parents choose to spend their money), you are guaranteed to come up with a zero level of causal effect of SES on SAT.

    But I’m glad that we at least see eye to eye on the “my favorite textbook” shtick.

  • 8 Jarret // Aug 31, 2009 at 11:19 pm

    Tangentially, Stanley Kaplan died last week.
    http://www.nytimes.com/2009/08/25/education/25kaplan.html

  • 9 econgirl // Sep 1, 2009 at 2:05 am

    @ Dan L: When I get around to writing a textbook, it will totally be your favorite economics textbook. =P

  • 10 patrick // Sep 1, 2009 at 4:15 am

    glad to see that you’ve taken a very non-partial / even-handed approach to this issue. “correlation is not causation” is a decent idea to keep in mind, but there’s always worth in investigating a bit further, eh?

  • 11 econgirl // Sep 1, 2009 at 9:56 am

    from Mom:

    “Based on your SAT scores, we should be millionaires! :-)

    If only the causality worked in that direction…(and yes, she gets it…)

  • 12 g4m3th30ry // Sep 2, 2009 at 1:36 pm

    Yeah, in what appears to be an attempt to simplify things, just ends with using only one variable in a multi-variable question.

    Though it’s not surprising as the media is prone to this quite a bit :)

    Michael S. Langston

  • 13 Carl Peter Klapper // Sep 2, 2009 at 5:08 pm

    Why does nobody do a study of the correlation between speed in filling in little ovals with a number 2 pencil and SAT score? I know that, in my case, I was able to add a good 200 points to my total SAT score primarily through fine motor drills. The results of such a study would not only show a causal relationship, but also be practical.

    BTW, do economists have correlative relationships or are they only for statisticians?

  • 14 Ben Johnson // Sep 6, 2009 at 9:16 pm

    Greg Mankiw’s follow-up on his posting.

    http://gregmankiw.blogspot.com/2009/08/test-scores-and-income.html

  • 15 SAT prep for all. Why bother? « Superplexa! // Dec 23, 2009 at 7:45 pm

    [...] spend a great deal of time insisting that the test is in no way biased towards high income kids. No one is sure exactly why this is the case–Smarter parents? More test prep? Better schools? Higher expectations? A test [...]

  • 16 Tiger Mom Observations | big yipper designs // Jan 13, 2011 at 2:55 am

    [...] academics and talent is a high price. For example, she makes no claims about the connection between high socioeconomic status and SAT scores. While we like to believe that the AA community is poor and every one comes out working hard [...]

  • 17 Mark xfc // Apr 14, 2013 at 2:58 am

    |

  • 18 Siobhan // Jul 10, 2013 at 8:58 pm

    Awesome article. I’m using it even though “Economists Do It With Models” might not look so hot in my bibliography

  • 19 AtaraMac // Aug 4, 2013 at 11:54 pm

    Wow, structural equation modeling. I’m impressed.

  • 20 Daniel // Sep 17, 2013 at 5:03 pm

    Why does it still surprise us that kids from rich families have an advantage in life?

  • 21 B.A.Martin // Sep 21, 2013 at 11:12 am

    Why does it still surprise us that genetic characteristics are inheritable?

    There are many genetic characteristics (such as intelligence, reasoning ability, etc.) that not only lead to success in business, but also lead to higher scores on tests (such as the SAT “Reasoning” tests!)

  • 22 THE ASIAN WHITE HISPANIC BLACK TESTING PATTERN | Blacktistics // Jun 30, 2014 at 6:43 pm

    [...] such as income may explain some of these difference, but even black students from middle income families perform [...]

Leave a Comment