On Thursday, the NYT Economix Blog reported on the correlation between (self-reported) family income and SAT scores. The main thesis is summarized by the following graphic:
My first thought was “How on earth do these kids know how much their parents make?” since, when I was in high school, I really had very little clue how much dough my parents were pulling down. Actually, I still don’t really know. (Feel free to enlighten me, Mom.) From an economic perspective, my initial thought was “This is clearly a correlation rather than necessarily a causal relationship.” (For a primer on correlation vs. causation, click here.) I really don’t mean to be as snarky as this sounds (okay, maybe I do), but perhaps something that announces itself as an economics blog should be careful to be, well, either economically sound in its analysis or careful about pointing out the limitations of its interpretation. I am apparently not alone in this opinion, and I will outline for you the s**tstorm that ensued:
Step 1: Greg Mankiw’s blog gives the above graph the title “The Least Surprising Correlation of All Time”. (I would argue that this one showing the relationship between beers consumed and perceived employment outlook could be a contender for that title. But I digress.) He then subtly (and rightly, in my opinion) criticizes the Economix Blog for not specifically cautioning against inferring a causal relationship from this data. He goes on to detail the problem with the causal interpretation:
This graph is a good example of omitted variable bias, a statistical issue discussed in Chapter 2 of my favorite textbook. The key omitted variable here is parents’ IQ. Smart parents make more money and pass those good genes on to their offspring.
Suppose we were to graph average SAT scores by the number of bathrooms a student has in his or her family home. That curve would also likely slope upward. (After all, people with more money buy larger homes with more bathrooms.) But it would be a mistake to conclude that installing an extra toilet raises yours kids’ SAT scores.
It would be interesting to see the above graph reproduced for adopted children only. I bet that the curve would be a lot flatter.
I like his diving into the absurd bathroom analogy almost as much as I dislike him referring to his “favorite textbook.” (It’s his textbook, in case that wasn’t obvious. I feel like a dash more irony or self-mocking is needed to make statements like that work, but in a small way I commend the effort.) Now, he’s right in that it *should* be enlightening to see the graph reproduced for adopted children, since this would theoretically take parents’ intelligence as a genetic driver of childrens’ SAT scores out of the equation. But wait, there’s more…
Step 2: I read the following in my Twitter feed, via @mattyglesias (See here if you are not familiar): RT @conorjclarke: 15 seconds of googling 2 find problems with mankiw: high-income adoptees are 12 IQ points higher http://bit.ly/2RzLIp. Wow, people are quick to jump on the “Mankiw is wrong” bandwagon, though in checking my Google Reader I notice that Mankiw’s post went up at 5:24am. What in the hell is he doing up and writing economic rants that early? (The Twitter comment didn’t come until 10:52am, which I consider to be a much more reasonable hour.) Anyway, the link is (via Google Books) to a book entitled Intelligence and How to Get It: Why Schools and Cultures Count, and the relevant information is the following:
On average, the biological children of high-SES [socioeconomic status] parents had IQs that were 12 points higher than those of low-SES parents, regardless of whether they were raised by high-SES or low-SES parents…
The crucial finding is that children adopted by high-SES parents had IQs that averaged 12 points higher than those adopted by low-SES parents- and this was true whether the biological mothers of the children were of low or high SES.
The book goes on to give further evidence from a natural experiment to support the claim that environmental factors are significant determinants of IQ. So wait- high-SES parents can actually cause their kids to have higher IQ’s? If we believe that there is a positive relationship between IQ and SAT scores, this would imply that the relationship between SES and SAT scores would persist even if we looked exclusively at adopted children. Stay tuned…
Step 3: The s**tstorm continues with a post by Brad DeLong where he quotes what Mankiw wrote in his post and then starts commentary with:
But merely saying that correlation is not always causation and dropping the issue is, I think, profoundly unhelpful–and shows a… lack of work ethic as well.
For the record, that is about how civil academic economists generally are to each other. He follows up with a lot of words and math and then the following conclusion:
The rule of thumb, I think, is that half of the income-test score correlation is due to the correlation of your test scores with your parents’ IQ; and half of the income-test score correlation is [sic] coing purely from the advantages provided by that component of wealth uncorrelated with your parents’ (genetic and environmental!) IQ.
The curve is less steep, but there is definitely a “what” here to be thought about.
Sheesh. Okay, well, let me throw my brain into the ring here:
First, you can see the data from The College Board that everyone is using here. The chart data comes from the page labeled 4 in the report, which is actually page 8 of the pdf. You will notice that, in addition to the scores versus income category, there is a category for scores versus highest level of parental education. Now, one would expect that SES and education are fairly correlated, so it’s not surprising that we see the same pattern as above:
(Hey look, I can make random charts too!) There are plenty of charts like this that we could make but none of them would tell the whole story. The whole story looks something more like this:
To be able to see the causal impact of socioeconomic status on SAT score, we would need to control for all of those factors that affect SAT scores that are correlated with SES- namely parents’ IQ, parents’ level of education and parents’ focus on education. Otherwise you will mistakenly attribute differences in SAT scores to money in and of itself as opposed to those qualities that got the parents the money in the first place, among other things. That said, if you’re going to do some wishful thinking on omitted variables, why not ask to control for the student’s IQ directly in addition to using the parents’ IQs?
As for Matt’s critique, the presumption that IQ and SAT scores are related is a fair one. See here for some evidence. (Granted, the article is from 2004, and the test has been redesigned in recent years, but I doubt that the relationship would have disappeared entirely with the redesign.) I’m giving this part of the battle an “Conor J. Clarke:1; Mankiw:0″. (You will note that he even added his own analysis as to this point here as I was writing.) As for Brad’s critique, I’m not sure that I buy all of his math (nor do a lot of the commenters, so I feel like I am at least in good company), but the thought process seems qualitatively valid. I think the main important points are to a. not draw conclusions that the data doesn’t support, and b. think carefully about how one would go about answering the question of interest and then try to get as close to that as logistically possible.
Ha. Apparently over the weekend Tyler Cowen felt the need to weigh in on the issue, and then Greg gives a (unintentionally, I am guessing) humorous response to the criticism.
P.S. I find it funny that follow up NYT posts now explicitly say things like “As always, though, correlation doesn’t necessarily mean causation.” At least we now know that they’re paying attention…