Economists Do It With Models

Warning: “graphic” content…

Bookmark and Share
It’s Way Less Fun to Pick Apart Amusing Findings, Voting Preferences Edition…

August 4th, 2016 · 5 Comments
Fun With Data · Gender · Politics

Okay, you guys, I know as well as anyone else how entertaining it is to see a reported research result that is hilariously eye-roll inducing. Case in point:

Even the Thought of Earning Less than Their Wives Changes How Men Behave
by Dan Cassino

Masculinity is a fragile thing. Volumes of research in sociology and political science over the past 20 years have shown that men often react in surprisingly strong ways to what they see as threats to their masculine identities.

Source: Harvard Business Review


Like I said, HAHAHAHAHAHA silly males. Tempting, right? So you can understand why people are less likely to question results that seem intuitive or that they want to be true or whatever. Obviously this is a problem within the universe of research, so I try as much as I can to be picky about all research results and not just those that my brain doesn’t like. In this case, we can go to the (surprisingly informal, given the amount of attention it has received) study writeup and investigate the nuts and bolts of what is going on.

So the priming question is

Would you say that your spouse earns more than you, less than you, about the same, or is your spouse unemployed? [rotate response options, only asked to respondents who are married or living as married]

and the experimental manipulation is whether this question is asked before or after the questions about candidate preference. (The idea is that if the question is asked before, people might be thinking about it when they answer the later candidate question, but the same cannot be true if they are asked the question at the end, because that’s not how time works.) The main relevant result is stated as follows:

Among men who did not receive the gender question, Clinton bests Trump by 16 points—49 percent to 33 percent. Among men who were asked the gender question, Trump has more support than Clinton by a margin of 50 points to 42 points. All told, reminding men about gender issues, leads to a 24-point swing in the match up, from a 16-point advantage for Clinton to an eight-point deficit.

Interestingly, women who were asked the gender question before being asked about their vote choice became more likely to vote for Clinton. Without the gender prime, Clinton had a 21-point edge over Trump among New Jersey women—57 percent to 36 percent. Asking about gender reduced Trump’s support among women by 10 points, giving Clinton a 33-point margin—59 percent to 26 percent.

Let’s do a little thought exercise- what does the priming question make you focus on? Go ahead, think about it, I’ll wait…

The assumption of researchers is that it makes people think about gender roles (specifically, that it makes men defensive about their masculinity), but it’s also possible that it makes people think about issues of economic stability (or lack thereof) more generally. This is important because the researchers’ interpretation of the result is that the difference in responses is due to gender role insecurity.

In order to justify one particular explanation for a result (in this case, that it is poking the male ego that results in Trump support), it is necessary to rule out other potential explanations. In this case, an obvious alternative explanation is that making people think about money arouses some feeling of economic insecurity that in turn affects people’s political choices. Can we rule out this explanation given the available data?

On one hand, men and women do respond differently to the priming question- whereas it drives men to choose Trump, it drives women to choose Clinton. This could be because the question does in fact make people think about gender roles and gender roles, in turn, cause people to choose a candidate of similar gender. But..it could also be the case that the priming question makes everyone think about economic insecurity, but males tend to think that Trump is better for the economy and women tend to think that Clinton is better for the economy. Unfortunately, the study doesn’t have data to distinguish between the two possibilities.

Another way that researchers make a case that their explanation is the correct one is to think about what other features their data would have if their explanation is correct and then show that the data is in fact consistent with these implications. In this case, the gender role explanation seems to suggest that men who make less than their spouses would feel more of a threat than men who make more than their spouses. By this logic, I would expect the increase in Trump support as a result of the priming question to be greater for men who reported lower incomes than their spouses than for other men. Similarly, I would expect the increase in Trump support as a result of the priming question to be greater for men who report lower household income than for men who report higher income. The data exists to examine at least the first of these implications, so I was surprised to see that this analysis was not reported. (To put it less charitably, I wonder whether such a table is purposely being left out because it doesn’t fit with the proposed explanation.) It seems like the initial intent was to do this sort of cross-tab, since otherwise the survey could have just not asked the income question to the control group in the first place without diminishing the comparability of the treatment and control groups.

These are the sorts of issues that are brought up in the peer review process, so I guess what I’m really saying is that this gets us back to the issue of whether to report on research before it has gone through peer review and is accepted for academic publication. I think I might be changing my mind on this issue, though we could have the best of both worlds if those who write and edit the articles were able to critically examine the studies and provide caveats as appropriate. (This is where I point out that the author of the article here is the primary researcher on the study. 🙂 )

Tags: Fun With Data · Gender · Politics

5 responses so far ↓

  • 1 AJ // Aug 5, 2016 at 12:06 pm

    I think you meant to say “peer review process” in the last paragraph..

  • 2 Dan Cassino // Aug 5, 2016 at 4:19 pm

    First off, I’ll be happy to put together some of the xtabs you’re interested in. I don’t think we can do the income xtab, as we don’t generally do income levels in the demographics (it makes people hang up the phone. A lot), but maybe there’s some others we can use to test your proposed effects.
    As for why we think the spousal income question is priming gender roles, rather than pure economic concerns, it’s because it’s been studied in the gender/sociology literature pretty widely. There’s lots of work on how spousal income disparities lead to really weird behaviors (I covered some of it in the HBR piece), and many of the effects that have been shown are at odds with a pure economic conception.
    As for why we didn’t put all of the xtabs in the release, it’s mostly due to small ns, once we chop up the data too many ways. The poll has a policy about releasing figures below a certain threshold (I’m of the opinion that a significant difference is a significant difference, whatever the n, but I don’t run things), so once there’s an experiment in the poll, it limits how detailed the xtabs put in the release are.
    For me, the finding that really makes me think that this is about gender is the lack of an effect of the prime on support for Sanders among men. If we’re going by a pure economic explanation, I don’t know how making men worry about the economy pushes them towards Trump, but only when he’s running against Clinton. If anything, Sanders has a rather more radical economic agenda, so I would expect a rather larger effect, rather than the no difference observed.
    As for whether it’s a good idea to put out findings before they’ve been peer-reviewed, my sense is that when I have my pollster hat on, it’s my responsibility to put out newsworthy findings while they’re still newsworthy. The conflict comes because I use the polls I work on to gather my data for my academic work, which gets published months or years after the fact.
    Anyway, I’m happy to talk with you about the study, if you’re interested. Let me know.

  • 3 econgirl // Aug 6, 2016 at 2:12 am

    @Dan:

    Thanks for the additional information! It’s helpful in part because your study doesn’t have a lit review section yet. I think my favorite (in an eyeroll at humanity sense at least) paper is the one that shows that women are more likely to quit their jobs when they make slightly more than their partner than when they make slightly less.

    I think I agree with you on most points, particularly the one about the small n- it’s always confused me that people pretty automatically complain about small samples when p-values already take this into account (both in the n parameter in the test statistic and in the shape of the appropriate distribution).

    I do still think that there’s something weird going on regarding people’s perceptions of Clinton’s ability to handle the economy, and there are people (who I don’t understand) that seem to think that both Sanders and Trump would be better on that dimension than Clinton. That’s the reason that I didn’t think about the Sanders question in the same way that you did.

    Does the small sample issue make it impossible to show the results for the two conditions separated by response to the gender income question (or perhaps more aggregated groups of responses)? If so, is it possible to report the spreads without showing the whole table? It seems like that could bolster your explanation a lot if the relationship follows the intuitive pattern. Also, I’m sooooo curious as to what that looks like and I’m selfish. 🙂

  • 4 Dan Cassino // Aug 8, 2016 at 1:46 pm

    So, I’ve run the xtabs you’re interested in – I’m not worried about the small n in this context because (a) my PR department isn’t involved, so I can report whatever I want, and (2) (unlike reporters) you know what “significant” is. I’ll send you the images of the tables via Twitter.
    Give it a look – any criticism is welcome, so I can make sure to address them in the peer-reviewed version.

  • 5 Chris // Aug 9, 2016 at 3:33 pm

    Dan, Jodie, I’m interested in the xtabs too. Can you link to the twitter images here?

Leave a Comment