Okay, you guys, I know as well as anyone else how entertaining it is to see a reported research result that is hilariously eye-roll inducing. Case in point:
|Even the Thought of Earning Less than Their Wives Changes How Men Behave
by Dan Cassino
Masculinity is a fragile thing. Volumes of research in sociology and political science over the past 20 years have shown that men often react in surprisingly strong ways to what they see as threats to their masculine identities.
|Source: Harvard Business Review|
Like I said, HAHAHAHAHAHA silly males. Tempting, right? So you can understand why people are less likely to question results that seem intuitive or that they want to be true or whatever. Obviously this is a problem within the universe of research, so I try as much as I can to be picky about all research results and not just those that my brain doesn’t like. In this case, we can go to the (surprisingly informal, given the amount of attention it has received) study writeup and investigate the nuts and bolts of what is going on.
So the priming question is
Would you say that your spouse earns more than you, less than you, about the same, or is your spouse unemployed? [rotate response options, only asked to respondents who are married or living as married]
and the experimental manipulation is whether this question is asked before or after the questions about candidate preference. (The idea is that if the question is asked before, people might be thinking about it when they answer the later candidate question, but the same cannot be true if they are asked the question at the end, because that’s not how time works.) The main relevant result is stated as follows:
Among men who did not receive the gender question, Clinton bests Trump by 16 points—49 percent to 33 percent. Among men who were asked the gender question, Trump has more support than Clinton by a margin of 50 points to 42 points. All told, reminding men about gender issues, leads to a 24-point swing in the match up, from a 16-point advantage for Clinton to an eight-point deficit.
Interestingly, women who were asked the gender question before being asked about their vote choice became more likely to vote for Clinton. Without the gender prime, Clinton had a 21-point edge over Trump among New Jersey women—57 percent to 36 percent. Asking about gender reduced Trump’s support among women by 10 points, giving Clinton a 33-point margin—59 percent to 26 percent.
Let’s do a little thought exercise- what does the priming question make you focus on? Go ahead, think about it, I’ll wait…
The assumption of researchers is that it makes people think about gender roles (specifically, that it makes men defensive about their masculinity), but it’s also possible that it makes people think about issues of economic stability (or lack thereof) more generally. This is important because the researchers’ interpretation of the result is that the difference in responses is due to gender role insecurity.
In order to justify one particular explanation for a result (in this case, that it is poking the male ego that results in Trump support), it is necessary to rule out other potential explanations. In this case, an obvious alternative explanation is that making people think about money arouses some feeling of economic insecurity that in turn affects people’s political choices. Can we rule out this explanation given the available data?
On one hand, men and women do respond differently to the priming question- whereas it drives men to choose Trump, it drives women to choose Clinton. This could be because the question does in fact make people think about gender roles and gender roles, in turn, cause people to choose a candidate of similar gender. But..it could also be the case that the priming question makes everyone think about economic insecurity, but males tend to think that Trump is better for the economy and women tend to think that Clinton is better for the economy. Unfortunately, the study doesn’t have data to distinguish between the two possibilities.
Another way that researchers make a case that their explanation is the correct one is to think about what other features their data would have if their explanation is correct and then show that the data is in fact consistent with these implications. In this case, the gender role explanation seems to suggest that men who make less than their spouses would feel more of a threat than men who make more than their spouses. By this logic, I would expect the increase in Trump support as a result of the priming question to be greater for men who reported lower incomes than their spouses than for other men. Similarly, I would expect the increase in Trump support as a result of the priming question to be greater for men who report lower household income than for men who report higher income. The data exists to examine at least the first of these implications, so I was surprised to see that this analysis was not reported. (To put it less charitably, I wonder whether such a table is purposely being left out because it doesn’t fit with the proposed explanation.) It seems like the initial intent was to do this sort of cross-tab, since otherwise the survey could have just not asked the income question to the control group in the first place without diminishing the comparability of the treatment and control groups.
These are the sorts of issues that are brought up in the peer review process, so I guess what I’m really saying is that this gets us back to the issue of whether to report on research before it has gone through peer review and is accepted for academic publication. I think I might be changing my mind on this issue, though we could have the best of both worlds if those who write and edit the articles were able to critically examine the studies and provide caveats as appropriate. (This is where I point out that the author of the article here is the primary researcher on the study. 🙂 )