I’m not quite sure why this is in the news again now (there was a paper critique I can read ok), but it is so let’s take a look at it. Here’s the setup:
In 2005, she [Warren], along with David Himmelstein, Deborah Thorne and Steffie Woolhandler, published a paper in the journal Health Affairs documenting a memorable statistic: More than 40 percent of all bankruptcies in America were a result of medical problems, they wrote. In 2009, they updated that research with an even more startling number: Medical bills were responsible for more than 62 percent of all American bankruptcies.
One main problem here is the may that “medical problems” or “medical bills” is conveniently defined, but let’s put that aside for the moment. Logically, these statements mean that the study authors claim that medical bills cause bankruptcy. If you look at the methodology, however, you’ll see that the researchers get to this notion of causality by only looking at those people who declared bankruptcy, and…well, that’s definitely not how causality works.
To see why, let’s consider a logically equivalent situation that has the benefit of being far more obviously absurd. Namely, I could claim that having two legs causes bankruptcy because most people who declare bankruptcy have two legs. (I said most do not start with me) In this scenario, your brain’s knee-jerk reaction is probably “hey wait, but most people who don’t declare bankruptcy also have two legs, so this makes no sense.” And your brain is not wrong! For some reason though, people aren’t primed in the same way to conjecture that a large chuck of people have medical bills, not just those who declare bankruptcy. (I think the bill threshold to be counted in this context was $1,000, and really, who among us…)
We see this principle get abused and debunked in a legal sense all the time- and by that I obviously mean on Law and Order. IIRC, there’s an episode where a defense attorney argues that video games lead to murder because the murdery kid played video games and Jack McCoy is all “but what about all of the kids who play video games and don’t kill anyone?” I’m pretty sure Jack won the case, and we all learn an important lesson about holding ourselves to this standard of logic even when the bad conclusion is not intuitively absurd.
But wait, there’s more…my commentary on the matter led to this tweet from a friend of mine:
My initial reaction in a nutshell:
In this case, my lizard brain was temped to answer no, since, well duh. But, technically speaking yeah, you need a counterfactual group of people who don’t get shot (maybe not exactly the same size) to see how many of them die within an hour. I guess the thing here is that we generally know enough about human robustness to conclude that the chances of randomly kicking the bucket in the next hour without getting shot is negligibly small. (*knocks on wood*) But even then, I’m implicitly assuming that people who get shot aren’t systematically close to death anyway…which, I’m pretty sure this is just a different episode of Law and Order. (that link is a pretty good test of whether law school is for you tbh)
So what have we learned from this? Yes, in general, to establish that X causes Y, you need to look at both people who experience X and don’t experience X. (That said, I think you could rule out the possibility that X causes Y by only looking at people who experience X if they don’t also experience Y.) BUT, we can sometimes make do by only looking at people who experience X if we have a good read on what would happen if people didn’t experience X. Be careful though, since this only works if the people who experience X resemble the overall population, since otherwise selection bias bites us in the ass.
This shortcut, if you will, is more than just a thought exercise- in pharmaceutical trials, for example, it’s supposed to be the case that half of subjects get a placebo rather than the treatment being tested. This is ethically fine when the treatment is, say, Viagra, but what if it’s a cancer treatment? The ethics of denying people a potentially successful treatment for the sake of the pure scientific method is far less clear, especially when we’ve previously learned what happens when people don’t get the treatment. (spoiler: they die)
Note, however, that none of this “domain knowledge,” loosely speaking, allows us to conclude that X causes Y by only looking at people who experience Y. You could get closer by establishing that people who don’t experience Y generally don’t experience X (i.e. people who don’t declare bankruptcy don’t have medical bills), but even then you’ve only established correlation, and correlation does not imply causation.
Coming back to the original study and the ensuing debate…I can’t help but be annoyed with Elizabeth Warren here (even though in general I like her quite a bit), specifically because she’s stamping her academic credentials with information that is either poorly thought out or in bad faith. And for what purpose, even? Medical bills aren’t a problem only because they can result in bankruptcy, bankruptcy just happens to be an outcome that is enticingly easy to observe. In a way, they’re overcomplicating things, since a $10,000 medical bill imposes a cost of $10,000 on a household, full stop, and that matters in the amount of, wait for it, $10,000. Focusing on bankruptcy as the dealbreaker outcome even kind of gives the impression of “oh, you’re medical bill didn’t result in you lying in a ditch somewhere? You’re fine then!” which is, well, probably not fine…but I know someone who might think otherwise:
