I’m a methodologist and I spend all day thinking about causal inference and epidemiologic methods. It would we weird if I didn’t sometimes catch methodological mistakes that other researchers–researchers who spends the bulk of their time asking and studying real epidemiologic questions–make in their research. That’s what I’m here for. To help improve the methodological quality of epidemiologic research.
Now, first I’m going to say I absolutely enjoy helping people with their research and I will never judge anyone who makes a mistake partially because I also make mistakes and partially because we have to be allowed to make mistakes because it’s one of the ways we learn. So, everything below is meant to be all in good fun.
But there are some errors that are so egregious that they should be entirely unacceptable by everyone at all times. Like, it should be entirely unacceptable that anyone can stand up in front of an audience, or worse, publish a paper, that contains this kind of error. Mostly because, as researchers, we’re spending other people’s money with the idea of improving society or human health. There are some errors we should not be allowed to make because they’re so fundamental to doing good science. So when I see methodological mistakes made, I think, is it acceptable that this error was made or is it unacceptable? Most of the time, they fall in the acceptable category and I try and address the issue. In the unacceptable case, as a joke, I’ve started thinking, “this person should go to epidemiology jail.”
(Now I’m realizing I may have gotten this idea of epidemiology jail from this episode of Parks and Recreation. You misinterpret a test? Right to jail. Adjust for a mediator? Jail. Fail to adjust for a confounder because your effect is no longer significant? Also jail.)
My first rule about epidemiology jail is that students are never eligible for it. Anytime I see a student make a mistake the blame almost always lies with the people responsible for their training. The supervisor, the instructors of the courses they took (possibly me), the program they’re enrolled in if it doesn’t provide enough instruction on the relevant material. Of course, we were all once students and it may be that we were failed at that level. But at some point, I guess, we have to be responsible for our own errors.
Here are some examples of things I think make someone eligible for epidemiology jail:
Misinterpretation of a statistical significance test: People use tests everyday and it’s integral to their research (even though I think it shouldn’t be). And yet so many people misinterpret significance tests almost every time they use them. Technically, I guess, this would be under the jurisdiction of statistics but, as far as I know, they don’t have a jail.
Adjusting for mediators when estimating the total effect: This usually happens because people think that adjusting for more things is always better.
Using mediation analysis without knowing the assumptions or even how to interpret the estimates: But really this applies to using any method without making any effort to understand how it works or how to interpret the results. I’m using mediation here because it happens a lot for that topic.
Excluding people based on post-baseline events: This happens, I think, because people are not able to ask clear, causal questions. If they could, I think they’d instantly realize that excluding people on future events is a bad idea.
Misdemeanour: Not having a clear causal question: Wait, how is this a misdemeanour? On one level it seems a no-brainer to send people without a clear question to epidemiology jail. I agree that not having a question damns people from the start but we can’t put everyone in jail. More often than not people have a vaguely worded research question and the answer to that question is a coefficient from a regression model, which has a very specific interpretation. But I’m willing to give some leeway on this one until people have better instruction.
Not having a relatively nuanced understanding of what a confounder is: Same reasoning as statistical testing. This is crucial to research! Conducting a causal study without properly understanding confounding is like letting me run a nuclear power plant because I read a popular science book on physics.
Saying you’re studying an association so you don’t have to be rigourous about your analysis: It’s the only reason I can think of to use association in a research question when the goal is clearly causal.
There’s more on this list, but this is all I can think of right now. Email me if you have something you think should be on this list or something you think shouldn’t be on this list.
