Just a quick note on what a recent kerfuffle about the Bendavid et al SARS-CoV-2 antibody seroprevalence study in Santa Clara County says about epidemiology.
A few weeks ago Bendavid et al published a study looking at the SARS-CoV-2 antibody seroprevalence in 3330 participants sampled from Santa Clara County, California. Many people found issue with the study, among others, Andrew Gelman who had a nice write up on his blog.
There were various issues but the main issue was that they did not incorporate the uncertainty around the estimate of the specificity the test. This is a big deal because it takes a potentially informative study and widens the confidence intervals into a much much less informative study.
To their credit, Bendavid et al turned around and obtained more data on the specificity of the test giving them a much more precise estimate of the test specificity. Their study was, once again, informative.
But then wfithian steps in and demonstrates that the there is important heterogeneity in the data pooled to get the more precise estimate. When this heterogeneity is accounted for, the study returns to being largely uninformative.
Ok. I covered the story quickly to get to my point. On one level this is very nice example of self-correcting science where publich peer review played an important role. I’m an advocate for peer review in any form or forum and think that public peer review (e.g. on social media) can play an important role in science.
But what does this say about the original study? Incorporating uncertainty in a diagnostic test is relatively basic epidemiology. The failure to do so has two non-exclusive interpretations in terms of what it means about epidemiology as a field. And I don’t mean to pick on the authors of the Bendavid specifically, I come across articles that suffer from basic mistakes in epidemiology almost daily.
If we want to be charitable to the authors, we can say that they did everything in their power to publish an informative study to help with the COVID effort. They made a mistake in their analysis but everyone makes mistakes. We shouldn’t require that people make no mistakes but we should require that people admit their mistakes and remedy them as quickly as possible.
But this does imply that basic training in epidemiology is in a bad state when researchers from a well-respected institute such as Stanford fail at such a basic epidemiologic task. And this goes for other basic concepts such as adjusting for mediators (when not doing a mediation analysis) or interpret null-hypothesis signficance tests as support for the null hypothesis to name just two off the top of my head. We, as researchers, are always going to make mistakes but the role of basic training should be to avoid as many of these mistakes as possible.
The less charitable interpretation is that the authors knew they should include the uncertainty in the estimate of specificity but knew that it would mean their article was less-informative and more difficult to publish. Let me be clear that I don’t think that the authors’ sole purpose was to publish. I’m certain their goal was to help. But researchers get a lot of pressure to publish and we’d be naïve to think that epidemiologic methods never suffer for it.
In general, I worry about both these issues. Methods development is important but I think we could make much more important gains in the quality of epidemiologic research (in the short term at least) by improving the quality of epidemiologic training. Also, by being more severe when some of these basic mistakes are made (I’m purposely being vague about what ‘severe’ means here because I don’t have a more specific idea).
And the pressure to publish is undoubtedly responsible for methods short cuts that lead to important errors. I have seen this type of behaviour first hand.
I think both of these are resposible for the mistakes in the Bendavid et al paper and I hope we can find ways move forward beyond these types of errors in the future. But I’m not holding my breath.
