A lot of people are upset about some research communities shunning quantitative measures of research output such as H-indexes or impact factors. Those who want to keep it argue that it’s a valid measure of a person’s contribution to research. Those who want to eliminate it say that it’s a poor measure of the impact of a person’s research and it’s a system that is easily gamed. I’m 100\% part of this latter group.
First, let’s get at what we’d really like to measure. What we’d like to know is how much a person, in the future, will contribute to improving human health (this may differ depending on type of grant or reason for evaluating a researcher). We, of course, can’t see the future so we use the past to predict the future. But let’s not forget that it’s future impact that we’re trying to predict.
There are many avenues to do this. You can publish articles with direct, clear clinical or public health impact. You can actively work with people from public health agencies. You can mentor students that will go onto to having incredible careers. You can teach important concepts clearly to hundreds or thousands of students leading to increased quality of all their research. To me at least, it seems clear that what we want to know is a combination of many different aspects of a researchers career. And notice that nowhere does the quantity of publications appear.
As scientists, we don’t feel comfortable in a situation where we have to make decisions that aren’t driven quantitatively. This is the whole reason people think that using p<0.05 is a good way to make decisions–they feel like obeying a simple rule avoids their own bias creeping in. But simple rules don’t work in complex settings. If you want to make a decision about whether a drug should be on the market, you need to consider the cost of production, efficacy, effectiveness, side-effects. There are no simple rules. This is why people love H-indexes and impact factors too. It’s an easy way to quickly judge someone’s research.
But H-indexes and impact factors only work under the assumption that more publications in more prestigious journals is better. There are many reasons this might not be true. What is required to become an author on a paper? I know there are guidelines but in practice, let me tell you, these are not followed. I have been offered to be co-author on papers where I am nowhere near the guidelines to be an author (which I regularly turn down) so I imagine this has happened to many others as well. I’m not saying everyone else would say yes to authorship on all these papers but surely different researchers will have different thresholds to which they’ll say no to being included on a paper. And I can’t count the number of times I’ve heard anecdotes about a highly published researcher who were unaware they were a co-author on a paper.
As for high impact factor journals, I know that higher impact factor doesn’t necessarily mean more public health or clinical impact and it DEFINITELY doesn’t mean higher quality research.
In the end, if people with higher H-indexes and who publish in higher impact journals truly do have more clinical and public health impact, what are they worrying about? Surely that’ll come through even if they’re only allowed to talk about 10 of their publications? If this policy of only letting a researcher talk about 10 publications makes people focus less on quantity and more on quality, I’m almost convinced that it will have a huge net impact on how much good research does in the world.
