Skip to main content

Measuring research impact

Limitations of bibliometrics

Quantitative measures provide just one part of the picture. They should only be used where it is appropriate to do so, and with an understanding that it is extremely difficult to compare across disciplines given each discipline’s individual publication and citation patterns.

The main limitations include:

  • Quality: high citation counts may not necessarily indicate quality – an article may be cited frequently because other authors are refuting its findings.
  • Discipline variation: some research fields cite papers more extensively than others. For example, in medicine and health there is a strong culture of citing and using other studies to validate findings.
  • Bias and discrepancies: people may be citing their own or colleagues’ outputs or citing studies from the journals in which they publish. However, a number of bibliometric tools do allow you to exclude self-citations.

It is also worth noting that early career researchers may have low citation scores. Experienced researchers may have built up long lists of outputs, some of which are published in high impact journals, and therefore may have more citations. Early career researchers can find it difficult to build up their publications and may choose to publish in a lesser-known journal that may not be indexed.

Altmetics are useful for early career researchers who may have a smaller body of work and a low citation score. Many researchers who are starting out raise the impact of their publications by using techniques such as blogging.