Research analytics

the best decisions are taken by combining robust statistics with sensitivity to the aim and nature of the research that is evaluated. Both quantitative and qualitative evidence are needed; each is objective in its own way

As a research leader you may want to use research metrics (along with qualitative and expert judgement) when analysing the impact of research within your school, department of faculty.

Comparing results between disciplines

When analysing the performance or impact of research within your school, department or faculty, you must be cautious when making comparisons between disciplines. For example, the frequency of publication is not the same in medicine than in social sciences and some disciplines are still under-represented in certain databases.

Comparing results within a discipline

It is also important to remember that within the same discipline such as Medicine there are sub areas that have faster publication rates than others and receive a greater impact. You will find that some factors besides performance may affect the value of a metric, for example, size, discipline, publication-type, database coverage, manipulation (unnecessary self-citations) and time.

Using total measures

If looking at total amounts, eg total number of citations, you should use them with caution. It may be more appropriate in some cases to use percentages, or size normalised measures. Normalisation is a process that takes into account citation pattern differences within disciplines, time periods and types of publication to enhance their comparability. View a primer on how (not) to normalise from PLOS Biology.

Metrics such as total number of scholarly outputs and citation counts should be used with care when benchmarking the productivity or impact of entities of different sizes, such as group of researchers, institutions, faculties, disciplines etc.

Metrics for tracking research impact

The following list includes possible metrics that may help you to track the impact of a department, school or faculty and some of the considerations you need to take into account. If you are thinking about using research metrics when recruiting new staff, please see Metrics for promotion panels.

Citations per publication

Citations per publication indicates the average citation impact of a group of publications. It is not advisable to use this metric to compare entities in different disciplines without accounting for these differences.

Collaboration type

Each publication is assigned to one of four mutually exclusive collaboration types, based on its affiliation information: international, national, institutional, or single authorship. This metric should be used with care when benchmarking the collaboration of entities in different disciplines. The typical collaborative behaviour of academics may differ between disciplines, such as mathematics and medicine, or humanities and molecular biology.

Collaboration impact

Collaboration impact calculates the average citations per publication for publications with different types of geographical collaboration, and indicates how beneficial these collaborations are with respect to citation impact.

Field-weighted citation impact

Field-weighted citation impact is the ratio of citations received relative to the expected world average for the subject field, publication type and publication year. It can apply to a research output or group of research outputs. This metric should not be used for entities with smaller numbers of publications. Because this metric calculates an average value, it is strongly influenced by outlying publications in small data sets, so it should be used with caution.

Outputs in Top Citation Percentiles

Outputs in Top Citation Percentiles indicates the extent to which an entity’s publications are present in the most-cited percentiles of a data universe. This metric should be used with care when comparing entities in different disciplines. Citation counts tend to be higher in disciplines such as immunology and microbiology, where academics tend to publish frequently and include long reference lists, than for example, in mathematics, where publishing one item every five years that refers to one or two other publications is common.