Citation data

The information on this page was published whilst the exercise was being conducted; this material is provided for background information only and will not be updated.

The following sub-panels will make use of citation data to inform their assessment of the quality of research outputs:

Sub-panel 1: Clinical Medicine
Sub-panel 2: Public Health, Health Services and Primary Care
Sub-panel 3: Allied Health Professions, Dentistry, Nursing and Pharmacy
Sub-panel 4: Psychology, Psychiatry and Neuroscience
Sub-panel 5: Biological Sciences
Sub-panel 6: Agriculture, Veterinary, and Food Science
Sub panel 7: Earth Systems and Environmental Sciences
Sub-panel 8: Chemistry
Sub-panel 9: Physics
Sub-panel 11: Computer Science and Informatics
Sub-panel 18: Economics and Econometrics

Details of how citation data will be used in the assessment are set out in general terms in ‘guidance on submissions’ at paragraphs 131 to 136. Information on how these sub-panels will make use of citation data to inform their assessment is set out in the 'panel criteria' and the additional statement from Sub-panel 11.

The arrangements for collecting and reporting citation data, set out below, apply only to these sub-panels. We will not collect or process citation data for any other sub-panel.

Collection of citation data

Elsevier have been selected to provide citation data for the 2014 REF. Elsevier own the Scopus database which covers almost 18,000 titles from over 5,000 publishers.

Institutions will make their submissions via the REF submission system. The submission system will be interfaced with Elsevier’s systems to:

Match research outputs entered by HEIs onto the submissions system to the Scopus database, using Digital Object Identifiers (DOIs) and other metadata and;
Return the corresponding citation count for each matched output to institutions, via the submission system.

Institutions will be able to bulk-match outputs and bulk-retrieve citation counts, and export the data from the submission system throughout the period that the submission system is live. A reasonable use policy will be published alongside the REF submission system user guidance.

We anticipate the vast majority of matches to Scopus will be made automatically, and that most outputs in the relevant UOAs will be found in Scopus. (While testing this process in 2011, we matched a sample of outputs that were submitted to the 2008 Research Assessment Exercise in relevant UOAs. Over 94 per cent of this sample was matched to Scopus using automated algorithms.)

Through the REF submission system HEIs will be able to view details of the Scopus items that have been matched to their outputs, and query any instances considered to be incorrect. Where ‘ambiguous’ matches are made (more than one item on Scopus may be a potential match), the HEI will be able to select the correct record.

The citation count for each matched output will be displayed on the submission system. We expect to update citation counts on the submission system on a weekly basis.

Institutions will also be able to view matched outputs on Scopus.com by making use of the Scopus Preview function which will be accessed via a link from the submission system. This will provide additionalmetadata and information such as the abstract, references used, and an up-to-date citation count.

Institutions will be able to raise queries directly with Elsevier regarding the matching of outputs to Scopus, or where they have concerns about the number of citations reported. Elsevier will provide dedicated support for institutions throughout the submission phase.

No subscription to Scopus will be required by institutions to make use of any of these services.

The funding bodies do not sanction or recommend that HEIs rely on citation information to inform the selection of staff or outputs for inclusion in their submissions (see paragraph 136 of ‘guidance on submissions’.)

Providing citation data to REF panels

We will take a final ‘snapshot’ of citation counts for all matched outputs shortly after the submissions deadline. We will provide the citation counts to the relevant sub-panels for use in the assessment. The final citation counts provided to sub-panels will also be available to institutions, for information, via the submission system after the submission deadline.

We will not update the citation counts during the assessment phase of the REF, to avoid potential inconsistencies.

We will also provide contextual data to the sub-panels to assist them in interpreting citation counts, given that citations depend partly on the field of research and a publication’s age.