All FAQs about the Research Excellence Framework (REF)

Yes. However, if you request double-weighting for the output, you are effectively choosing to submit two outputs for the individual. If the double-weighting request is rejected and there is no reserve, the 'missing' output will be unclassified. Alternatively the HEI could choose to enter a single output without double-weighting.

If an individual is entitled to a reduction in outputs due to individual staff circumstances, they may be submitted with more than the minimum number of outputs required. Any submitted member of staff may have four outputs listed against them irrespective of their circumstances.

The journal article should still be submitted as output type 'D' = journal article. The ISSN field should state that this information is not available.

All research outputs should be submitted against the output type which an institution considers to best represent the research being submitted for assessment. The choice of output type should not be dictated by the metadata available for that output. If mandatory metadata does not exist for a particular output you should enter 'n/a' in the required submission system field. Wherever the DOI is not available for journal articles, this field should be left blank. 

When submitting web content as a research output, URLs alone should not be submitted and you should provide a portfolio with material sufficiently substantial to constitute evidence which allow sub-panel members to assess the research dimensions of the work. This may mean a combination of media are used which could include, for example, screen shots from the website, images and photographs, a written description of the structure of the section and how it interacts with other sections, descriptive or contextual information to evidence the research process. You will need to decide whether it is necessary to provide a copy of the whole website or a representative selection. You should also provide evidence that this was indeed first publically available within the publication period.

You may include URLs in these parts of the submission, only for the purpose of verifying or corroborating claims made in the submission. Panels will not follow URLs to access additional evidence or information to supplement the submission.  

 

The template format and page limits stated at Annex F of the 'guidance on submissions' must be adhered to. Changes should not be made to the margins, headers or footers provided in the template. You may use a template with or without borders (see ). The template also includes section headings for the impact case study; these may not be removed. However, the text stating the indicative word limits and number of references may be removed from your final submission if you wish.

(Updated September 2013)

Yes. From September 2013 templates were made available with and without borders. This follows institutional feedback that the templates with borders were difficult to use when inserting tables and diagrams. The REF team have revised guidance to allow the template to be completed without borders, if preferred. The section headings provided within the 'no borders' templates should not be changed. Changes should not be made to the margins, headers or footers provided in the templates.

(The templates are available within the REF Submission system and on the ).

The research underpinning an impact must have been carried out by staff 'while working in the submitting HEI'. This may include research undertaken by unpaid staff or staff who would have been considered as 'Category C' in a former RAE, on the basis that their research was clearly focussed in the submitting HEI. The individuals need not be working in the submitting HEI on the census date but must have been at the time they carried out the underpinning research. Research undertaken by research students is not considered as having been carried out by staff while working in the submitting HEI.

The submission system requires that a reserve output is associated with a specific output for which double-weighting has been requested. If the request is declined, the panel will assess the associated reserve output. However, where two double-weighting requests are made for an individual and the institution wishes to rank the two reserve outputs, it may use the additional information field in REF2 to indicate which of the two reserves should be assessed if only one of the double-weighting requests is accepted.

All additional information for REF2 will be published as part of the submission information. It is not possible for HEIs to redact elements of the additional information field. Therefore, if it is not appropriate for the name and contact details of the individual to be published, you should instead state the individuals' position and company. If audited, the institution will be expected to provide contact details for the individual.

If you are submitting two different chapters within a book as two separate outputs, you should provide two copies of the book.

No. Each output must be provided either as a PDF upload or deposited as a physical output; not as a mixture of the two. A physical output may include multiple items to represent one output. It is not possible to provide some of the output as a PDF upload and the send another part of the same output as a 'physical' output.

The environment template may describe the contribution of all staff that support research within the submitting unit, and need not refer only to those staff included in REF1a.

No, this income should not be double-counted. If the institution is returning an element of this funding as research income in REF4b it should remove the equivalent amount from the income-in-kind figures. 

If an individual in this circumstance is returned by both institutions, the REF team will rectify this through verification. If the member of staff meets the eligibility criteria to be returned by their first employer (institution A), then the individual will be deemed eligible for return by that institution and will be removed from institution B’s submission.                       

Institutions do not need to follow a naming convention for PDF files they upload to the submission system. When PDF files are uploaded, the REF system will automatically save files according to the REF naming convention.

Guidance on provides information on the arrangements regarding case studies that include confidential or sensitive data.

The REF submission system contains a field 'Conflicted panel members' which can be used to identify panel members or assessors that should not be able to view the case study, due to commercial conflicts. However, if a member or assessor becomes unable to participate we may replace them, and will update the panel membership on the REF website to reflect such changes. Since the panel membership may change over time, in addition to the guidance above, HEIs can use the 'Conflicted panel members' field to identify companies or organisations that should not have access to the case study. Any impact case study with data in the 'conflicted panel member' field will not be circulated to the whole sub-panel; it will be made available only to the specific individuals who are assessing the case study. When the sub-panel chairs allocate the case studies to individual members/assessors, they will not allocate the case studies to individuals or organisations identified in the 'conflicted panel members' field.

A total of five individuals to corroborate an impact case study can be provided. This may include any combination of statements already gathered by the institution and individuals who can be contacted by the REF team.

We do not envisage auditing more than five statements for any particular case study, which is why we have set this limit. The corroborating sources listed should focus on the key claims made within the case study. If a larger number of individuals could potentially provide such corroboration, then five should be selected that best represent this larger group.

Yes. The information provided in an impact case study may be presented in any form the institution considers to be appropriate. This may include tables and non-text content, so long as the guidance on maximum page limits and minimum font size, line spacing and margin widths are adhered to.

Unless specifically requested by the REF team, artefacts, devices, products and items of great rarity or value should not be deposited. For these types of outputs, a representation of the output and sufficient information about the research it embodies should be provided to enable the panel to assess it, without providing the item itself.

The REF team will reimburse HEIs for outputs that are lost during the assessment phase. Arrangements will be made directly with the institution concerned where this is required.

HESA will publish data about the volume of staff eligible to be submitted to the REF. HESA have released further information on the HESA website. (This FAQ was updated in July 2013).

Institutions should calculate the mean FTE of these staff using the number of hours or days worked from the start of their current employment contract up to the census date (31 October 2013), based on the standard hours or days of a full-time employee at that institution.

The following additional guidance was added in July 2013:

If the contract started prior to 2008-09 then it would suffice to calculate the mean FTE of HESA reporting years that fall wholly within the REF period (2008-09 to 2012-13) and the period up to the REF census date, 31 October 13.

A reduction of two outputs for junior clinical academic staff who have not gained their Certificate of Completion of Training takes account of significant constraints on research work, and is normally sufficient to also take account of additional circumstances that may have affected the individual's research work (including ECR status). Where a junior clinical academic also qualified as an ECR between August 2009 and July 2011 a reduction of two outputs would generally be considered sufficient. If you wish to return the individual with an output reduction of more than two, this should be returned via the complex route with clear explanation as to why a greater reduction would be justified.

If a junior clinical academic staff member first satisfied the definition of an ECR on or after 1 August 2011, they would be eligible for a reduction of 3 outputs as an ECR.

Institutions may use research groups for internal purposes in developing their submissions. In addition, the sub-panels within Main Panels A and B encourage institutions to structure their submissions by research groups. This is intended to assist the panels in their work, especially for larger or more diverse submissions. Quality profiles will not be provided at research group level; they will be provided for the submission as a whole. However, the sub-panel's qualitative feedback to the HEI may highlight individual research groups of particular note. 

Outputs of type A (Authored book) and type B (Edited book) cannot be uploaded to the submission system as a PDF and must be provided in physical form. For e-books, institutions may choose to submit the output on a CD, DVD or USB, or provide a printed hardcopy. For output type C (Chapter in book), a PDF of the book chapter (e-book or otherwise) may be uploaded to the REF submission system. In these instances the PDF should include a copy of the page(s) of the book that bear the title, publisher, editor and publication date.

The REF team will not source e-books submitted to the REF. They must be provided by the HEI as described above and it is not sufficient to provide only the URL for an e-book within REF2.

In the disciplines covered by Main Panel A, many (but not all) journals have well established conventions for the order in which the lead and/or corresponding author are listed - typically as first or last author. Corresponding authors are also often indicated by a typographic mark and/or a footnote. Where these conventions are well established, the sub-panel will accept them as identifying the lead or corresponding author. If in doubt about the journal’s conventions, then it would be safest to use the 'additional information' field to state what the author’s role was.

The numbering of research outputs has no significance and will play no role in the assessment of output quality. It is simply a means to identify the outputs and any associated information within the REF submission system.

Scopus has a Content Selection and Advisory Board which advises Elsevier on new journals which should be added to the database. Further details on the criteria which the Board uses to consider new material can be found on the Scopus website (https://www.info.sciverse.com/scopus/scopus-in-detail/facts). It is unlikely that the request to add new journals to the Scopus database will be resolved within the REF submission phase and therefore a citation count will not be returned. This does not preclude an institution from including the research output in their REF submission.

To ensure that this report is generated correctly and that all outputs of type D and E which need their journal articles supplied are listed, no outputs within that submission should be removed and re-added or have their DOI changed for a period of three weeks before the report is run. If some outputs have been altered in this way the three week count held against them will reset and they will appear on the report later than the outputs which have not been modified.

In this instance there are two reasons why the report 'Journal articles REF have been unable to retrieve' may not contain any data. If within the three week period, outputs have been removed and re-added (for instance by clearing a submission before importing updated data) the submission system will reset the three week count. To enable this report to generate correctly, outputs should not be removed (cleared) and re-added to the submission system three weeks before the report is run.

It is also possible that REF has managed to locate all of the outputs' journal articles within a submission, meaning that none need to be provided. If the submission's outputs and their DOIs have been present within the system for three weeks (i.e. not removed and re-added) this report will show correctly (which may be blank).

Both these sub-panels require data for students registered for a research doctoral degree and who are actively pursuing their research programme. Institutions should supply the total FTE of students active in the academic year by summing the HESA Student Instance FTE (STULOAD) field for students that have a HESA General qualification aim of course (COURSEAIM) of ‘D00: Doctorate degree that meets the criteria for a research-based higher degree’ and ‘D01: New Route PhD that meets the criteria for a research-based higher degree’ and that are in the HESA session population, for each of the academic years 2008-09 to 2012-13.

Where students have changed their mode to writing-up within the same academic year, institutions should include their writing-up activity for that academic year in the FTE figure reported for that academic year (there should be a single total FTE of students actively pursuing their research programme, to include students who have changed their mode to writing up during the year). Where a student has been writing-up for the whole of the academic year, institutions should exclude the student from the FTE figure reported for that academic year. 

Where you are including the writing-up activity, please use the HESA guidance to calculate this:

The FTE allocated to writing-up students should be 10% for both full and part-time students. The 10% applies where the student has been writing-up for a full year. Where a student has only been writing up for part of the year the FTE should be adjusted accordingly. So, where a student does 6 months full-time study and then 6 months writing up within an academic year, they would be recorded as 0.55 FTE.

If an HEI identifies an exceptional need to make significant changes to the content of its code of practice (COP) after it has been approved by the funding body, the HEI must provide a revised COP to the REF team, along with a covering letter from the head of institution. This letter should:

• Outline how the COP has been amended and the reason for the amendment.
• Confirm that the changes have not reduced the extent to which the COP adheres to the published guidance.
• Provide details of how the institution will communicate the changes to all relevant staff with ample time before final selection decisions are made.

The letter and revised COP should be uploaded to the REF extranet and an e-mail should be sent to info@ref.ac.uk to notify the REF team that this has been done. 

For minor changes to a COP, for example a change in an individual's membership of a committee, we will not require the HEI to follow the above procedure. Any changes made to selection procedures, the terms of reference of a committee, timetables and so on, will be considered significant changes to the content of a COP and will require the above procedure to be followed. Institutions should contact info@ref.ac.uk if they require any clarification on this point.

Final versions of COPs will be collected from all submitting institutions in early 2014, for publication along with the submissions in 2015.

Yes, all HEIs involved in a joint submission are eligible to return the same individual within their part of the joint submission, provided the individual fulfils the eligibility criteria with each HEI.

Each HEI returning this individual in their part of the submission should return them with the same research outputs. The information will be collated and presented to the sub-panel as a single submission. Each distinct output listed against the individual will be assessed only once and count once towards the outputs sub-profile.

The edited work could be submitted as a single output or alternatively, could be submitted separately if they are coherent as separate outputs. If returning them as separate outputs, you should consider whether they have significant material in common. (Further guidance on how outputs with significant material in common will be assessed is available in Part2 of the ''.)

Main Panel A have requested standard statements about the individual's contribution to a co-authored output, where required (see paragraphs 34–37 of Part2A of the ''). In addition, to assure themselves about an author's substantial contribution, the sub-panel may request further details of the specific contribution of the author, through an audit query. Main Panel A have indicated that such audit queries are especially likely for outputs with 15 or more authors (and where the individual is not the lead or corresponding author).

Where Scopus returns multiple matches for different formats of the same output, you should not confirm the match and should contact Scopus user support (via the REF submission system). Scopus user support will be able to create a single record for the output which consolidates citations for both formats. You will then be able to match this new record against the output within the submission system.

The HEI can claim the impact from the point at which research carried out by staff while working in the HEI made a distinct and material contribution to it. Any impact prior to that point can be described in the case study as context only (to help explain the background the impact), but cannot be claimed as part of the impact underpinned by the submitting HEI's research.

Time spent working within the higher education sector is not accepted as an 'absence' for the purposes of allowing a reduction in outputs. The only exception to this is where an individual had a change in contraced duties (for example, to a teaching-only contract) due to a complex circumstance or protected characteristic. Please see examples 26 and 27 of the ECU’s complex circumstances.

Staff who are employed by the HEI to work in an overseas department or campus of the submitting HEI are eligible to be returned as Categroy A staff if:

• they satisfy the general eligibility criteria at paragraph 78 of the '', and
• the HEI demonstrates that the primary focus of their research activity on the census date is directly connected to the submitting unit based in the UK (see paragraph 79d of the ''). In the submission, the HEI should indicate that such staff are 'Non-UK based' and provide the 'Non-UK based explanatory text'.
  

Staff who work at another HEI outside of the UK who also have a fractional contract with a UK HEI, may be returned as Category A staff (with a fractional FTE) if they satisfy the general eligibility criteria at paragraph 78 of the ''. For these staff, as with any other member of eligible staff on a fractional contract, the HEI is not required to demonstrate their research connection with the submitting unit.

HEI staff in managerial positions, for example, might not hold a contract with a primary employment function of 'research' or 'teaching and research'. In such cases, in order to be eligible for the REF, the individual would be required to hold a separate fractional contract which does satisfy the eligibility criteria (at paragraph 78 of 'guidance on submissions'). They would not be entitled to a reduction in the number of outputs on the grounds that they have a fractional contract for research. Any output reduction should be made in accordance with the guidance provided in paragraphs 64-91 of the .

Yes. As stated at paragraph 79f of the '', an individual may be returned as Category A by more than one HEI if they have a contract with and receive a salary from more than one HEI. The same research outputs may, but need not be, listed in each submission.

Where an individual works or worked part-time for a number of HEIs, then each of the HEIs submitting the individual will need to calculate if they are entitled to a reduction in outputs. A reduction in outputs may be made for the amount of time (calculated in months) in which the individual was not working in the HE sector, over the whole of the REF period. A reduction may not be made on the basis that the individual worked part-time at other HEIs. 

Each HEI returning the individual (with a fractional FTE) will need to submit the number of outputs required of that individual. The same outputs may, but need not be, listed in each HEI's submission.

If individual circumstances apply to the individual, the number of outputs may be reduced in all submissions in which the individual is returned. If research outputs are 'missing' without a qualifying individual circumstance, the 'missing' outputs will be recorded as 'unclassified'.

For example:

An individual has worked 0.5FTE at HEI A, and 0.3FTE at HEI B, for the whole of the REF period (1 January 2008, up to the census date 31 Octo0ber 2013). The individual did not work at any other HEI for the remaining 0.2FTE.

The number of outputs may be reduced as follows:

• 70 months working in HE at a total of 0.8FTE
• The equivalent number of months 'absent' from HE = 70 x 0.2FTE = 14 months 'absent'
• This entitles the individual to a reduction of 1 output.

HEI A may return the individual at 0.5FTE, in which case 3 outputs are required.

HEI B may return the individual at 0.3FTE, in which case 3 outputs are required.

HEI A and HEI B may, but need not, list the same outputs.

In order to be returned to the REF as Category A staff an individual must have a contract of employment with a primary employment function to undertake 'research only' or 'teaching and research'. They should be returned with an FTE relating to the entirety of that contract. The number of outputs may not be reduced on the basis that the individual also has other (non-academic) duties.

Yes, as stated at paragraph 86 of Part 1 of the ‘’, in UOAs 1-6 the number of outputs may be reduced by up to two, without penalty in the assessment, for Category C staff who are employed primarily as clinical, health or veterinary professionals (for example by the NHS), and whose research is primarily focused in the submitting unit.

For the purpose of identifying an individual as an early career researcher (ECR) in form REF1a: Staff details, the individual must first satisfy the definition of an ECR on or after 1 August 2009 (see paragraphs 85 – 87 of '').

For the purpose of calculating output reductions in form REF1b, you should consider the date that the individual started their career as an independent researcher, according to the criteria at paragraph 85a-b of the ''. This may be at any point within the REF period, and should be considered the date at which they first met the definition of an ECR.

If an individual first met the definition of an ECR before 1 August 2009 and this is the only circumstance for the individual, Table 1, Part 1 of the '' applies, and the individual is not entitled to a reduction in outputs.

If an individual met the definition of an ECR before 1 August 2009 and other circumstances also apply,  the number of months from 1 January 2008 up to the point the individual became an ECR can be combined with the other circumstances. In this case, Table 2, Part 1 of the '' applies to the total number of months 'absent'.  

Please also see ECU's worked complex circumstances, example 4.

The calculation method used in the example above should be applied when combining ECR status with other circumstances and supersedes the method used in example 12 of ECU’s worked complex circumstances.

This functionality was deliberately not included within the REF2014 submission system due to a number of problems which this functionality caused within the 2008 RAE system. It is possible to create a bulleted list in another application such as Word and then paste this in to the submission system. However this is not recommended as the formatting may be altered / lost when the data is extracted from the submission system for assessment. Instead you should make use of standard keyboard characters to create formatting. For example, asterisks(*) can be used to create a bulleted list, and the return key can be used to separate paragraphs.

The submission system will only send a new password to email addresses that are registered within the system. Please check that you have entered your email address in exactly the same way as the administrator who created your account. This situation most frequently occurs when institutions have email accounts that accept email from multiple domains. For example, even if an email account receives mail from both john.smith@university.com and john.smith@uni.com. You will only be able to log in to the submission system with the exact email address that your institution's administrator registered to your account during its creation. If your administrator has created your account with an incorrect email address, you should ask them to change it for you.

If you are unable to log in to the submission system using the password which has been sent to you via email, it is likely that your account is locked. This occurs when an incorrect password has been entered three or more times. In this instance you should contact your institution's administrator to unlock your account. Once your account has been unlocked you may then log in to the submission system using the last password which you requested from the system.

Alternatively your account may have expired. If your administrator reports that your account is not locked, ask them to check the expiry date which they set when your account was created.

If your account has become locked, you must contact one of your institution's administrators and ask them to unlock it for you. If you are an administrator you must contact another administrator to get them to unlock your account. The REF team strongly recommend that each institution has more than one administrator account present for this reason.

REF support and the REF team are unable to create user accounts. To obtain an account you should contact your system administrator within your institution. If you do not know who this is you should contact your institution's designated REF contact.

A list of all which the submission system contains is available.

The most common reason for this occurring is due to the sheets within the Excel import being named incorrectly. If the import file has been created without using our template (available under ) you must name the file's sheets to match those in our template. The default sheet names (sheet1, sheet2, sheet3) prevent the submission system from locating any data. Therefore, although the file is 'successfully' processed, no records are imported.

This occurs when all output records have been given the same output identifier within the import file. In this case, instead of creating a separate output record for each output, the submission system will create the first output record, and then subsequently overwrite that record repeatedly until the import process has completed. To avoid this, each output should be given a unique output identifier.

This is occurring because the output records within your import file do not contain output identifiers. When the import file is modified and re-imported, the submission system uses the output's identifier to detect that it is already present. When it detects an output is already present, it overwrites it rather than creating a new one. To resolve this, outputs should be given output identifiers.

Because of the validation errors found, the submission system has not added the import data to your submissions - the data has been rolled back instead. In order to add data that contains validation errors you should select 'Allow data to be imported even if there are validation errors' when setting up the import job.

You have entered an incorrect institution code. The 'institution' field must contain your institution's UKPRN (UK Provider Reference Number). This is provided by the UKRLP (UK Register of Learning Providers). Your institution's UKPRN can be obtained by visiting: https://www.ukrlp.co.uk and searching for your institution.

The submission system will always display this warning when a staff member is linked to fewer than four outputs. The warning is intended to prompt users to check that fewer than four outputs may be submitted without penalty. Provided that the REF1b section has been filled out correctly, this warning can be ignored.

It is only possible to convert a regular submission to a joint submission if all common forms (RG, REF3a, REF3b and REF5) are empty. After removing any data present within these forms you should be able to accept the invitation. This will convert the regular submission into a joint submission

There are some ISSNs that do not conform to the standard. Due to this the submission system will allow you to submit with an ISSN that appears to be incorrect. Providing you are certain that the ISSN entered is correct, you can ignore this warning.

Both the bulk citation retrieval and bulk CrossRef service rely on external systems provided by other companies (Elsevier and CrossRef) to function. During busy times these external systems may have large queues to process, consisting of requests from both the submission system and other systems. Once your request reaches the top of the queue, it will be processed. Should you feel for any reason that your request has 'stalled' and has not been processed after a reasonable amount of time, please contact REF user support. 

No. When a PDF is uploaded for an output record, the submission system links it to that output record's output identifier. This means that even if an entire submission is cleared and then re-imported, the uploaded PDFs will remain present within each output record. Note that the output records must be re-imported with the same output identifiers. Also, if a submission is deleted rather than cleared, all output PDFs in the submission will also be deleted. Note that REF3a, REF3b and REF5 PDFs will be cleared when a submission is cleared.

The list of research outputs included in the published REF submissions will indicate the research outputs that the HEI requested be double weighted. The list will also indicate the research outputs that HEIs marked as 'reserve' outputs. No indication will be given of which requests for double weighting were accepted by the sub-panels, or which reserve outputs were assessed. The published submissions reflect the submissions made by HEIs, rather than the decisions made by sub-panels.

There are worked examples of complex circumstances published on the ECU web-site. These worked examples indicate the kind of information that should be provided and how it should be presented.

As far as practicable, an estimate should be provided of the equivalent number of months absent from work. Sufficient information should be provided to show how this was calculated, by stating the nature of the circumstance(s), the period(s) of time affected and the equivalent number of months absent for each period affected.

Any further constraints on the individual’s research work in addition to the equivalent months absent should also be explained (for example, constraints on research while at work). Sufficient detail should be provided to understand the overall effect on the individual's capacity to carry out research, by outlining the nature of the constraint(s), the proportion of their research capacity that was reduced, and the length of time affected.

For example:

• An individual had time off work due to ill health and no further constraints after returning to work. REF1b need only state what the illness was, when the individual was off work, and the equivalent number of months absent.
• An individual had time off work due to ill health, and after returning to work had limited capacity to pursue their research due to ongoing effects of the illness. REF1b should include the information as above, and in addition state what the ongoing effect of the illness was on their capacity to carry out research, and provide an estimate of the proportional reduction in their research capacity and the length of time to which this applied. 

REF1b should also show how the reduction in outputs has been calculated, according to Table 2 of the 'panel criteria' in relation to estimated months absent from work, with further constraints taken into account as appropriate.  

All information should be anonymous so that EDAP can review complex circumstances anonymously. Where the circumstance has involved other individuals (for example, where the member of staff cared for a relative), those individuals should not be named and it is not necessary to specify their exact relationship to the member of staff.

HEIs need to assure themselves that the circumstances cited in their REF submissions took place. In the event that the REF team audits REF1b, the HEI will need to state how it assured itself that the cited circumstance(s) took place, and what evidence it had consulted. Given the potential sensitivity involved in keeping records about such information, the timescale for these audits will provide HEIs with an opportunity to check the evidence before responding to the REF team, if a record of this information is not already held. Further details about the timetable for audit will be published in 2013.

'Substantially' means that the majority or a substantial portion of the maternity leave was taken in this period. For example, an individual who went on maternity leave in June 2007 and returned in January 2008, or an individual who goes on maternity leave in October 2013 due to return during 2014, would not be considered to have taken maternity leave substantially within the REF period. Given that periods of such leave vary, a more precise definition cannot be given, and institutions will need to exercise their judgement.

The following additional guidance was added in October 2012:

In reaching these judgements, institutions should also consider how to treat staff whose maternity leave was taken partially within the REF period fairly, in comparison to staff whose maternity leave was taken entirely within the REF period. For the latter group of staff, there is no minimum qualifying length for maternity leave, and each period of such leave qualifies for the reduction of one output. Therefore, we consider it would be fair to allow the reduction of an output for a period of maternity leave that started prior to 1 January 2008 and continued for any length of time beyond that date, where the member of staff had been submitted to the 2008 RAE without any reduction in output due to that maternity leave. 

Online first journal articles should be returned as output type 'D - Journal article'. The REF team recognises that some of the meta-data may not be available for an online first publication, but sufficient information must be provided to enable the REF team to identify the output.

There may be some fields for which data is not available, that are mandatory for type D outputs (e.g. first page, volume number). These fields should be completed with ‘n/a’ to allow the submission to be made. You will not be able to submit without completing these mandatory fields.

The individual should be returned with the FTE of the contract that makes them eligible for submission to the REF; not the FTE specifically related to their research duties within that contract.

The published submissions will include a single list of all the outputs, and a separate list of all the staff, in each submission. The list of outputs will include standard bibliographic data (including the author name) for each output, but will not be listed by author name.

The HESA guidance has been simplified to the following:

Individuals whose primary employment function is to undertake either 'research only' or 'teaching and research' are staff returned to the HESA Staff Collection with an academic employment function of either 'Academic contract that is research only' or 'Academic contract that is both teaching and research' (identified as codes'‘2' or '3' in the ACEMPFUN field).

This guidance should be read in conjunction with staff eligibility guidance at paragraphs 77 to 81 of ''.

In paragraph 78 of the ‘’, Category A staff are defined as ‘academic staff with a contract of employment of 0.2 FTE or greater and on the payroll of the submitting HEI on the census date (31 October 2013), and whose primary employment function is to undertake either ‘research only’ or ‘teaching and research’.’

Individuals who do not meet these criteria are not eligible to be returned to the REF as Category A staff. Therefore, staff employed on full-time managerial contracts are not eligible for the REF. Staff who hold academic contracts of 0.2 FTE or greater with a primary employment function to undertake either ‘research only’ or ‘teaching and research’ are eligible for return to the REF, regardless of whether they also undertake managerial duties.

Any member of staff employed by the HEI, including heads of HEIs, is not eligible for return to the REF as Category C staff.

For clearly defined circumstances, the explanation field of REF1b should be used to state what each of the circumstances was and when it occurred, to show how the reduction in outputs was calculated, and to provide enough information about the circumstances to enable the sub-panels to confirm that the criteria for output reductions have been met. The following specific details should be provided for each type of circumstance:

• For early career researchers, provide brief details of the individual’s research career history, specifically identifying the point at which they first met the REF definition of an early career researcher. It is important to state why the individual did not, prior to that point, meet the definition of an early career researcher. This could be either by stating the nature of their contract (which did not meet the definition at paragraph 85a of ‘guidance on submissions'), or by outlining how their work did not include independent research.
• For part-time working, provide the start and end dates and the FTE for each period in which the individual worked part-time, and state the equivalent ‘months absent’ for each period. These must be ‘absences’ in which the individual was neither working in the higher education sector, nor undertaking academic research at some other organisation (see this ).
• For secondments and career breaks, provide the start and end dates, and state the equivalent ‘months absent’ for each period. Provide sufficient information about the career break or secondment (for example, the name of the organisation seconded to and the job role) to confirm that the individual was neither working in the higher education sector nor undertaking academic research.
• For maternity, paternity or adoption leave, state the start and end dates of each period of such leave.

Where multiple circumstances have been combined to calculate a reduction in outputs, the explanation should clearly show that where circumstances have overlapped, only one circumstance has been taken into account for that period.

No. Information provided in REF1b is used only to determine reductions in the number of outputs, and will not be taken account of for any other purpose.

EDAP’s role is to make recommendations on complex circumstances to the chairs of the main panels, who are responsible for the decisions. Details of these decisions or any consequent reduction in outputs will not be provided to the submitting HEI, as no scores for individual outputs – missing or otherwise – are identified in the quality profiles award to submitted units, and HEIs do not receive feedback on the individual parts of submissions. In order for HEIs to understand how EDAP will form its recommendations, the Equality Challenge Unit (ECU) has published a series of worked examples of complex staff circumstances. These examples include EDAP’s rationale: https://www.ecu.ac.uk/documents/ref-materials/complex-circumstances-examples.

No, HEIs may not submit a reserve output for individuals submitted with a reduction in outputs due to individual circumstances. Allowing a reserve output would diminish the case for an individual’s circumstances significantly constraining their ability to produce four outputs or to work productively throughout the assessment period. Additionally, allowing a reserve output would potentially result in unequal treatment of individuals who do not have an additional output compared to those who do. 

The reduction for part-time working applies only to time ‘absent’ in which the individual was neither working in the higher education sector, nor undertaking academic research at some other organisation. This is consistent with the allowances for secondments and career breaks.

Time spent working part-time in higher education, whether on an academic or non-academic contract, and whether in a UK or overseas HEI, does not qualify as an ‘absence’ when calculating reductions due to part-time working.

Yes, for the purposes of the REF an individual is deemed to have started their research career from the point at which they held a contract of employment of 0.2 FTE or greater, which included a primary employment function of undertaking ‘research’ or ‘teaching and research’, with any HE or other organisation, whether in the UK or overseas, and they undertook independent research, leading or acting as principal investigator or equivalent on a research grant or significant piece of research work. Individuals qualify as early career researchers if both of these criteria were satisfied for the first time on or after 1 August 2009, even if one of these criteria had been satisfied prior to 1 August 2009.

If submitting a product (output type = ‘P’) the date that the product was produced would be considered the date it became publicly available in the form in which it is submitted (see the ‘Output information requirements’ spreadsheet, available for download on the '' page ).

Separately published papers are discrete outputs and cannot be grouped together as a single output. Substantial dictionary or encyclopaedia entries and groups of short items including groups of entries may be submitted as a single output (see paragraph 51, Part2D of the ‘’).

The following additional guidance was added in November 2013:

Where research has been split by a publisher and would only be considered a single coherent work when viewed together, these should be returned as one research output. 

A portfolio is not an output type in itself but is the format in which the evidence of certain outputs can be submitted (see paragraph 59 of Part2C and paragraph 71d, Part2D of the ‘’.) For example, an output that is a performance or exhibition may be submitted in the form of a portfolio of evidence. Where submitting a portfolio, the most applicable output type should be used, and the ‘media’ field should be used to state that a portfolio will be submitted (see the ‘Output information requirements’ spreadsheet, available for download on the '' page).

Where a research output is submitted multiple times to the same UOA, it will normally be allocated to the same individuals on the sub-panel and reviewed only once (although the additional information may differ). Please note, an output may only be submitted a maximum of 2 times within one submission.

If a research output is submitted to multiple UOAs, it will be assessed by each of the sub-panels according to their respective criteria, including in terms of its significance to the disciplines concerned. The output will not, therefore, necessarily receive the same quality score.

If the sub-panel considers that the individual may not have made a substantial research contribution to the output and there is not sufficient evidence in the submission to make an informed judgement about this, the sub-panel will raise an audit query. The HEI will then be asked to explain the individual’s research contribution, and provide any appropriate evidence. The sub-panel will consider all the information provided before deciding. (Note that in situations where this information has already been requested as part of the submission, for example for sub-panel 9 where there are more than 10 co-authors, the sub-panel will normally decide on the basis of the submitted information without any further audit query.) If the sub-panel does not accept that the individual made a substantial research contribution, the output will be graded as ‘unclassified’ (see paragraph 126e, Part 1 of the ‘’.)

If the corroborating evidence is a pre-existing document not available in English, the HEI should reference the document and state what language it is in. The REF team will use the expertise of specialist advisers with the relevant language skills, if corroboration through these sources is required.

Any corroborating statements provided by users and held on file by the HEI will need to be provided to the REF team in English if requested through an audit. Where necessary the HEI will need to arrange for their translation.

Corroborating contacts should be given only for people who the REF team can communicate with in English.

A case study should include references to up to six research outputs that represent the body of research or a research project that was carried out at the submitting institution. These should be key outputs that underpinned the impact, and that best demonstrate the quality of the body of work or project. The sub-panels will not expect each individual output to meet the quality threshold, but will wish to be satisfied that the listed work was predominantly of at least two-star quality.

Where more than one submitting unit made a distinct and material research contribution to an impact, each of those submitting units may submit a case study of the impact. However, the case studies should not be identical, because each submitting unit will need to show that its research made a distinct and material contribution to the impact. This applies whether an HEI wishes to submit the same impact in different submissions, or different HEIs wish to submit the same impact.

Yes, we recognise that individual researchers may undertake research across multiple disciplines over time and that UOA boundaries are not rigid. Provided the underpinning research is within the scope of the UOA in which it is submitted, a case study may be submitted in a different UOA from the individual.

The only qualifications that may be included in REF4a are those returned to the HESA record as ‘D00: Doctorate degree that meets the criteria for a research-based higher degree’ and ‘D01: New Route PhD that meets the criteria for a research-based higher degree’. Data returned to the HESA record as E00: ‘Doctorate degree not obtained primarily through advanced supervised research written up as a thesis/dissertation’ are excluded from REF4a. PhD by publication, PhD by portfolio and professional doctorates recorded as E00 are not eligible for inclusion in REF4a.

The REF team will hold briefing events for additional assessors and users in advance of the assessment phase. In addition, all sub-panels (and main panels) will undertake calibration exercises of outputs and impacts towards the beginning of the assessment phase, fully involving the assessors as well as the panel members. Assessors will attend the meetings at which the material they have been involved in assessing is to be discussed.

The extranet login information that has been provided to the institution’s main REF contact can be used to create logins for multiple staff. It is up to the institution to decide who should have access to the REF extranet and to manage who submits the relevant documents through the extranet. Institutions’ HESA data has been provided by the REF team through the REF extranet and will include sufficient data to identify individual research students. This should be considered when disseminating extranet login information.

No. (The number of outputs may be reduced without penalty if the career break involved not working in the higher education sector and not undertaking academic research.)

The definition of independent research is at paragraph 85b of . In leading a significant piece of research work, the individual may, but need not, have acted as principle investigator or equivalent on a research grant.  An individual who meets the definition of a research assistant at paragraph 80 of 'would not be considered to be conducting independent research (unless, exceptionally, they are eligible for selection according to paragraph 81).

The definition of early career researchers for the REF is not based on a particular job role or job title as these may vary from institution to institution. The completion of a PhD is not always a useful indicator of the first point at which an individual undertook independent research.

Staff who use a different name on published research outputs should be returned to the REF with this name on REF1a. If staff data verification is required from the institution, we will require additional evidence to verify the individual’s identity.

An individual who is not on the payroll of the submitting institution on the census date is not eligible for inclusion as Category A staff. Such an individual may be eligible for inclusion as Category C: given that their salary is directly paid by a body external to the institution, this qualifies as ‘employed by an organisation other than an HEI’. (See paragraph 82 of .)

No, requests to make multiple submissions are not binding. An institution may decide to make only one submission in a UOA where they have been given approval for multiple submissions.

Information on clearly defined circumstances will be available to the panel secretariat and panel members.

Information on complex circumstances will be provided only to the Equality and Diversity Advisory Panel (EDAP), the REF team and main panel chairs. Sub-panels and the secretariat will be informed that there are complex circumstances and of the decision on the number of outputs to assess, but they will not have access to any information about the nature of the circumstances.

To further maintain confidentiality, EDAP will review complex circumstances in anonymous form, where they are submitted by institutions in this form. Institutions are advised to complete the REF1b form for complex circumstances, without stating the individual's name or other personal identifiers. 

Each single output in a submission contributes an equal proportion to the outputs sub-profile for that submission, not as a proportion of the maximum of 4 outputs per member of staff. See paragraph 126, Part 1 of the .

The calculations should be made according to the guidance at paragraphs 72-75, 80 and 82-84 of the . The final sentence of paragraph 82 should be applied as follows: For all clearly defined circumstances to which Table 2 applies, the total ‘months absent’ should be added together, and the reduction for these circumstances calculated by applying Table 2 to the combined total. Any further reductions should then be applied due to qualifying periods of maternity, adoption or additional paternity leave.

The REF submissions system will provide a calculator function to determine the maximum reduction in outputs for combinations of clearly defined circumstances, based on the data entered about these circumstances. Some worked examples are provided below.

See paragraph 43 of the . The final proviso should be interpreted to mean that such an output is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to the 2008 RAE by the same institution, or was not listed against the same member of staff (if they were in a different institution at the time).

Yes, 'online first' and other pre-published outputs that are in the public domain during the publication period (1 January 2008 – 31 December 2013) are eligible for submission.

Both a granted patent and a published patent application are eligible as research outputs. If you are submitting a granted patent you should include the patent number and date of the granted patent; if you are submitting a published patent application you should include the publication number and the date that this was published. Where a patent application was submitted to the 2008 RAE, the granted patent is not eligible for return to the REF.

No sub-panel will make any use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs. An underpinning principle of the REF is that all types of research and all forms of research outputs across all disciplines shall be assessed on a fair and equal basis.

A working paper is often written in the style of a journal article, is usually made freely available on line, may not have been peer reviewed, and has not yet been formally published (typically, in a journal). Working papers will not have a Digital Object Identifier. The use of working papers is more prevalent within certain research communities, for example ArXiv is often used in particular disciplines. The output must still meet the eligibility definitions described in .

No. In addition to the maximum of 6 research references, grant information details may also be provided, where appropriate.

Yes. PGR degrees awarded to all students supervised within the submitting unit should be included. The research income of the submitted unit as a whole should be included.

Yes. In such cases the funding will be allocated to each HEI according to the funding method of their respective funding body, using the number of Category A staff (FTE) submitted by each institution as the volume measure.

Yes. Following feedback from the sector, HEI's will be invited in early 2012 to make their requests for multiple submissions at any time from March 2012 until December 2012. We will consider all requests received at regular cut-off points during this period, and will provide a response within six weeks of each of those cut-off points. Each HEI may only make one set of requests for multiple submissions during this period.

An was published in February 2012.

When making submissions institutions will be able to identify individual staff, listed outputs or impact case studies that should not be published as part of the submission; and to identify other textual parts of submissions (REF3a, REF3b and REF5) that require some content to be redacted prior to publication. Following the submission deadline, institutions will have the opportunity to return redacted versions of these parts of submissions for publication. Further guidance will be provided in due course.

Yes. Any member of staff that is eligible for submission and meets the definition of an early career researcher (at paragraphs 85-86 of '') qualifies.

No.

Emeritus professors who continue in salaried employment contracted to carry out research and meet the definition at paragraph 78 of  are eligible to be returned as Category A staff.

Each institution will be able to determine which users of the submission system within the HEI have permission to read and/or edit each of the REF 'forms', including REF1b, for each UOA they are submitting in.

No.

Following publication of the final panel criteria and working methods, the Equality Challenge Unit (ECU) will publish worked examples of complex staff circumstances, including EDAP's recommendations on these hypothetical cases. This is intended to provide clear indications to HEIs about the operation of the arrangements for complex circumstances. Following publication of these examples, HEI will be able to suggest further hypothetical examples for publication by ECU, to add to the range of published examples.

The ‘’ was published in February 2012.

Examples of complex staff circumstances are available on the ECU web-site.

Yes.

Yes. It is recognised that research may have had impact prior to the publication of the output(s). The end of the publication period for the underpinning research (31 December 2013) extends beyond the end of the period for the impact (31 July 2013).

Submitted impacts may be at any stage of development or maturity so long as some effect, change or benefit meeting the REF definition of impact (at Annex C of '') took place during the assessment period (1 January 2008 to 31 July 2013). Regardless of the stage of maturity, case studies will be assessed in terms of the 'reach and significance' of the impacts that had occurred during this period.

Yes. The case study should state what research was carried out at the submitting institution, by whom, and when. As with all information provided in submissions, such information should be capable of verification by the institution.

Yes, provided that they carried out the underpinning research while working in the submitting HEI.

Yes. Impacts must be submitted in an appropriate REF UOA, regardless of where the underpinning research may have been submitted to a former RAE. See paragraph 162 of ''.

No. The panels may include general comments on this issue in their overview reports, if applicable.

A submitted impact must be underpinned by research that was carried out by staff while working in the submitting HEI. The HEI need not remain active in the area of research.

Note that in reaching their overall judgements about the impact element of submissions, sub-panels will consider the submitted unit's recent activity and strategy for enabling impact from its current research.

Where a submitting HEI is the result of a merger between former HEIs, the submitting HEI can submit impacts from the research undertaken by the former, now merged, HEIs.

Where a submitting HEI has taken over a research unit – whether from another HEI or from elsewhere – the submitting HEI can submit impacts from research that was undertaken by the absorbed unit before it became part of the submitting HEI, with prior agreement from the relevant UK funding body.

Prior agreement must be sought by providing details of the nature of the research unit and of when and how it became part of the submitting HEI, to info@ref.ac.uk, no later than 30 June 2013. The REF team will liaise with the relevant funding body and communicate the decision to the HEI.

In each case the funding bodies will take into consideration whether a distinct unit was absorbed by the submitting HEI in its entirety, and the extent to which there has been genuine structural change.

For clarity, these arrangements do not apply to impacts from research carried out by individuals before they joined the submitting HEI. See paragraph 160a of ''.

Data on research income and research doctoral degrees awarded for academic year (AY) 2008-09 to AY 2010-11 will be provided by May 2012; and data for AY 2008-09 to AY 2011-12 by April 2013. HESA will make AY 2012-2013 data on research doctoral degrees awarded available in a suitable form through the checkdoc facility, from August 2013. Further details, including arrangements for research income-in-kind, will be provided in due course.

This data has been provided through the REF extranet, for further details please contact the for your institution.

We are aligning data on research income and research degrees awarded as far as possible with existing HESA data, and the decision was taken not include 2007-08. This is to balance data quality considerations (2007-08 data was returned to HESA before its potential use in the REF was known), with panels' requirements for a sufficient time-series of such data for the purpose of informing their assessment of the research environment. Five years of such data (2008-09 to 2012-13) is considered sufficient for this purpose.

The REF team are looking into the feasibility of this, and will provide further information in due course.

This has been provided to HEIs in the data workbook available on the REF extranet.

The income sources for the 'Research grants and contracts' table (currently Table 5b) in the HESA Finance Statistics Return have not changed during the period 2008-09 to 2010-11 inclusive, and we do not expect them to change for the remainder of the REF period.

Research degrees awarded are returned to HESA by the HEI that registered the students. This HEI should include the research degrees awarded in their REF submission.

Institutions can disseminate their code of practice when finalised, in advance of submitting this to the REF team. If any changes to the code are subsequently required by the relevant funding body following examination by EDAP, the institution will need to notify staff of the changes.