Performance-based Research Funding Systems (PRFS) are one of the mechanisms through which countries try to increase the performance of their public sector research systems. The nature of these systems – based on peer reviews, metrics or a combination of both – varies considerably among countries. The MLE will provide a learning opportunity for countries willing to better understand the advantages and drawbacks of various options, improve ongoing PRFS and deepen the assessments of the impact of different systems.
A growing number of countries are using performance-based research funding systems (PRFS) as part of their research policy mix. This article brings together evidence and experience about when and how PRFS can fruitfully be used to support policy development based on a number of studies conducted under DG RTD’s Policy Support Facility (PSF), in particular the Mutual Learning Exercise (MLE) on performance-based research funding systems in universities, but also in the MLE on Open Science and in the Specific Support actions for Latvia, Slovenia, Georgia and Bulgaria.
This report summarises the main learnings drawn from the exercise and makes a number of policy recommendations about how and when to design and use PRFS for policymakers and PRFS designers. A longer report is published in parallel with this one, which presents the evidence and analysis underlying this one, together with the lessons and recommendations.
The case for Performance-based Research Funding Systems (PRFS) is building as more countries adopt them to increase the performance of their research system. To date, few comparative studies have been conducted bringing together practices and experiences with this relatively new policy instrument.
This report reviews wider international experience with Performance-Based Funding Systems (PRFS) as well as the specific experiences of the participating countries. It constitutes a ‘handbook’ for policy makers and PRFS designers, providing analyses of experience and discussing about policy role of PRFS, key parameters of PRFS design, strengths and weaknesses of different ‘models’ for PRFS design (discussions of peer review, indicators, societal impact of research…). It includes also drawing lessons and making policy recommendations about how and when to design and use PRFS for policymakers and PRFS designers.
This report sets the overall framework for the discussions in this MLE. It sets the introduction of a PRFS within the overall public funding system for research and investigates the role and importance of the policy objectives that govern the design of a PRFS and its two components, i.e. the assessment process and the funding formulae. The report also gives an overview of the key design parameters that define a PRFS and the design options available for policy-makers. For the evaluation component, these are the model to use for the assessment, the scope of research activity included, the indicators and assessment criteria, and the granularity and periodicity. These aspects are investigated further in the other thematic reports.
The particular focus of this thematic report on Performance-based Research Funding Systems (PRFS) is on the use of bibliometrics and similar data and methods in relation to such funding systems. The key questions of this report are how bibliometrics is used directly or indirectly in the funding system, the indicators and data sources used, and the degree to which bibliometric indicators or bibliometric information influence funding allocation.
The focus of this report is on the use of peer review in the context of Performance-based Research Funding Systems (PRFS). It aims to support discussion and mutual learning among the participating countries on different aspects, issues and questions related to peer review-based assessment systems. The scope of this paper is peer review systems that are not directly linked to evaluations aimed at funding distribution but nevertheless form an important part of the national assessment culture.
The focus of this paper is on the use of or demand for third stream metrics in the context of PRFS. The intention is to support discussion and mutual learning on different aspects, issues and questions related with third stream metrics and their use in evaluation systems. Currently, only a few countries in the world have experience with third stream metrics as a routine element in their evaluation systems. Nevertheless, the increasing importance that policy makers attribute to the relevance of research for societal problems has led to a rising interest and demand for knowledge and practical guidelines for the design of such metrics.
This document sets out the methodological approach and workplan for the PSF Mutual Learning Exercise (MLE) on Performance-based Research Funding Systems (PRFS). It gives a view on the scope, objectives, time schedule and work distribution.
A growing number of countries are using performance-based research funding systems (PRFS) as part of their research policy mix. The European Commission as well as other international entities such as the OECD and the UNESCO often promote PRFS as a solution for research governance issues. Their comparative novelty as policy instruments, however, means that often, the complexity in their design is underestimated, their value as policy instruments misunderstood, or the risks and negative (undesired) effects that they may have on the research system not sufficiently accounted for.
At the Scoping Workshop main decisions were adopted for the implementation of the MLE: scope, preliminary work plan and distribution of work.
At the kick off meeting, the main decisions were adopted for the implementation of the MLE (Modus Operandi) and a first discussion took place on key concepts, overview of models, achievements, policy drivers and challenges of Performance-based Research Funding Systems (PRFS).
This first working meeting on Performance-based Research Funding Systems (PRFS) focused on the links between policy objectives and design options for performance assessment and performance-based funding models. The broad characteristics of PRFS in the participating countries were discussed, as well as their potential effects – intended or unintended – on the behaviour of the research community.
How to use bibliometrics to assess performance and inform research funding? The first country visit of the MLE on Performance-based Research Funding Systems – to Rome, Italy – focused on the use of bibliometrics in PRFS and their potential effects. The host, ANVUR (the Italian National Agency for the Evaluation of University and Research Systems), provided insight into the characteristics of the bibliometrics used in the Italian PRFS.
The second country visit of the MLE on Performance-based Research Funding Systems (PRFS) – hosted by the Higher Education Funding Council for England (HEFCE) – provided insight into the approach to peer review-based evaluations in the United Kingdom, the Research Assessment Exercise (RAE) and Research Excellence Framework (REF). In addition, the meeting looked at the use of peer review in PRFS and possible approaches for increasing efficiency and assuring necessary rigour in assessments.
This meeting focused on an element all performance-based research funding systems (PRFS) are struggling with: the assessment of the societal relevance of research. Possible approaches were discussed, based on international practice and the participating countries' expertise.