In recent decades, some EU Member States, particularly the Nordic countries, have invested in high-quality data infrastructures to capture the outcomes attributable to research and innovation (R&I) grants to businesses. While quantitative methods have become more and more sophisticated, a simultaneous interest in studying more qualitative, behavioural aspects can be observed. This kind of holistic systems analysis was explored and discussed during the first MLE on Ex-Post Evaluation of Business R&I Grant Schemes, which ran throughout 2016. The present MLE follows a challenge-driven approach and will include themes such as the use of Big Data in R&D grant evaluations, as well as methods for capturing behavioural change.
An overview of the Mutual Learning Excersice on Evaluation of Business R&D Grant Schemes
Grants and loans play a vital role in public innovation policy, prompting businesses to spend more on R&D and helping them overcome barriers to innovation such as risk aversion and market failures. With large sums – and even larger outcomes – at stake, R&D policy-makers need robust and reliable information, which demands a lot from evaluation methodologies.
This report has been prepared for a Mutual Learning Exercise on the Evaluation of Business R&D Grant Schemes.
This thematic report addresses the topic of applying mixed method approaches to the evaluation of public schemes to support R&D and innovation in firms, specifically business R&D grants and associated innovation schemes. It sets out the broad context for the use of mixed-method approaches in the evaluation of innovation support schemes, with a focus on business R&D grants.
This thematic report addresses the topic of understanding and measuring the behavioural change in firms through these schemes and challenges to capture these. It is essential that the community of STI scholars, STI evaluators and STI policy-makers acknowledge more fully the importance of measuring and capturing behavioural change through R&D and innovation schemes.
This report discusses the use of big data to evaluate grant schemes and other types of support for R&D and innovation by businesses. One aspect of big data – data linking – is already being implemented by several public agencies, while others – such as web scraping, text mining and machine learning – are less mature. The report includes examples from Norway, Spain, Sweden, the Netherlands, the UK and other countries, and provides an overview of the challenges faced: from data collection, data platforms and data analytics to the development of shared ontologies of relevant R&D actors, activities, support schemes and results, as well as data confidentiality and inclusion.
The kick-off meeting for this Mutual Learning Exercise (MLE) focused on the main themes chosen by participants of the first MLE on 'Ex-post evaluation of business R&I grants schemes' as well as on the organisation, planning and finalisation of the modus operandi of the exercise. The main findings of the previous MLE were presented, as were the themes of the first two country visits: big data and methods for capturing behavioural change. Participants from the host countries presented the state of the art in their countries, as well as the learning opportunities from the visits. Participants also had the opportunity to discuss their current practices, expectations and priorities regarding the chosen themes.
This first country visit provided participants the opportunity to learn about the methodologies and practices followed in Norway for the evaluation of research and innovation grant schemes for businesses and to discuss specific methodological challenges. These methodological issues included: the use of advanced analytics in management and policy development from the perspective of different policy actors, quasi experimental designs for the evaluation of business R&D grant schemes, and the use of big data. Big data approaches in policy evaluation face specific challenges related to data linking and data sharing, using new data sources and analytics and dealing with ethical issues related to linking, sharing and analysing more data.
Methods for measuring and understanding behavioural effects when evaluating R&D business grant schemes are essential to better understand the outcomes of this type of policy instrument. Such effects can include organisational routines and innovation capabilities, for example. The methodological and practical challenges of assessing behavioural change are complex and diverse. A firm’s behaviour is influenced by many more factors than just the evaluated scheme or set of schemes. There was a discussion on methods that open up the ‘black box’ of beneficiaries and look at how they benefit from R&D business grants in detail, as well as the associated challenges, during this two-day learning exercise in Stockholm.
This learning visit and workshop explored how evaluation approaches can be combined in order to better understand if and why a particular instrument is successful in addressing its objectives and achieving the expected impact. The participants had the opportunity to discuss with local experts the approaches used in the United Kingdom for evaluating the measures used to support R&I. Emphasis was given to how advanced quantitative methods can be combined with qualitative ones in order to better understand the impact of the schemes on the performance and behaviour of companies.
The final event of this MLE was a forum to discuss the participants’ main findings and reflections on the way forward. Belgium, Germany, Spain and Sweden presented their learning experiences, while the following sessions addressed the critical issue of how evaluation and policy-making should relate to each other. Liviu Stribat, Deputy Head of Unit for Better Regulation (A5) at DG Research & Innovation in the European Commission and Mart Laatsit, PhD fellow at the Copenhagen Business School, addressed the European perspective and the question of how to enhance evaluation-based policy-making. In a panel discussion, experts from industry, academia and policy-making debated how best to get evaluation and policy development to work together.