In recent decades, some EU Member States, particularly the Nordic countries, have invested in high-quality data infrastructures to capture the outcomes attributable to research and innovation (R&I) grants to businesses. While quantitative methods have become more and more sophisticated, a simultaneous interest in studying more qualitative, behavioural aspects can be observed. This kind of holistic systems analysis was explored and discussed during the first MLE on Ex-Post Evaluation of Business R&I Grant Schemes, which ran throughout 2016. The present MLE follows a challenge-driven approach and will include themes such as the use of Big Data in R&D grant evaluations, as well as methods for capturing behavioural change.
An overview of the MLE on Evaluation of business R&D grant schemes
This report discusses the use of big data to evaluate grant schemes and other types of support for R&D and innovation by businesses. One aspect of big data – data linking – is already being implemented by several public agencies, while others – such as web scraping, text mining and machine learning – are less mature. The report includes examples from Norway, Spain, Sweden, the Netherlands, the UK and other countries, and provides an overview of the challenges faced: from data collection, data platforms and data analytics to the development of shared ontologies of relevant R&D actors, activities, support schemes and results, as well as data confidentiality and inclusion.
This thematic report addresses the topic of understanding and measuring the behavioural change in firms through these schemes and challenges to capture these. It is essential that the community of STI scholars, STI evaluators and STI policy-makers acknowledge more fully the importance of measuring and capturing behavioural change through R&D and innovation schemes.
This thematic report addresses the topic of applying mixed method approaches to the evaluation of public schemes to support R&D and innovation in firms, specifically business R&D grants and associated innovation schemes. It sets out the broad context for the use of mixed-method approaches in the evaluation of innovation support schemes, with a focus on business R&D grants. The report focuses on a number of major issues which were addressed in the third site visit to London and illustrates these with selected example, and presents a number of the major lessons learned during the site visit concerning programme and evaluation design.
This report has been prepared for a Mutual Learning Exercise on the evaluation of business R&D grant schemes.
The kick-off meeting for this Mutual Learning Exercise (MLE) will focus on the main themes chosen by participants of the first MLE on 'Ex-post evaluation of business R&I grants schemes' as well as on the organisation, planning and finalisation of the modus operandi of the exercise. The main findings of the previous MLE will be presented, as will the themes of the first two country visits: Big Data and methods for capturing behavioural change. Participants from the host countries will present the state of the art in their countries, as well as the learning opportunities from the visits. Participants will also have the opportunity to discuss their current practices, expectations and priorities regarding the chosen themes.
This first country visit will provide participants the opportunity to learn about the methodologies and practices followed in Norway for the evaluation of research and innovation grant schemes for businesses and to discuss specific methodological challenges. These methodological issues will include: the use of advanced analytics in management and policy development from the perspective of different policy actors, quasi experimental designs for the evaluation of business R&D grant schemes, and the use of big data. Big data approaches in policy evaluation face specific challenges related to data linking and data sharing, using new data sources and analytics and dealing with ethical issues related to linking, sharing and analysing more data.
Methods for measuring and understanding behavioural effects when evaluating R&D business grant schemes are essential to better understand the outcomes of this type of policy instrument. Such effects can include organisational routines and innovation capabilities, for example. The methodological and practical challenges of assessing behavioural change are complex and diverse. A firm’s behaviour is influenced by many more factors than just the evaluated scheme or set of schemes. We will discuss methods that open up the ‘black box’ of beneficiaries and look at how they benefit from R&D business grants in detail, as well as the associated challenges, during this two-day learning exercise in Stockholm.
This learning visit and workshop explored how evaluation approaches can be combined in order to better understand if and why a particular instrument is successful in addressing its objectives and achieving the expected impact. The participants had the opportunity to discuss with local experts the approaches used in the UK for evaluating the measures used to support R&I. Emphasis was given to how advanced quantitative methods can be combined with qualitative ones in order to better understand the impact of the schemes on the performance and behaviour of companies.