COMMISSION STAFF WORKING DOCUMENT INTERIM EVALUATION of HORIZON 2020 - Hoofdinhoud

Council of the European Union

Brussels, 31 May 2017 (OR. en)

9787/17

RECH 211 COMPET 454 IND 142 MI 458 EDUC 272 TELECOM 148 ENER 255 ENV 551 REGIO 65 AGRI 289 TRANS 227 SAN 224

COVER NOTE

From: Secretary-General of the European Commission, signed by Mr Jordi AYET PUIGARNAU, Director

date of receipt: 29 May 2017

To: Mr Jeppe TRANHOLM-MIKKELSEN, Secretary-General of the Council of the European Union

No. Cion doc.: SWD(2017) 221 final - PART 1/16

Subject: COMMISSION STAFF WORKING DOCUMENT INTERIM EVALUATION of HORIZON 2020

Delegations will find attached document SWD(2017) 221 final - PART 1/16.

Encl.: SWD(2017) 221 final - PART 1/16

EUROPEAN COMMISSION

Brussels, 29.5.2017 SWD(2017) 221 final

PART 1/16

COMMISSION STAFF WORKING DOCUMENT

INTERIM EVALUATION

of

HORIZON 2020

{SWD(2017) 220 final}

{SWD(2017) 222 final}

• 1. 

  I NTRODUCTION

  Summary box: Key features of this evaluation

 A legal requirement.

 Focused on the evaluation criteria of effectiveness, efficiency, relevance, coherence, EU added value.

 Based on the analysis of a wide range of data sources contextualised with stakeholder consultation results.

The Horizon 2020 interim evaluation requires an assessment of the progress of the different parts of Horizon 2020 towards achieving its objectives, three years after its launch. This Staff Working Document summarises the main results of the evaluation, based on in-depth analyses which are available in the accompanying Staff Working Document on the In-depth Interim Evaluation of Horizon 2020 and the Annexes.

This interim evaluation covers the Horizon 2020 Specific Programme including the European Research Council with the exception of public-public partnerships (initiatives based on Article 185 of the Treaty), public-private partnerships (initiatives based on Article 187 of the Treaty), activities of the European Institute of Innovation and Technology, and the Euratom Framework Programme. While references are made to those initiatives in this evaluation, this is done without prejudice to the forthcoming separate dedicated interim evaluations of those

initiatives. 1 Joint Research Centre direct actions are part of the European and Euratom

Framework Programmes, but are evaluated separately.

• 2. 

  B ACKGROUND TO H ORIZON 2020

This Commission Staff Working Document presents the interim evaluation of Horizon 2020 - the Framework Programme for Research and Innovation 2014-2020 -, in line with Article 32

of the Regulation 1291/2013 i and the Commission's Better Regulation Guidelines 2 .

At the start of this document, it is useful to recall the key features of Horizon 2020, which should be kept in mind when reading the evidence gathered by the Interim Evaluation and summarised in this document.

Summary box: Key features of Horizon 2020

1 The European Institute of Innovation and Technology, the Euratom Framework Programme and the Article 185 and 187 initiatives have a separate legal base and will be covered by self-standing interim evaluations in separate Staff Working Documents to be published in the second half of 2017.

2  An EU research and innovation Framework Programme that is truly unique in the world in

terms of budget (about EUR 80 billion, the largest Framework Programme budget ever, but still below 10% of all EU public expenditure on research and innovation), duration (7 years), budgetary framework stability, and scope (research plus innovation; grants as well as loans, equity, and procurement; broad top-down focus on grand societal challenges as well as bottom-up frontier research; cross-border, cross-sectoral, inter-disciplinary collaboration, mobility, coordination).

 Pursuing an ambitious general objective (building a society and economy based on knowledge and innovation).

 Pursuing a number of focused specific objectives, mainly: to strengthen the EU's science base; to boost the technological leadership and innovation capability in the private sector; and to address the contribution of research and innovation to tackling societal challenges.

 A simple structure, aligned with the specific objectives, comprising three pillars: 'Excellent science'; 'Industrial leadership'; 'Societal challenges'.

 With a built-in innovation and impact orientation (challenge-based approach; funding all the way from lab to market; enhanced involvement of business, in particular Small and Medium-sized Enterprises (SME); impact-oriented call texts; expected impact to be spelled out in proposals; impact looked at in evaluation; regular reporting and monitoring).

 Inclusion of cross-cutting themes in societal challenges (e.g. Blue Growth/maritime 3 ,

circular economy, Internet of Things, Smart and Sustainable cities, Digital Security).

 Radical simplification (e.g. e-signatures, Participant Portal, single reimbursement rate, flat rate for indirect costs).

 Making in many respects a decisive positive break with past Framework Programmes (integration of research and innovation, accessibility, harmonisation).

 Allocation of funding through a strategic programming process and two-year work programmes.

 Wide range of instruments and actions.

 Excellence as guiding principle and main evaluation and selection criterion (no geographical consideration except in the Spreading Excellence and Widening Participation part).

 Close Commission monitoring of implementation.

Horizon 2020 is a European research and innovation Framework Programme that is truly unique in the world. It has an ex-ante decided total indicative budget of about EUR 80 billion - the largest Framework Programme budget ever, but still below 10% of all EU public budget for research and innovation - and a duration of seven years, which means that it stands alone in the world in terms of scale, duration and budgetary framework stability.

Horizon 2020 also integrates, for the first time, research and innovation support into a single programme, which means that through a wide range of different instruments seamless funding can be provided all the way from basic research to deployment, from the laboratory to

3 The focus area on Blue Growth and Maritime Research receives the second largest amount of funding under

Societal Challenge 2.

market 4 ; it funds both programmed topics in a broad range of grand societal challenges as well as fully bottom-up frontier research; and it is heavily focused on cross-border, crosssectoral, inter-disciplinary research and innovation project level collaboration, researcher mobility, and programming coordination.

In many respects, Horizon 2020 constitutes a break with the past. Before Horizon 2020, EU funding for research, education and innovation was covered by separate European

programmes (the 7 th European Framework Programme for Research, Technological

Development and Demonstration Activities (FP7), the innovation-related part of the Competitiveness and Innovation Programme, and the European Institute of Innovation and Technology, with different rules and implementation modalities.

When Horizon 2020 was adopted, a single framework integrating research, education and

innovation aspects for these programmes 5 was developed to deliver enhanced scientific,

technological and innovation impacts that would translate into larger downstream economic, competiveness, social, environmental and EU policy impacts. SMEs were targeted to benefit in particular from administrative simplification and enhanced access to innovation finance.

Far-reaching integration, simplification and harmonisation aimed to reduce costs for the Commission and for applicants. Measures like the acceptance of e-signatures, and the introduction of a single access point via the Participant Portal, of a single reimbursement rate, and of a flat rate for indirect costs increased the accessibility of the Programme.

Horizon 2020 pursues an ambitious general objective. Based on the broad recognition that research and innovation are key for helping Europe move towards smart, sustainable and inclusive growth and for tackling urgent societal challenges, Horizon 2020's overall objective is to contribute to: building a society and economy based on knowledge and innovation across the Union; the implementation of the Europe 2020 strategy and other Union policies; and the achievement and functioning of the European Research Area.

It intends to do so notably by leveraging additional research, development and innovation funding across the European Union and by contributing to attaining research and development targets such as spending 3% of the Gross Domestic Product on research and development by 2020.

Horizon 2020 pursues a number of focused specific objectives. These objectives 6 reflect

the key issues identified in the Horizon 2020 ex-ante impact assessment as needing to be addressed in order to remedy Europe's innovation gap: the need to strengthen the science base; the insufficient technological leadership and innovation capability in the private sector; and the insufficient contribution of research and innovation to tackling societal challenges and insufficient cross-border coordination.

4 It covers the scope of FP7, the innovation activities from the former Competitiveness and Innovation Framework Programme, as well as EU funding to the European Institute of Innovation and Technology. 5 A big part of the European action related to education is covered by ERASMUS+ and is thus outside the framework of Horizon 2020.

6 For an overview of the Horizon 2020 objectives, see Section 4.1 of the Staff Working Document on the indepth

interim evaluation of Horizon 2020 and Sections 2 of each thematic annex.

Horizon 2020 has a simpler structure compared to FP7 and it is aligned with the specific objectives comprising three pillars and two additional priorities, each involving a number of different actions. The three pillars are 'Excellent science', 'Industrial leadership' and 'Societal challenges'. The two additional priorities are 'Spreading excellence and widening participation' and 'Science with and for society'. The Joint Research Centre and the European Institute of Innovation and Technology are also expected to contribute to the Horizon 2020 objectives and priorities.

Horizon 2020 has a stronger built-in innovation and impact orientation than its predecessor FP7. Horizon 2020 takes a challenge-based approach, which means that it defines the challenges to be addressed but leaves it to the researchers and innovators to come up with the best solutions. For the first time, Horizon 2020 includes key performance indicators to measure the results and impacts of EU Framework Programmes. The Horizon 2020 Work Programmes also contain statements on expected impacts and proposals are expected to set out in detail how they propose to tackle the pre-specified challenge. Projects are expected to report regularly on progress towards achieving results and impacts.

Horizon 2020 provides seamless funding from fundamental research to deployment, from the laboratory to the market, with excellence as the core underlying principle. Unlike FP7, Horizon 2020 is based on a broad innovation concept, which is not limited to the development of new products, processes and services based on scientific and technological results but also incorporates the use of existing technologies in novel applications, and non-technological and social innovation. It includes activities closer to the market and end-users (e.g. proof-ofconcept, prototyping, testing, demonstrating, piloting, product validation, and market replication) and demand-side approaches. The 'Industrial leadership' and 'Societal challenges' pillars involve industry and SMEs and research and innovation at higher Technology Readiness Levels is supported.

The detailed allocation of Horizon 2020 funding to calls is decided through a strategic programming process resulting in two-year Work Programmes. These Work Programmes are prepared by the Commission in consultation with Member States and stakeholders and with input from advisory groups of experts. 7 In the case of the European Research Council, the Work Programme is established annually by an independent Scientific Council.

Each WP sets out the different funding opportunities available through calls for proposals and other actions such as public procurement. Each call for proposals contains topics and each topic describes the specific challenge to be addressed, the scope of the activities to be carried out, and the expected impacts to be achieved. Reacting to these calls for proposals and other actions, applicants submit as part of a competitive process project ideas that are evaluated by panels of independent experts. Grant agreements are concluded for selected proposals once all administrative and technical details are fixed. Operational and programme management tasks are to a large extent externalised to four Executive Agencies.

7 For this purpose 19 Horizon 2020 Advisory Groups have been set up as consultative bodies representing the broad constituency of stakeholders ranging from industry and research to representatives of civil society. Additional open and targeted consultation activities aim to obtain further views and contributions, including from the Enterprise Policy Group, European Innovation Partnerships and European Technology Platforms.

Different types of instruments and actions are used to implement Horizon 2020. It

deploys new types of action: the SME Instrument, innovation actions 8 , innovation procurement and inducement prizes. 9 The bulk of the budget is granted to collaborative

research and innovation projects whereas single beneficiaries are supported through the European Research Council, Marie Skłodowska-Curie Actions and the SME Instrument. In the first years of the programme, a special form of collaborative projects was also piloted under the Fast Track to Innovation scheme, focusing on industrial actors and rapid turnaround. Moreover, the Commission undertakes direct research and innovation activities through its Joint Research Centre.

The management of the programme is mainly entrusted to Executive Agencies. Continuing the trend of externalising programme implementation to Executive Agencies that began under FP7, four Executive Agencies are responsible for operational and programme

management tasks across most of the programme. 10 For specific parts of the programme,

programme management is carried out by different types of partnerships, with the private

sector (Public-Private Partnerships) and the public sector (Public-Public Partnerships). 11

Excellence is the main guiding principle for selecting projects to be supported throughout the programme together with impact and the quality of implementation. Geographical origin criteria are not used, except for the 'Spreading Excellence and Widening Participation' part.

The Commission monitors the implementation of Horizon 2020 through annual

monitoring reports 12 , based on Horizon 2020’s key performance indicators listed in the legal base. 13 The fact that for the first time these Key Performance Indicators are identified

prior to the start of the Framework Programme is a significant development compared to previous Framework Programmes. In addition, the legal base indicates a list of 14 crosscutting

issues that need to be closely monitored in the Horizon 2020 implementation. 14 .

8 These actions focus on demonstrations, tests and activities close to applications. They are used for areas where the scientific and technology insights are available and the focus shifts to turning these into applications. 9 Prizes are a 'test-validate-scale' open innovation approach that brings together new-to-industry players and small players that may pursue more radically new concepts than large, institutionalized contestants. Inducement prizes offer an incentive by mobilising new talents and engaging new solver communities around a specific challenge. Inducement prizes are awarded upon the achievement of the target set, solving the challenge defined.

10 Executive Agency for Small and Medium-sized Enterprises (EASME), European Research Council Executive

Agency (ERCEA), Innovation and Networks Executive Agency (INEA); Research Executive Agency (REA).

11 Support is also provided for coordination actions, studies, expert groups, conferences, as well as the

dissemination and exploitation of results, including specific actions to underpin R&I policy initiatives (e.g. Policy Support Facility, Belmont Forum, Innovation Deals). There is also support to communication measures, including to the public at large.

12 https://ec.europa.eu/research/evaluations/index_en.cfm?pg=monitoring

13 https://ec.europa.eu/programmes/horizon2020/en/news/horizon-2020-indicators-assessing-results-and-impact

href="https://ec.europa.eu/programmes/horizon2020/en/news/horizon-2020-indicators-assessing-results-and-impact-horizon">horizon

14 An overview of the results of the key indicators of Horizon 2020 benchmarked against FP7 can be found in Figure 8 of the Staff Working Document on the in-depth interim evaluation and in Annex 1 Part D to this Staff Working Document.

• 3. 

  E VALUATION QUESTIONS

In accordance with the Commission's Better Regulation Guidelines 15 , the Horizon 2020

interim evaluation addresses evaluation questions under each of the sections, which are structured around five evaluation criteria:

• Relevance: assessment of whether the original objectives of Horizon 2020 are still relevant and how well they still match the current needs and problems;

• Efficiency: the relationship between the resources used by Horizon 2020 and the changes it is generating;

• Effectiveness: how successful Horizon 2020 has been in achieving or progressing towards its objectives;

• Coherence: how well or not the different actions work together, internally and with other EU interventions/policies;

• EU added value: assessment of the value resulting from Horizon 2020 that is additional to the value that could result from interventions which would be carried out at regional or national levels.

• 4. 

  M ETHOD

The evaluation is based on the triangulation of a wide range of data sources contextualised with results from stakeholder consultations. This section summarises the main methods used for this interim evaluation. A complete overview of the methods used can be found in Annex 1, section C.

Summary box: Data sources used for this evaluation

 Horizon 2020 monitoring reports, statistical data from the Commission’s internal IT Tools, Eurostat/OECD data;

 In-depth analyses carried out by responsible Commission services - often with support from external expert groups, studies and/or dedicated beneficiary surveys – on specific Horizon 2020 objectives ('thematic assessments'), cross-cutting issues, the funding model, and various Horizon 2020 instruments/actions (Article 185/187 initiatives, Fast Track to Innovation, SME Instrument and European Institute of Innovation and Technology) (see Annexes);

 External horizontal studies - covering the entire Horizon 2020 programme - on publications and networking based on Scopus data ('Elsevier study'), and on EU Added Value and economic impact ('PPMI study'), which includes a counterfactual analysis and macroeconomic modelling, as well as work of an Expert Group on Evaluation Methodologies providing inputs on the relevance of the programme using text- and data mining tools;

15 More information here: http://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htm. The Interim Evaluation of Horizon 2020 also covers an evaluation of the issues mentioned in Article 32 of the Regulation 1291/201315 (cross-cutting issues, an in-depth assessment of public-private partnerships including the Joint Technology Initiatives, the Fast Track to Innovation pilot, and the funding model of Horizon 2020)

 Analyses from other EU institutions, such as the Council Conclusions on the FP7 ex-post Evaluation, opinions and reports on the interim evaluation from the European Parliament, the European Economic and Social Committee and the Committee of Regions, relevant Court of Auditors reports.

Summary box: Stakeholder consultations used in this evaluation

 Annual surveys of National Contact Points launched in the context of the Horizon 2020 Annual Monitoring reports (415 replies in 2016, 33.2% of target group);

 A survey on the impact of the simplification measures introduced in Horizon 2020 targeting participants in ongoing Horizon 2020 projects (4,125 replies, 10% of target group) and all types of stakeholders (595 replies);

 A Call for Ideas launched within the context of preparations for a possible European Innovation Council (over 1,000 replies, 183 supporting documents);

 The stakeholder consultation on the Interim Evaluation of Horizon 2020, (close to 3,500

replies; over 300 position papers). 16

Stakeholder box: The stakeholder consultation on the Interim Evaluation of Horizon 2020

• 3,483 online questionnaire responses out of which 24% are not beneficiaries

• More than 350 position papers

• Disaggregated analysis of questionnaire responses (according to stakeholder type and number of replies)

• Qualitative analysis of the position papers

• Testimonials used as illustrations in the interim evaluation

• Key stakeholder views (complementing evaluation findings):

o High satisfaction rate: 78% of respondents are satisfied or very satisfied with the programme overall; 86% among business representatives;

o 89% of respondents agree that an increased budget is needed for financing

research and innovation at EU level.

This interim evaluation has been coordinated by the Evaluation Unit of the Commission's Directorate-General for Research & Innovation with the support of other Commission Services via working and inter-service groups. Work started in 2016 based on Terms of Reference formally adopted by the Commission after a vote by the Member States’

Programme Committee. 17

16 A full analysis of the stakeholder consultation (both the questionnaire and the position papers) is provided in Annex 1 Part B. Input received from stakeholders, including in position papers, is summarised in blue boxes throughout this Staff Working Document.

17 C(2016)5546

• 5. 

  L IMITATIONS AND CHALLENGES OF THIS EVALUATION

This section summarises the key limitations and challenges that characterise this interim evaluation and contextualise its results. A complete overview of the limitations and challenges of this interim evaluation can be found in Annex 1 section C.

Summary box: Key challenges and limitations of this evaluation

 It is too early to discuss the impact of Horizon 2020. The overwhelming majority of Horizon 2020 outputs, results and impacts has not yet materialised as most of the projects to be funded have not been selected and started yet. Projects already started have not had the time yet to produce the full set of outputs, results and impacts, in line with the usual and widely acknowledged long time lags in research and innovation. Only 11,000 projects have started, of which only 10% were completed so far.

 Research and innovation programmes are notoriously difficult to evaluate because the pathways to impact are not linear.

 It is difficult to capture all direct and indirect results and impacts of a comprehensive programme like Horizon 2020, which operates in a multi-faceted policy context, raising the challenge of the attribution of the changes observed.

 The Horizon 2020 interim evaluation has been confronted with data availability, measurability and reliability challenges and by the lack of a full-fledged indicator system to track progress towards (societal) impact.

 As Horizon 2020 is unique in terms of budget, duration and scope,this interim evaluation has

used FP7 as a benchmark when possible and instructive. 18

It is too early to discuss the impact of Horizon 2020. The overwhelming majority of Horizon 2020 outputs, results and impacts has not yet materialised. This evaluation was carried out after only three years of this seven year programme. Less than 10% of all the projects signed had finished by the time of this evaluation (shorter and smaller projects from the SME Instrument and the European Research Council’s Proof of Concept, totalling only 0.6% of budget committed so far). Most projects did not have the time yet to produce the full range of anticipated results and impacts because of the substantial period of time it takes to produce scientific publications, patents, prototypes, demonstrators, etc., in line with the usual and widely acknowledge time lags in research and innovation.

Concretely this means that it is too early to carry out a full 'effectiveness' assessment, i.e. an analysis of progress towards achieving the objectives. The emphasis in the 'effectiveness' assessment will therefore necessarily be on expectations based on Horizon 2020 design features, first outputs and results (publications, patents and the like), and 'expected results and impacts'. On the other hand, much more can already be said on relevance, coherence, European added value, and efficiency (mostly on the inputs parts since the effects can only be estimated so far).

18 See Figure 8 of the In-depth Interim Evaluation of Horizon 2020 for an overview of the key benchmark

indicators used.

Research and innovation programmes are notoriously difficult to evaluate. The causal relation between research and innovation investment on the one hand and impact on the other hand is often indirect, and difficult to identify, measure, demonstrate and attribute. It is difficult to identify and describe all potential mechanisms to transfer research and innovation results to society. Each research and innovation field and industry is specific in how output is created and channelled to the end user. The identification of all end users who benefit from the research and innovation outputs can be difficult and/or costly, especially in the case of basic research. Basic research may have an impact in several dimensions, not all of which might be easily identified.

It is easier to identify and measure intended effects than unintended ones. As a result of spillovers, impacts could be partially the result of research performed abroad instead of local research investments, or the result of research carried out in other sectors. At the same time, some of the impacts, whether intended or unintended, may be achieved in other sectors, or other regions or countries, than the ones intended. Different research and innovation investments may have different time lags in having an impact on society. Research and innovation outputs, e.g. improved skills, may contribute to a set of different impacts, and therefore it may be difficult to identify them all in order to evaluate the contribution of the specific output, let alone that one of the research and innovation investment.

It is difficult to capture all direct and indirect results and impacts of a comprehensive programme like Horizon 2020, which operates in a multi-faceted policy context, raising the challenge of the attribution of the changes observed. The main reason is that a largescale programme like Horizon produces a wide range of direct as well as indirect results, impacts and spill-overs that are not easy to capture no matter how wide the evaluation net is cast, or are not quantifiable. Also, the research and innovation performance of countries is influenced by many other factors than Horizon 2020 only. The performance of Horizon 2020 should thus be seen in the context of its role in the wider research and innovation support system in particular taking into account the fact that Horizon 2020 accounts for about 10% of total public allocations and outlays for research and innovation in the EU as regards its positioning against (and impact on) the national and regional policy initiatives.

The Horizon 2020 interim evaluation has been confronted with data availability, measurability and reliability challenges and by the lack of a full-fledged indicator system to track progress towards (societal) impact. These relate to the fact that most Horizon 2020 indicators focus on inputs and results (publications, patents, prototypes) rather than impact (changes in CO2 emissions, health, security); are collected for one specific programme part only rather than the whole programme or have to be aggregated from uncoordinated data sources; or are not always fully reliable as data on publications and patents, for instance, are mostly based on self-reporting by project coordinators.

The programme's intervention logic also had to be reconstructed since the one of the impact assessment did not cover all changes made to the proposal during the co-decision procedure and lacked specificity on how to assess progress towards impact.

To overcome/mitigate these limitations, the interim evaluation is transparent in indicating its data sources and all underlying data sources will be made publicly available. Conclusions have been drawn based on the systematic triangulation of evidence from various data sources. All evaluation results have been thoroughly checked against input from stakeholders.

Furthermore, both short and long term areas for improvement have been identified in order to

overcome these limitations and challenges for the future. 19

• 6. 

  S TATE OF PLAY OF H ORIZON 2020' S IMPLEMENTATION

Horizon 2020 has only been implemented for 3 full years. This has nevertheless already resulted in large numbers of proposals and supported projects. This section summarises the key implementation data. A detailed overview of the implementation state of play can be found in section 5 of the In-depth Interim Evaluation of Horizon 2020 and Annex 1, part D.

Summary box: Key features of Horizon 2020's implementation

 Over 100,000 eligible proposals received so far.

 EUR 20.4 billion – just about one fourth of the total Horizon 2020 budget - allocated to 11,108 signed grants so far.

 Higher education, research and private sector organisations as well as SMEs as main beneficiaries so far.

 Three quarters of funding going to instruments supporting collaborative research and innovation so far.

 Participants from 131 different countries so far.

19 See section 12.1 of the In-depth Interim Evaluation of Horizon 2020.

As of 1 January 2017, over 100,000 proposals had been received, EUR 20.4 billion – just about one fourth of the total Horizon 2020 budget - had been allocated to 11,108 signed

grants. 20 EUR 7.5 billion had been allocated to 'Excellent science' (36.8%), EUR 4.5 billion

to 'Industrial leadership' (22%), EUR 7.4 billion to 'Societal challenges' (36.3%) and EUR 944.1 million to additional priorities. Within 'Societal challenges', 'Secure, clean and efficient energy' received 8.6% of Horizon 2020 funding, 'Health, demographic change and wellbeing' 7.6% and 'Smart, green and integrated transport' 7%. The challenge 'Secure societies' so far received 2.3% of overall Horizon 2020 funding.

The main beneficiaries of Horizon 2020 are higher education and research organisations, which together received 64.9% of the funding; the private sector received 27.7%, and public authorities and other types of organisations 7.3%.

Approximately 75% of all funding so far went to instruments supporting collaborative

research and innovation 21 bringing many organisations across countries together. 25% of funding went to single beneficiaries 22 to support excellent science through European Research

Council grants, for instance, or research and innovation projects for SMEs.

In the first three years of programme implementation, participants from 131 different countries (including 87 third countries) received funding from Horizon 2020. Participants in EU-28 countries received 92.9% of the funding, participants from Germany and the UK being the most frequent. Participants from Associated countries received 6.5% of all funding, participants from Israel and Norway being the most frequent, and participants from third countries received 0.6% of all funding, participants from the USA and South Africa being the most frequent.

• 7. 

  H OW RELEVANT HAS H ORIZON 2020 BEEN SO FAR ?

This question aims to determine whether the original objectives of Horizon 2020 as defined in its impact assessment are still relevant and how well they still match the current needs and problems of stakeholders. It also addresses the question of the flexibility of the programme against new scientific and socio-economic developments. This section summarises the key findings on relevance. The supporting evidence for the key findings can be found in section 6 of the In-depth Interim Evaluation of Horizon 2020; specific assessments of the relevance of

each individual Horizon 2020 programme part can be found in Annex 2. 23

20 Including EUR 0.5 billion in grants under Euratom.

21 E.g. Research and Innovation Actions, Innovation Actions, Marie Skłodowska-Curie Actions International Training Network, and Coordination and Support Actions.

22 E.g. Grants under the European Research Council or the SME Instrument.

23 See sections 3 of Annex 2 Part A – R.

Summary box: Key conclusions on relevance

 Horizon 2020’s original rationale for intervention and objectives remain largely valid.

 Closing the innovation gap and maintaining industrial leadership remains valid key objectives for the EU and Horizon 2020. The importance of supporting breakthrough, market-creating innovation is now more clearly recognised than when designing Horizon 2020.

 Further strengthening the EU's science base is as necessary as ever and remains a valid Horizon 2020 objective.

 The societal challenges identified when conceiving Horizon 2020 still exist and are valid continued priorities for the EU and Horizon 2020.

 The continued relevance of Horizon 2020 also lies in its contribution to the achievement of a wide range of EU and global objectives such as the Sustainable Development Goals.

 Horizon 2020 has been flexible enough to support research on urgent new needs (e.g. Ebola and Zika outbreaks, migration) as well as new, promising science and research.

 Emerging priorities and new developments still need to be scouted continuously and the right balance has to be found between being too prescriptive or not prescriptive enough.

 The strategic programming process improved the intelligence-base underpinning programming choices though stakeholders call for even greater transparency.

 The high application rate, including from newcomers, also shows the relevance and attractiveness of Horizon 2020 for stakeholders.

 Stakeholders' substantial reasons for participating illustrate the relevance of Horizon 2020.

 The wider public's understanding of the benefits of publicly supported research and innovation and the involvement of civil society in Horizon 2020 can be further improved.

Horizon 2020’s original rationale for intervention and objectives remain largely valid as the challenges identified at programme launch still exist.

Closing the innovation gap and maintaining industrial leadership remains valid key objectives for the EU and Horizon 2020. The importance of supporting breakthrough, market-creating innovation is now more clearly recognised than when conceiving Horizon 2020. Investing in research and innovation remains a precondition for Europe’s competitiveness and achieving many of the key EU policy objectives. The EU currently underinvests in research and innovation. The level of R&D expenditure in the EU-28 lies at 2.03% in 2015, which is still below the 3% target of the Europe 2020 Strategy. In spite of some improvements, the ‘innovation gap’ identified at programme launch still exists. The EU- 28 continues to be less innovative than key competitors, but performance differences have

become smaller. 24

In particular, Europe still displays a structural gap in R&D investments (public and private), and in the uptake of innovations, which are key to improving productivity growth and spurring the creation of new jobs. Relative to the USA, the EU lags far behind in high tech sectors and is home to fewer young companies that have grown into world-leading innovators. It is now more clearly recognised that such companies play a key role in bringing about the

24 See section 6.1.1. of the In-depth Interim Evaluation of Horizon 2020.

necessary breakthrough, market-creating innovation. In addition, patent applications are

declining in many EU countries.. 25

Further strengthening the EU's science base is as necessary as ever and remains a valid Horizon 2020 objective. The EU’s public sector research system is large and diverse and remains the largest producer of knowledge in the world, having a strong educational. However, it still has to catch up in terms of the production of highly influential research with, relative to its size, comparatively few centres of excellence that stand out at world level and with large areas of average and poor performance. Between 2005 and 2014, the EU-28 caught up with the USA in terms of the share of top 1% most highly cited publications, each accounting for about 40% of the world's top-cited publications, but China's scientific production is on the rise.

The societal challenges identified when designing Horizon 2020 still exist and are valid continued priorities for the EU and Horizon 2020. The Societal Challenges identified at programme launch remain valid in the present socio-economic context and are reinforced by the Sustainable Development Goals/COP21-22 framework. There is also a clear scientific rationale for investing in research and innovation to address these Goals.

The relevance of Horizon 2020 also lies in its contribution to the achievement of a wide range of EU and global objectives. Strengthening Europe’s science base, boosting industrial leadership, addressing societal challenges and cooperating internationally remain instrumental for achieving many of the current key EU and international policy objectives. Evidence collected within the thematic assessments overall shows that Horizon 2020 remains an important mechanism for supporting and delivering on the current set of EU policy objectives.

Even if preceding the Juncker Commission priorities (from 2014) and the "3Os" put forward by the research and innovation Commissioner (in 2015) - which call for open science, open innovation and openness to the world - Horizon 2020 is highly relevant, contributing and delivering on these Commission priorities. International obligations such as the implementation the SDGs have further increased Horizon 2020's relevance since investing in

research and innovation has been recognised as essential for achieving this 26 . A text mining

approach by an independent expert group also concludes that Horizon 2020 is very much in

line with EU and international priorities. 27

Horizon 2020 has been flexible enough to support research on urgent new needs (e.g. Ebola and Zika outbreaks, migration) as well as new, promising science and research. The bottom-up, open and non-prescriptive nature of most of the actions supported under the 'Excellent science' pillar for adapting flexibly as needs arose,channelling funds to new and promising research areas, including on multidisciplinary research. Conversely, biennial programming under the 'Societal challenges' pillar may at times be too rigid to integrate

25 See section 6.1.1. of the In-depth Interim Evaluation of Horizon 2020.

26 Investment in R&I is also recognised as an important aspect of EU's comprehensive response to harnessing globalisation, COM(2017) 240 i.

27 For more information on this expert group, see Annex 1 Part C.

swiftly new and "urgent" topics dictated by disruptive and counter-intuitive technologies and

business models that cannot be anticipated over any length of time. 28

A text mining approach by an independent expert group also concludes that Horizon 2020 takes into account subsequent technological and scientific advances to a high degree. Grantees funded through the European Research Council work in 25 of 28 independently identified key research fronts. According to the results of a keyword-based content analysis of their abstracts, an important number of FET projects focus on technologies that are expected to have significant potential to drive economic impact and disruption by 2025.

Stakeholder box: What do stakeholders say?

77% of stakeholder consultation respondents agreed fully or to some extent that the thematic coverage of Horizon 2020 is flexible enough to cope with changing circumstances, 12.4% fully disagreeing. Non-governmental organisations tend to disagree more than any other category of respondents.

93% of consultation respondents also agree at least to some extent that Horizon 2020 supports the latest developments in research and innovation at the national/European and international level, the most positive being businesses and public authorities and the most negative being non-governmental organisations.

Emerging priorities and new developments need to be scouted continuously and the right balance has to be found between being too prescriptive or not prescriptive enough. Evolutions of the socio-technological framework (incl. digitisation, servitisation,, data revolution, social conflict, violence and security concerns, emerging epidemics, SDGs) are expected to profoundly impact the Horizon 2020 context in the coming years, calling for a constant review of priorities and scouting of developments. A right balance is also to be found between being too prescriptive or not prescriptive enough, depending on the pillars and areas. There is also scope for ensuring a stronger strategic alignment of basic/fundamental research with future needs.

28 See section 6.2 of the In-depth Interim Evaluation of Horizon 2020.

Example box: Horizon 2020's quick reaction to the outbreaks of

Ebola in 2014 and Zika in 2015

The outbreak of Ebola in West Africa was the major international public health emergency of the past few years. Horizon 2020 promptly supported urgent research on Ebola by launching – for the first time

– two fast-track procedures completed in a very short timeframe. 29

Funding was mobilised despite not being foreseen in the Work Programme. In parallel, IMI (a Public Private Partnership between the EU and the European Federation of Pharmaceutical Industries and Associations) launched a call in record time taking into consideration the dual nature of IMI. This very significant Horizon 2020 response of EUR 140 million leveraged, in turn, a further EUR 101 million from the European pharmaceutical industry.

Horizon 2020-funded Ebola actions have supported the development of all three leading Ebola vaccine candidates, developed diagnostic tests and produced critical new knowledge about the virus itself. In spite of the enormous challenges, the research was done timely and with due respect to all Horizon 2020 and international ethical standards 30 . These actions have placed the EU second only to the US Government in terms of commitments made 31 . The Commission has also strived to coordinate other Ebola research funders by establishing frameworks for cooperation to enable a swift and effective global research response in future outbreaks.

Horizon 2020 has taken the lead in establishing the Global Research Collaboration for Infectious Disease Preparedness (GloPID-R) that links together research funders, the scientific community, industry, patient groups and public health actors. Its goal is build up the research capacity so that an effective research response can be launched within 48 hours of an outbreak. It was tested with the Zika outbreak in Latin America in 2015, when an emergency call for research on Zika was published, in coordination with other funders of preparedness research. Overall, Horizon 2020 allocated EUR 30 million to address the urgent Zika research gaps and EUR 15 million for research on Zika vaccines and for infrastructures for mosquito research.

The strategic programming process works, though stakeholders call for even greater transparency. Compared to FP7, the involvement of stakeholders in programme design has increased. Thematic assessments highlight that, compared to FP7, the strategic programming process has improved the intelligence base underpinning programming choices and has helped better define the focus of the programme in line with stakeholder needs. However, the transparency in the Work Programme formulation process, the participation of stakeholders/citizens in the agenda-setting and the ease of finding the right call are areas for

improvement. 32

The high application rate, including from newcomers, also shows the relevance and attractiveness of Horizon 2020. The relevance of Horizon 2020 is illustrated by the high

29 While following all Horizon 2020 rules as the Financial Regulation foresees the possibility to award grants without a call for proposals in exceptional and duly substantiated emergencies. This procedure was planned during September 2014, with results announced and projects launched in October. The IMI-Ebola+ call was launched on 6 November, with results announced mid-January 2015. 30 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5240928/ 31 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5112007/ 32 See sub-sections 3 of Annex 2 Parts A – R.

demand for programming funds. 33 More than 100,000 proposals have been submitted so far;

more than 33,000 per year, compared to 20,000 in FP7. The increased interest is evident throughout the programme but especially in the SME instrument, which has generated more than 30,000 proposals (only 5,000 in the 'Research for the benefit of SMEs' programme under FP7).

The private sector demonstrates an increased interest based on the share of total applications (37.4% in Horizon 2020 against 26.2% in FP7). 29.5% of the applications came from newcomers to the programme: 28,551 new SMEs (out of 46,034 new companies), 3,024 new higher and secondary education institutions, 2,464 new research organisations, 2,815 new public bodies, and 4,309 ‘Other’ organisations. The thematic assessments highlight, however, that the involvement of some participants/stakeholder groups can be superficial –with

organisations being involved only to meet programme requirements 34 .

Stakeholders' substantial reasons for participating illustrate the importance and relevance of Horizon 2020. Stakeholder consultation results highlight three main reasons for participating in Horizon 2020: financial support; access to new knowledge and know-how; unique opportunities for collaboration with European or international partners and for contacts

with key players that were often the best in their field. Respondents value the opportunities to strengthen partnerships inside existing networks as much as the ability to meet new partners or build new networks. Interdisciplinary work and the opportunity to work with other types of actors also stand out.

As regards the forms of funding, 76% of stakeholder consultation respondents agreed that grants for collaborative projects are 'fully' or 'largely' relevant to their needs, while 49% agree for grants for single beneficiary projects. The other forms of support (co-fund, prizes, financial instruments and public procurement) were less well known to the respondents. 7.5% of the consultation respondents who did not participate in Horizon 2020 underline that they lacked an adequate type of financial support and 14.6% mentioned that the programme lacked a relevant area for their needs.

The wider public's understanding of the benefits of publicly supported research and innovation and the involvement of citizen and/or civil society in Horizon 2020 can be further improved. Innovations resulting from Horizon 2020 are likely to benefit all types of stakeholders – including citizens - and have the capacity to address several of Europe’s most pressing societal challenges, from climate change to improved civil security. There is, however, a gap in society in the understanding of the benefits of publicly-funded research.

Moreover, research can be brought closer to the public. There is strong support for the

involvement of civil society in Horizon 2020. Responsible Research and Innovation 35 is a

cross-cutting issue in Horizon 2020, which aims to encourage societal actors to work together

33 The In-depth Interim Evaluation assesses this issue further and concludes that this is not due to a substitution

of decreased national funds, see section 8.2.2.3.

34 See section 6.3.3 of the In-depth Interim Evaluation of Horizon 2020.

35 Responsible research and innovation is promoted via: public engagement, open access, gender, ethics, science

education, and integrated actions that promote institutional change.

Stakeholder box: What do stakeholders say?

86% of stakeholder consultation respondents agreed fully or to some extent that Horizon 2020 priorities address the main citizens’ needs, whereas 5% judge that it is not the case at all. The most negative respondents are non-governmental organisations.

The involvement of non-governmental organisations and civil society organisations still appears to be low. 48.5% of respondents agree that an increased involvement of citizen in priority setting is needed to further maximize the socio-economic impact of the EU framework programme for research and innovation whereas 37.9% disagree (13.6% don’t know).

The most positive are the non-governmental organisations. An overwhelming majority of representatives of civil society organisations surveyed by the European Economic and Social Committee (81%) either agree or strongly agree that civil society organisations should be involved in Horizon 2020 project consortia.

during the whole research and innovation process to better align research and innovation with the values, needs and expectations of society. An external study found that the participation of civil society organisations increased slightly compared to FP7 (2.3% in Horizon 2020 compared to 1.4% in FP7). This contrasts with the monitoring data, which suggest that 11% of Horizon 2020 projects are relevant for responsible research and innovation. However, it is unclear to what extent citizens, civil society organisations and other societal actors really contribute to the co-creation of scientific agendas and scientific content in those projects. In fact, a network analysis shows that civil society organisations that do participate generally take on non-core roles in project consortia and rarely coordinate.

• 8. 

  H OW EFFICIENT HAS H ORIZON 2020 BEEN SO FAR ?

This question aims to consider the relation between the inputs of the programme (i.e. resources, budget, selection processes) and the outputs and impacts achieved by the programme. Since this is a mid-term review of the programme, the assessment mainly refers to the efficiency of the programme management (e.g. grant management, proposal evaluation) and implementation processes (e.g. selection and participation patterns). This makes it possible to shed light on whether the way in which the programme is managed is likely to influence positively or negatively the outputs that will be generated. This section summarises the key findings on efficiency. The underlying evidence for the key findings can be found in section 7 of the In-depth Interim Evaluation of Horizon 2020; specific assessments of the efficiency of individual Horizon 2020 programme parts can be found in the thematic annexes

(Annex 1). 36

Summary box: Key conclusions on efficiency

 Horizon 2020 is generally expected to be at least as cost-effective as FP7 if the most recent macro-economic projections materialise.

36 See sub-ections 5 of Annex 2 Parts A – R.

 Compared to FP7, Horizon 2020's efficiency is positively influenced by the extensive externalisation of programme implementation to new management modes including Executive Agencies.

 Horizon 2020's efficiency has been enhanced compared to FP7 through the creation of a Common Support Centre ensuring the harmonised implementation of Horizon 2020's rules for participation across the different actors implementing the programme.

 Horizon 2020's efficiency has been improved compared to FP7 through the large-scale simplification of the rules of participation, to the satisfaction of stakeholders.

 In particular Horizon 2020's funding model has been greatly simplified compared to FP7.

 Simplification is never finished - it is a continuous endeavour.

 Horizon 2020 suffers from underfunding resulting in large-scale oversubscription, much larger than under FP7, which constitutes a waste of resources for applicants and a loss of high quality research for Europe.

 The proposal evaluation process is generally highly regarded but some aspects such as the feedback to applicants could be improved.

 For successful applicants, benefits are estimated to largely outweigh costs.

 Horizon 2020 funding reaches a wide range of stakeholders and newcomers. Broadening participation is a point of attention across Horizon 2020.

 Horizon 2020 promotes intensive collaboration between different types of organisations and scientific disciplines.

 Horizon 2020 allocates its funding across economic sectors in a balanced manner.

 Horizon 2020 is open to the world and has a broad international outreach but international cooperation can be further intensified.

Horizon 2020 is generally expected to be at least as cost-effective as FP7. Based on the macro-economic modelling exercise, using projections up to 2030, the estimated internal rate of return of Horizon 2020 is 30% (in line with scientific literature estimations). EUR 19.9 billion (27%) of the Horizon 2020 budget has been spent on grants during the first three years of programme implementation. Implementation data for other, non-grant based instruments under the programme is currently not tracked in a comparable or centralised way to make an overall assessment.

Horizon 2020's efficiency has been increased compared to FP7 through the extensive externalisation of programme implementation to new management modes including Executive Agencies. Further externalisation increased cost-efficiency compared to FP7 since the most resource-intensive parts (i.e. actions with a high number of grants, 60% of the budget) were outsourced to New Management Modes such as Executive Agencies. There is evidence that this resulted in increased administrative efficiency. Currently, Horizon 2020's level of administrative expenditure stays below the level observed in FP7 and below the target of 5% of the overall budget (administrative expenditure is maximum 3.6% in the Executive

Agencies). 37

37 For the detailed assessment, see Section M of Annex 2.

Horizon 2020's efficiency has been enhanced compared to FP7 through the creation of a Common Support Centre ensuring the harmonised implementation of Horizon 2020's rules of participation across the different actors implementing the programme. A single Common Support Centre has been created that in a horizontal manner provides consolidated legal, audit, business process, IT and knowledge exploitation and dissemination services for all actors implementing Horizon 2020.

Horizon 2020's efficiency has been improved compared to FP7 through the large-scale simplification, including the rules of participation, to the satisfaction of stakeholders. The simplification of participation rules decreased costs for participating stakeholders. The simplification effort has dramatically reduced the time-to-grant, which is now 192 days on average, a decrease by more than 100 days compared to FP7. Horizon 2020's simplification has been welcomed overwhelmingly by stakeholders, as demonstrated in the box below.

Example box: Results from the dedicated survey of beneficiaries on simplification

Of those respondents with experience in FP7 and Horizon 2020 who expressed an opinion, 75% confirmed that, overall, the processes in H2020 are much simpler than in FP7. Only 20% indicated that they know other funding programmes that are simpler than Horizon 2020. About 90% considered the e-signature and the Participant Portal beneficial, 85% considered the 8 month time to grant target to be beneficial, 71% considered the single reimbursement rate and the single flat rate for indirect costs beneficial, and 69% considered the ‘no negotiation’ approach beneficial.

In particular Horizon 2020's funding model has been greatly simplified compared to FP7. The new funding model has mobilised and satisfied stakeholders. It is much simpler than the one used in FP7. In particular, the single reimbursement rate in a project and the single flat rate for indirect costs have been welcomed by a great majority of stakeholders. They can be assumed to have contributed to the attractiveness of Horizon 2020 as reflected in application statistics.

While a direct comparison of funding levels is not possible, estimations show that the average real funding level in Horizon 2020 remains at 70%, the same as in FP7. The new funding model has thus, overall, had positive effects on stakeholder appreciation, time-to-grant and attractiveness while not impacting significantly the level of co-funding brought into the projects.

Simplification is never finished, it is a continuous endeavour. Throughout the implementation of Horizon 2020, the Commission has continued to assess the scope for further simplification. One possible area for improvement is the broader acceptance of beneficiaries' usual accounting practices. Stakeholders indicate that there are still too many instances where they have to collect data and information specifically to meet obligations for their Horizon 2020 grants, in parallel to their usual accounting systems. This concerns in particular the obligations on staff time recording, the accounting for depreciation of equipment and for internally provided consumables and services, the handling of personnel costs outside closed financial years, and some accounting detail for beneficiaries outside the Euro zone.

The Commission has already reacted to these concerns and adapted the Horizon 2020 model grant agreements accordingly. Another area for improvement concerns the unintended effects of the additional remuneration scheme with the EUR 8,000 cap. Opportunities for further simplification will also open with the revision of the EU Financial Regulation and the Commission initiative on Budget Focused on Results. The Commission proposal for the revision of the Financial Regulation provides for better conditions for the use of simplified forms of funding (unit costs, flat rates, lump sums).

Horizon 2020 suffers from underfunding resulting in large-scale oversubscription, much larger than in FP7. Compared to FP7, Horizon 2020 has experienced a significant increase in the number of proposals submitted per year (34,025 per year or a 75.9% increase if compared to FP7). This has led to oversubscription and a drop in success rates from 18.5% in FP7 to 11.6% in Horizon 2020, making some parts of the Programme strongly underfunded, notably Societal Challenge 6, Science With And For Society, Marie Skłodowska-Curie Actions and the Fast Track to Innovation Pilot where less than 20% of the high quality proposals (i.e. those meeting evaluation thresholds) were retained for funding (for Future and Emerging Technologies, less than 10%). Overall, Horizon 2020 would have needed over EUR 60 billion of additional funds to support all proposals which scored above the quality threshold in the proposal evaluation process (which represent 44.7% of total eligible proposals).

Oversubscription leads to wasted resources for applicants and a loss of excellent research for Europe. The total cost of preparing proposals has been estimated to amount to roughly EUR 636 million a year, which, given the low success rates, is a waste of resources for applicants. Because of budget constraints, projects already judged to be excellent by independent expert evaluators cannot be funded, which is a loss for Europe.

The proposal evaluation process is generally highly regarded but some aspects could be improved. The thematic assessments point to a need for improvement with respect to the currently suboptimal level of feedback provided by evaluators to applicants, a trade-off between speed and quality of the evaluation reports in the face of large increases in numbers of applications. This was also reflected in the results of the stakeholder consultation: 34% judged that the quality of feedback from evaluators is 'poor' or 'very poor'. Stakeholders noted that feedback is not detailed enough and varies greatly from one evaluation panel to another.

For successful applicants, expected benefits outweigh costs. Despite the low success rate, the costs borne by different stakeholder groups for proposal submission, and concerns about the quality of feedback received from proposal evaluation, costs for participating stakeholders seems to be proportionate given the expected benefits of participation, which go beyond the financial contribution received.

Horizon 2020 funding reaches a wide range of stakeholders and newcomers. Overall participants come from no less than 131 countries. More than half of the organisations participating in Horizon 2020 did not participate in FP7 and almost half of them are SMEs. These newcomers received 14% of the total budget so far. A concentration of funding in terms of participants and geographical representations can be observed, which is explained by the focus on excellence of Horizon 2020.

Overall, in Horizon 2020, the Top 100 beneficiaries receive 32.9% of the total funding (1.7 percentage points lower than in FP7). The concentration is strongest for universities and research organisations and lowest for the private sector. Beneficiaries from the UK, Germany, Spain, Italy and France received 59.4% of the overall funding.

Broadening participation is a point of attention across Horizon 2020. This takes different forms, going from an in-depth analysis of the involvement of specific beneficiary groups in Horizon 2020 actions, to specific actions such as Coordination and Support Actions, establishing Memoranda of Understanding with relevant stakeholders and multipliers, or targeted awareness raising events.

The extent of current EU-13 participation in Horizon 2020 is broadly reflecting the size of the population, the number of researchers, or the scale of R&D investment. Though some themes register a better participation of countries with a lower research and innovation performance than others, and better than in FP7, it is still quite low and only slowly improving. There are noticeable performance differences and heterogeneity among the EU-13 countries and across Horizon 2020 programme parts. Participants from EU-13 countries represent 8.5% of the participations in Horizon 2020 and receive 4.4% of the overall funding (4.2% in FP7). Researchers in a number of EU-13 countries, e.g. Slovenia and Estonia, are in spite of an overall lower Horizon 2020 contribution outperforming the EU-15 average, taking into account the size of the population, the number of researchers and national investments in research and innovation.

Researchers based in Cyprus and Hungary also receive more grants from the European Research Council than would be expected based on national research investment. This points to heterogeneity in the EU-13 and widening groupings. In terms of project size, EU-13 countries participate more in larger projects (i.e. projects above EUR 5 million), but coordinate more often small projects (i.e. projects below EUR 200,000).

Example box: Widening participation instruments

The Teaming action (associating advanced research institutions to other institutions, agencies or regions for the creation or upgrade of existing centres of excellence) is a new feature under Horizon 2020. It is meant to provide new opportunities to the parties involved, with real prospects for growth through tapping into new collaboration and development patterns, including the establishment of new scientific networks, links with local clusters and opening up access to new markets. This will offer national and local research new possibilities for exploitation and value creation and boost the innovation potential of thecountries involved.

Twinning aims to strengthen a defined field of research in a knowledge institution through linking with at least two internationally-leading counterparts in Europe.

The ERA Chairs scheme provides support for universities and other research institutions to attract and maintain high quality human resources and implement the structural changes necessary to achieve excellence on a sustainable basis.

Horizon 2020 promotes intensive collaboration between different types of organisations and scientific disciplines. The main collaborations in Horizon 2020 occur between the higher education sector and private firms (2,355 collaborative projects), the higher education sector and research organisations (2,289 collaborative projects) and between the private-for-profit sector and research organisations (2,169 collaborative projects). Horizon 2020 projects are supporting networks crossing scientific disciplines.

Horizon 2020 allocates its funding across economic sectors in a balanced manner. The amount of grants awarded to each sector is roughly proportionate to the number of companies in that sector - indicating that the Horizon 2020 allocation is not sector-specific: 80% of total grants to Horizon 2020 companies go to the three biggest sectors: 35% to Manufacturing, 30% to Professional, Scientific and Technical Activities, and 16% to the sector of Information and Communication Technologies. SMEs account for more than 75% of all Horizon 2020 companies in terms of employees and receive almost 60% of the EC contribution to the private sector.

Horizon 2020 has a broad international outreach but international cooperation can be further intensified. In total, applicants from 188 countries have applied and participants from 131 countries have been funded. There is a greater level of investment in multilateral initiatives compared to FP7. Yet the mainstreaming of international cooperation across Horizon 2020 did not lead to a transversal increase of international participation across the programme. The decrease in participation of international partners in Horizon 2020 is a cause for concern.

The main reasons are that, contrary to FP7, Brazil, Russia, India, China and Mexico are no longer automatically eligible for EU funding, very few topics are mandating international partners so far, and recent conflicts and socio-political developments in neighbourhood countries have led to uncertainties. Third countries represent 2.5% of the participations and 0.8% of the funding in internationally open collaborative projects (compared to 4.3% and 1.8% respectively in FP7) and 3.9% of beneficiary or partner organisation participations (compared to 5.3% in FP7). Amongst countries that are not automatically eligible for funding from Horizon 2020, the most active in terms of participations are the USA, China, Canada, Australia and Brazil as compared to USA, Russia, China, Brazil and Australia under FP7. Nine of these countries have established co-funding mechanisms to provide funding to their participants in Horizon 2020 projects.

So far, projects resulting from joint/coordinated calls in Horizon 2020 have similar participations and EU contribution as in the corresponding period of FP7. Projects under public-private partnerships have either no or very few international participants (except for the Innovative Medicines Initiative) whereas public-public partnerships show a stronger international participation, with the third-country participation share in ERA-NETs at around 5% and the European and Developing Countries Clinical Trials Partnership featuring the participation of 14 African countries.

There is also a greater level of investment in multilateral initiatives compared to FP7. In health-related initiatives, around EUR 114 million were invested in 2014-2015, with matching investments of around EUR 532 million from partners in third countries. In activities related to climate action and the environment such as the 'Belmont Forum', the Group on Earth Observation and the Intergovernmental Panel on Climate Change, the Horizon 2020 contribution is close to EUR 200 million, while the total investment by all partners is estimated to be around three to four times this amount. Another example relevant in the context is the developing international maritime research, notably across the Atlantic (Galway Declaration).

In terms of associations to Horizon 2020, there are now 16 countries that have signed an

association agreement. 38 Some countries (Switzerland, Norway, Iceland, Israel and the Faroe

Islands) have long-standing participation in the EU Framework Programmes and a very strong performance. For the others (e.g. countries from the European Neighbourhood like Armenia, Georgia, Moldova, Tunisia and Ukraine) the association has contributed to the integration of their research and innovation systems in the European Research Area even though several still lack the national capacity needed to fully benefit from their association.

• 9. 

  H OW EFFECTIVE HAS H ORIZON 2020 BEEN SO FAR ?

This question aims to provide an insight into whether Horizon 2020 is on track to meet its objectives. This section summarises the key findings on effectiveness though a synthetic overview of the overall progress being made according to key expected impacts, which are not mutually exclusive and cover in each case the whole programme: scientific impact, innovation/economic impact and societal impact. The underpinning evidence for the key findings can be found in section 8 of the In-depth Interim Evaluation of Horizon 2020. Specific assessments of the effectiveness of individual Horizon 2020 programme parts can be

found in the thematic annexes (Annex 1). 39

9.1. Progress towards achieving scientific impact

As already mentioned, excellence is the main guiding principle in Horizon 2020, and all actions across all pillars are expected to contribute towards achieving scientific impact. It is expected that Horizon 2020 will contribute greatly to reinforcing Europe's scientific excellence; to improving cross-border and cross-sector coordination and integrating research and innovation efforts; and to enabling the emergence of new technologies or fields of science in the EU.

These changes are expected to result from the strengthening of research and innovation capabilities, scientific excellence and reputation (human capital development, reinforcement of EU research infrastructures, advancement of knowledge, publications and databases, scientific quality, reputation and scientific breakthroughs) and from an improved integration of research and innovation efforts (cross-sectoral, cross-border and interdisciplinary collaboration). Progress on these strands is expected to support the consolidation of the European Research Area.

As regards the continuous effort to build up research and innovation capacities across the EU- 28, the FP7 Capacities programme aimed specifically at developing the potential of EU-13

38 Of these, 12 since the start of the programme and 4 in 2015 and 2016, including Switzerland, which was partially associated until the end of 2016 and is now associated to all parts of Horizon 2020.

39 See sub-sections 4 of Annex 2 Parts A – R.

countries to participate to a larger extent in the programme. Horizon 2020 includes a legal mandate to maximise synergies with the European Structural and Investment Funds and developed a specific programme part dedicated to 'Spreading Excellence and Widening Participation', in addition to making it a cross-cutting issue in the whole programme. The objective is to ensure that participants from all EU countries are able to take part in the programme through a reinforcement of the excellence base and policy frameworks that are more conducive to research and innovation.

Summary box: Key conclusions on the progress towards achieving scientific

impact

 Horizon 2020 is making progress towards delivering scientific impacts through the reinforcement of research and innovation capabilities, scientific excellence and reputation and through the integration of research and innovation efforts.

 Horizon 2020 succeeds in attracting and involving the EU's and world's best research institutions and researchers.

 In particular the European Research Council and Marie Skłodowska-Curie Actions, and Future and Emerging Technologies but also other Horizon 2020 parts, train large numbers of researchers and contribute to Europe's human capital development, which in turns makes EU an attractive destination for excellent researchers worldwide.

 Pan-European research infrastructures supported by Horizon 2020 already contribute to Europe's excellent science with tools, materials and data accessible from across the EU and by supporting the mobility and training of researchers.

 Horizon 2020 has already succeeded in generating, and can legitimately be expected to continue to generate, a very large number of scientific publications and data.

 These are already to a large extent, but not yet fully, openly accessible to the wider scientific community and public.

 The first scientific publications resulting from Horizon 2020 are world class.

 Horizon 2020 is supporting projects with the potential to generate a large number of scientific breakthroughs.

 Horizon 2020 builds cross-sectoral, inter-disciplinary, intra- and extra-European research and innovation networks.

 Horizon 2020 is also making progress, albeit slowly, on spreading excellence in Europe.

 Stakeholders largely agree that Horizon 2020 produces excellent science.

 A substantial majority of stakeholders (64.7%) agreed fully or to a large extent that Horizon 2020 helps spread excellence and widen participation.

Horizon 2020 succeeds in attracting and involving the EU's and world's best research institutions. For instance, all EU top universities take part in Horizon 2020 as well as more than half of non-EU top universities. Almost all European research institutions in the list of the World's 25 Most Innovative Research Institutions take part in Horizon 2020, as do a third of the ones not based in the EU.

Horizon 2020 succeeds in attracting and involving the EU's and world's best researchers. Horizon 2020 has, for instance, supported a substantial number of Nobel Prize winners: 6 through the European Research Council, 8 through Marie Skłodowska-Curie Actions, and 10 through Future and Emerging Technologies, including several benefitting from different instruments. As of December 2016, grantees of the European Research Council had been the recipients of 526 major prizes, awards and other forms of recognition. As an illustration, the 2016 Nobel Prize in Chemistry - Prof. Ben Feringa – was awarded European Research Council Advanced Grants in 2008 and 2015.

Horizon 2020 involves and trains a very large number of EU-based researchers. An external study estimates, for instance, that no less than 300,000-340,000 researchers in the EU are fully or partly involved in EU-funded research activities. Horizon 2020 contributes greatly to the development of those researchers' capabilities and Europe's human capital. This is done through all supported activities but particularly the European Research Council, Marie Skłodowska-Curie Actions and Future and Emerging Technologies.

Marie Skłodowska-Curie Actions are attracting and retaining excellent researchers in Europe, with more than 140 nationalities funded since 2014. Around one in four Marie Skłodowska Curie Actions fellows are researchers from countries outside the EU Member States or Associated Countries and 9,000 researchers per year are undertaking international mobility under Marie Skłodowska-Curie Actions. Out of 3,200 Marie Skłodowska-Curie Actions grants (with 7,200 partners) to train researchers, 470 actions engage a total of 1,060 company labs and other commercial organisations.

No less than 23% of the PhDs and post-docs in European Research Council teams are from outside Europe. Over the course of the 6,500 currently running European Research Council projects, around 28,000 PhDs and postdocs will be part of the teams. There is also evidence of the longer term impacts of European Research Council grants on careers, on training highly skilled postdocs and PhDs, on raising the global visibility and prestige of European research, and on national research systems through its benchmarking effect. The prestige of hosting European Research Council grant-holders and the accompanying ‘stamp of excellence’ are also intensifying competition between Europe’s universities and other research organisations to offer the most attractive conditions for top researchers and to increase investment in research capacity and excellence.

The Future and Emerging Technologies programme has so far 1,278 participations of researchers in world-class research teams pursuing grand interdisciplinary scientific and technological challenges. The range of topics addressed is very broad, e.g. Artificial Intelligence for creativity, robots inspired by living creatures; artificial limbs that can feel as well as move; understanding financial crises and global epidemics; unbreakable cryptography, artificial photosynthesis, quantum technologies, the human brain, new materials like graphene, nanotechnologies, and next-generation computing.

Pan-European research infrastructures supported by Horizon 2020 contribute greatly to Europe's excellent science with tools, materials and data accessible from across the EU and by supporting the mobility and training of researchers. 29 EU infrastructures have reached the implementation phase. Another 21 are being developed. Thirteen new Pan European research facilities have been based on the new legal framework for the European Research Infrastructure Consortium. 365 national research infrastructures have been made accessible to all researchers in Europe and beyond, out of a target of 900 by the end of Horizon 2020.

Pan-European e-infrastructures support the networked provision of computing infrastructure and the development of major data-driven research infrastructures. A single open European space for online research, where researchers enjoy leading-edge, ubiquitous and reliable services and open access to e-Science environments, is being created through the federation of e-infrastructure resources at regional, national, institutional and European level realising the

European Open Science Cloud vision put forward in the European Cloud Initiative. 40 The

development of distributed European infrastructures and networked infrastructures has been transformative and stimulated scientific communities across Europe into cooperation – creating a solid basis of EU-level research. 35 e-Infrastructure grants have integrated, federated and/or consolidated e-infrastructure services into strong pan-European e Infrastructures that will form the nucleus of the European Open Science Cloud and enable the creation of new forms of science.

Example box: ELIXIR-EXCELERATE 41 , a Horizon 2020

infrastructure project

Project Type: INFRADEV; budget: 19 M€; duration: September 2015/August 2019

The project is aiming at accelerating the implementation and early operation of ELIXIR, the European life science Infrastructure for Biological Information, identified by ESFRI and the European Council as one of the three Europe’s priority research infrastructures. With 41 partners in 17 countries this grant coordinates and enhances existing resources into a world-leading data service for academia and industry, grow bioinformatics capacity and competence across Europe, and complete the management processes needed for a large distributed infrastructure. Four use cases: rare diseases, human data, plant genotype-phenotype and marine metagenomics, will help best tuning the services.

Horizon 2020 builds cross-sectoral, inter-disciplinary, intra- and extra-European research and innovation networks. Across the programme, more than one publication out of five (21.5%) so far is based on cooperation between academic and private organisations. The involvement of industry (including SMEs) in Research Infrastructures activities is still limited but a number of targeted actions have been launched to increase the involvement of industry, in particular as regard the supply of high tech components.

Interdisciplinarity is promoted throughout Horizon 2020 in order to develop solutions going beyond the scope of a single discipline or area of research practice. The share of Horizon 2020 publications that are inter-disciplinary is relatively high (7.5%) and slightly higher than in FP7. The EU-13 produce significantly more inter-disciplinary publications than the EU-15 and the share of the EU-13 of Horizon 2020 inter-disciplinary publications has doubled compared to FP7.

An analysis of co-publications shows that, compared to FP7, cross-border scientific networks are widening within the EU-28 to include smaller countries while networks at international level are decreasing, which is a cause for concern. Supporting the 'Open to the World' character of the programme, Horizon 2020 publications including authors from associated and third countries score up to over three times better in terms of citation impact than the world average. The most frequent co-publications occur between the EU-28 group, the USA, Japan,

40 European Cloud Initiative - Building a competitive data and knowledge economy in Europe, COM(2016) 178 i 41 https://www.elixir-europe.org/news/elixir-accelerates-major-horizon-2020-funding

Canada, China, Russia and Switzerland, just as in FP7. In addition, in FP7, many countries collaborated in publications with only one other EU-28 Member State, and this has so far also been the case for Horizon 2020. However, under FP7, many non-EU countries also had extensive links with other non-EU countries, whereas under Horizon 2020 this link is currently only observed with the USA.

Horizon 2020 has already succeeded in generating, and can legitimately be expected to continue to generate, a very large number of scientific publications and data. By 1 January 2017, Horizon 2020 projects had already produced 8,246 publications, about half of them peer-reviewed. The number of publications is expected to increase substantially in the years to come 42 .

These publications and data are already to a large extent but not yet fully openly accessible to the wider scientific community and public. Horizon 2020 aims at providing as much as possible open access to the publications and data resulting from EU funded research to the wider scientific community and public. The OpenAire database indicates that 65.2% of Horizon 2020 publications are in open-access and 65.4% of the projects covered by the Open Data pilot make scientific data accessible and re-usable. Furthermore, outside the areas covered by the pilot, a further 11.9% of projects participate on a voluntary (opt-in) basis.

The first scientific publications resulting from Horizon 2020 are world class. A preliminary assessment of the field-weighted citation impact of the 4,043 peer-reviewed Horizon 2020 publications so far confirms what was also observed for FP7: Horizon 2020 publications are cited twice as often as an average publication at world level. Horizon 2020 publications including authors from associated and third countries have three times more citation impact compared to the world average. 664 peer-reviewed publications in the database so far result from European Research Council funding. 7% of European Research Council publications so far (973, since its creation in 2007) are among the top 1% highly cited in the world by field, year of publication and type of publication compared with just 1.7% of

all publications with a European author. 43

Horizon 2020 is supporting projects with the potential to generate a large number of scientific breakthroughs. It is too early to identify major scientific breakthroughs for Horizon 2020 projects but a qualitative analysis of projects funded by the European Research Council since its creation in 2007 has established that 71% of them achieved a scientific breakthrough or major scientific advance. Four European Research Council grantees have been awarded the Fields Medal after being funded by the European Research Council. Horizon 2020 beneficiaries have also contributed to major scientific findings, including the Higgs Boson, the detection of gravitational waves, and the discovery of a planetary system composed of seven Earth-like worlds located relatively close to Earth. Projects funded under Future and Emerging Technologies also have a great potential to underpin new disruptive technologies leading to radically new markets and applications, in particular the Future and Emerging Technologies Flagships (Human Brain Project and Graphene). Major

42 Data related to indicators on publications are self-reported by beneficiaries during and at the end of the

projects, usually between 12 and 18 months from the projects’start date. There is a time-lag between the start of the project and the delivery of first scientific results. Based on the experience of FP7, the number of publications per year tends to increase significantly after the first three years of the programme and reaches its peak at its end.

43 See section 8.1 of the In-depth Interim Evaluation of Horizon 2020 for underpinning evidence.

breakthroughs in information and communication technologies can also be expected from the

Future and Emerging Technologies programme. 44

Example box: Results of the Graphene FET Flagship

The Graphene Flagship, which was launched in 2013 and will span over 10 years, is one of Europe's biggest ever funded research initiatives. It consists of an academic-industrial consortium of more than 150 partners in over 20 European countries. It covers the entire value chain, from materials production to components and system integration, and aims at developing applications in areas such as flexible electronics, printed electronics, 5G mobile technologies, batteries, aerospace, medical applications, filtration and automotive.

A recent breakthrough of the Flagship is the first fully functional microprocessor made from graphenelike materials that is a first step toward ultra-thin, flexible devices and holds promise for integrating computational power into everyday objects and surfaces. Another breakthrough is the development of graphene-based neural probes to examine brain activity in high resolution, which can help to better understand diseases such as epilepsy and disorders that affect brain function and motor control, as well as to improve neuroprosthetics by enabling control of artificial limbs. Additional promising results include highly efficient solar cells and ultrahigh sensitivity graphene infrared detectors (key for security screening).

Example box: Result of project funded by the European Research

Council amongst top ten physics discoveries of last decade

The European Research Council’s grantee Leo Kouwenhoven recently proved the existence of the “Majorana fermion”, a particle theorised in the 1930s. Detecting Majorana’s particles is not only exciting for particle physicists; thanks to their properties they could prove useful as stable “quantum bits” of information that could make quantum computers a reality.

In October 2015, the result of Prof. Kouwenhoven’s team was listed among the top 10 physics discoveries of the last 10 years by Nature Physics. The properties of the Majorana fermions could bring us one step closer to the much-talked-about high-speed quantum computers. In theory, the nature of the particles that can simultaneously be their own opposite could become a building block for quantum information processing and transmission.

Leo Kouwenhoven received an European Research Council Synergy Grant in 2012 together with Lieven Vandersypen and Carlo Beenakker to further work on bridging the gap between science and engineering in the field of quantum computing.45 Microsoft has recently hired four leaders in the field of quantum computing, including Leo Kouwenhoven, who will now build a Microsoft lab on the Delft campus. 46

44 For supporting evidence see section 8.1.1.4 of the In-depth Interim Evaluation of Horizon 2020 as well as Part

A Annex 2.

45 http://www.tnw.tudelft.nl/fileadmin/Faculteit/TNW/Actueel/Nieuws/Archief_2013/07_juli_2013/Mourik_Zuo_ copy_ENG.pdf

46 http://www.nature.com/news/quantum-computers-ready-to-leap-out-of-the-lab-in-2017-1.21239

Stakeholder box: What do stakeholders say?

Stakeholders largely agree that Horizon 2020 produces excellent science. 75% of stakeholder consultation respondents think that Horizon 2020 is fully or to a large extent helping foster excellent science. Research organisations and academia are above average in favour of the statement, while business recognises "to a large extent" that Horizon 2020 is helping to foster excellence science.

Horizon 2020 is also making progress, albeit slowly, on spreading excellence in Europe. It stimulates national reform of the research and innovation system and leverages higher and better research and innovation investments across Europe while always maintaining 'excellence' as the cornerstone objective and evaluation criterion. Dedicated Teaming, Twinning and ERA Chairs actions under the programme part ‘Spreading Excellence and Widening Participation’, as well as Policy Support Facility activities, contribute greatly to the strengthening of research and innovation institutions in the EU-13, in the process producing important structuring effects at national level and synergies with other EU programmes. With regard to Teaming phase 1, Twinning and ERA Chairs, 112 projects contribute to the ‘Spreading Excellence and Widening Participation’ objectives in the 19 Widening countries. Out of a total of EUR 254 million allocated, 73% of the funding went to partners from countries with a lower research and innovation performance.

The number of projects currently under implementation varies among countries, Portugal, Estonia, Poland and Cyprus being most successful in terms of participation. The Teaming action has attracted a lot of attention at political level, submitted proposals being either coordinated or supported financially by national or regional authorities. In several countries (e.g. Poland), national competitions were held by relevant Ministries in order to identify the best proposals to face the intense competition at the European level – a first in the history of Framework Programmes. Also, countries took the initiative of linking the actions with their Operational Programmes in European Structural and Investment Funds (e.g. Poland, Czech Republic). In a complementary way, with the aim of strengthening framework conditions, the Policy Support Facility (on-demand advice to policy makers on national research and innovation systems) has so far provided support to eleven countries.

Example box: SUPREME, Horizon 2020 twinning project for Polish

energy infrastructure 47

EU Contribution: EUR 1.047.551 ; Start date: 01/11/2015

The transition from fossil fuels to renewable and sustainable energy sources has become the European Union’s top developmental priority, with low-performing countries in Central Europe facing the most urgent need.

As the region’s largest country, Poland’s continuing economic progress has not come without significant costs; due to its history in electricity production, in 2009 it had the highest rate of production by coal of any EU member state. This makes Poland Europe’s third largest polluter in terms of damage to society, home to six of Europe’s 30 most damaging power-plants, and to be among

47 http://cordis.europa.eu/project/rcn/200260_en.html

Europe’s worst for public exposure to harmful pollution. At the same time it is experiencing rises in domestic electricity demand twice the EU average. This makes Poland the most urgent nation in the EU with regards to the need for immediate conversion to renewable energy systems and resources.

However, unlike traditional power facilities, energy produced by RES often produces unpredictable and variable outputs related to weather, season, and geographical location. While Polish research now has expertise in many of the technologies needed for energy transition, it lacks critical knowledge in modelling, planning, integrating, and managing large scale renewable energy systems in a flexible and effective manner.

The SUPREME project twins one of Poland’s best energy research centres, the Instytut Maszyn Przeplywowych Im Roberta Szewalskiego Polskiej Akademii Nauk with needed expertise in Denmark (Aalborg University), the Netherlands (University Twente), and Austria (the European Sustainable Energy Innovation Alliance).

Focusing on needed knowledge transfer in integrating energy technologies, the project’s wellformulated mix of extended staff exchanges, joint work, Summer Schools, and other events will create a long-lasting and effective partnership that will have a very significant impact on Poland’s energy systems infrastructure.

Stakeholder box: What do stakeholders say?

A substantial majority of stakeholders (64.7%) agree fully or to a large extent that Horizon 2020

helps spread excellence and widen participation. The share is similar for EU-15 and EU-13

respondents, but respondents from third countries (72.3%) and associated countries (67%) are

even more positive. The most positive types of stakeholders are SMEs and individuals whereas

the least positive are NGO.

9.2. Progress towards achieving innovation and economic impact

Compared to FP7 Horizon 2020 is providing a stronger emphasis on supporting closer to market applications and innovation. It is expected that this will lead to a reinforcement of EU industrial leadership and competitiveness; improve the innovation capability of EU firms; and generate jobs, growth and investments through the diffusion of innovation in the economy. Early signs show that progress is being made towards achieving this objective.

These changes are expected to depend on the advancement of knowledge and technologies, IPR and knowledge transfer (reinforcement of research and innovation capabilities of companies, knowledge flows and collaborations), on the reinforcement of framework conditions for research and innovation (leveraged investments and leveraged demand for future solutions) and the delivery of close to market outputs and diffusion of innovation in products, services and processes (standardisation, demonstration activities, innovation on the markets, growth of participating companies).

Summary box: Key conclusions on the progress towards innovation and economic impact

 More than FP7, Horizon 2020 succeeds in attracting and involving the private sector, a necessary precondition for the achievement of innovation and economic impact.

 More than FP7, Horizon 2020 succeeds in attracting and involving SMEs, the backbone of the European economy and necessary partners for achieving innovation and economic impact.

 Horizon 2020 is creating networks between businesses, and between the business sector, universities and research institutions, which is key for bringing knowledge quickly to market and gaining industrial leadership.

 Horizon 2020 provides companies, and in particular SMEs, with access to risk finance to carry out their innovation projects, thereby addressing an important market failure.

 Horizon 2020 invests in demand-driven innovation through innovative instruments including procurement and prizes but with low levels of take-up so far.

 Horizon 2020 already generates large numbers of high quality, commercially valuable patents and other intellectual property rights.

 Horizon 2020 already generates proofs of concept and demonstrators and supports the deployment of innovative solutions supporting the commercialisation and diffusion of innovation.

 Horizon 2020 projects already produce new knowledge, strengthen capabilities, and generate a wide range of innovation outputs including new technologies, products and services.

 Horizon 2020 supported projects already demonstrate potential in terms of generating breakthrough, market-creating innovation but such support can be strengthened substantially.

 Overall over half of all stakeholders (53%) think that Horizon 2020 is fully or to a large extent helping foster European industrial leadership, business being the most positive.

More than FP7, Horizon 2020 succeeds in attracting and involving the private sector, a necessary precondition for the achievement of innovation and economic impacts. In FP7, the private sector received 24.5% of the total EU financial contribution while in Horizon 2020, the private sector has so far received 27.7% of the total EU financial contribution. In FP7, the private sector accounted for 30.4% of all participations while in Horizon 2020, the private sector so far accounts for 33.2% of all participations. The share of the private sector is even higher in the 'Leadership in enabling and industrial technologies' and 'Societal challenges' parts where also two different types of Public-Private Partnerships (PPPs) are implemented, the Joint Undertakings (JUs) and contractual PPPs in industrial research and technological research and innovation at European level. Both pursue specific objectives through ambitious roadmaps, and assemble key players (with private sector participation of up to 60%) while demonstrating openness by a strong participation of non-members and SMEs.

Horizon 2020 succeeds in attracting and involving SMEs, the backbone of the European economy and necessary partners for achieving innovation and economic impacts. In FP7, SMEs accounted for 18.6% of the total number of participations and 14.6% of the total EU financial contribution. In Horizon 2020, focusing specifically on the key programme components 'Leadership in enabling and industrial technologies' and 'Societal challenges', SMEs so far account for 26.9% of the total number of participations and 23.9% of the total EU financial contribution.

Horizon 2020 is creating networks between businesses, and between the business sector, universities and research institutions, which is key for bringing knowledge quickly to market and gaining industrial leadership. Horizon 2020 generates industrial and crosssectoral knowledge creation and knowledge diffusion networks. Through its projects, Horizon 2020 creates partnerships that promote the flow of knowledge between different stakeholder communities, and the transfer of technology, data and information between participants and with the broader community, which is key for the creation and diffusion of innovation. The main collaborations in Horizon 2020 occur i.e. between the higher education sector and private firms (2,355 collaborative projects) and between the private-for-profit sector and research organisations (2,169 collaborative projects).

Horizon 2020 provides companies, and in particular SMEs, with access to risk finance to carry out their innovation projects, thereby addressing an important market failure. Under the Access to Risk Finance programme, 5,700 organisations have been funded – which is above the target of 5,000 - and EUR 13,235 million of private funds have been leveraged – the target being EUR 35 billion. The total investment mobilised via debt financing amounts to

EUR 29 600 million, which is above the target of EUR 25 billion 48 . Out of the 2,236 SMEs

taking part in the SME Instrument by end-2016, 88 companies secured a total of EUR 481 million venture capital during or after the project. SME Instrument funding (Phase 2) creates a leverage effect in the form of private co-funding of the innovation project of approximately EUR 800,000 per SME.

70% of surveyed Horizon 2020 project coordinators expect to secure additional R&D funding from private/industrial sources. Project coordinators based in the EU-13 expected to secure additional own and public national/regional funds less frequently than the EU-15 beneficiaries. Only a relatively small number of firms receiving grants under Horizon 2020 benefitted from financial instruments under Horizon 2020. This points to a potential lack of integration/interconnection between the grant and non-grant based instruments available to firms at different stages of the innovation cycle. This may hinder the scaling up to the European level of young innovative firms.

Horizon 2020 invests in demand-driven innovation through innovative instruments including procurement and prizes but with low levels of take-up so far. The use of new instruments such as pre-commercial public procurement, public procurement for innovation and inducement prizes clearly aims at leveraging the demand for future solutions. However given the small scale of the procurement actions so far, the main type of action supporting more user-driven innovation and leveraging demand for future solutions in Horizon 2020 comes from the inducement prizes. Overall, more could be done to support the demand for innovative solutions and user-driven innovation.

Horizon 2020 already generates large numbers of high quality, commercially valuable patents and other intellectual property rights. Beneficiaries of Horizon 2020 projects have so far declared 187 applications for intellectual property rights, of which 69 have been awarded..These numbers are expected to increase substantially in the years to come 49 . The vast majority of these consist of patents (153 applications; 39 awards) and trademarks (24 applications; all awarded). Two thirds of patent and trademark applications derive directly from SME instrument (Phase 2) projects, which can be explained by the higher technology readiness levels supported and the shorter duration of projects compared to other parts of the Horizon 2020 programme. 34 patent applications resulted from European Research Council’s 'Proof of Concept' projects.

48 For underpinning evidence see Annex 2 Part H dedicated to the financial instruments of Horizon 2020.

49 The bulk of patents are expected to come in from 2018 onwards, as the usual project lasts four years.

It is likely that applications for intellectual property rights deriving from projects in Innovation Actions and Research and Innovation Actions will take a more significant share in the near future. According to an external study based on a counterfactual analysis, EU-funded research teams are around 40% more likely to be granted patents or produce patent applications (25% of respondents produced at least one intellectual property right in 2015) than non-funded teams (18%). The data also show that the patents produced in the Framework Programmes are of higher quality and likely commercial value than similar patents produced elsewhere.

Example box: Nanopilot, a Horizon 2020 project on nanopharmaceuticals

Nanotechnology applied to medicine (nanomedicine) promises more effective and better targeted drugs, with reduced side effects for patients, but these nanopharmaceuticals are still at a very early stage of development.

The aim of NanoPilot (RIA; 6.3 million EUR; January 2015 – December 2018). is to establish a flexible and adaptable pilot plant for nanopharmaceuticals. It will provide specific tools and services to SMEs and researchers to validate their technologies and to be able to produce nanopharmaceuticals of sufficient quantity and quality to enter clinical testing. Not only does this help to overcome R&D challenges, but it also offers a solution to the high cost of manufacturing (e.g. clean rooms and special equipment), as well as compliance with regulatory requirements.

Three different applications show the flexibility of the planned facility: the treatment of dry eye syndrome, a HIV nanovaccine and a drug for the treatment of painful bladder syndrome. The pilot line will be validated in the project and will continue its certified services after the project, for further drugs and diseases. The consortium includes the operator of the pilot line, an SME, two university institutes which develop the nanopharmaceuticals, and a specialist institute on nanosafety.

Horizon 2020 already generates proofs of concept and demonstrators and supports the deployment of innovative solutions supporting the commercialisation and diffusion of innovation. 87% of the funding in innovation actions is allocated to demonstration actions and 8% to first-of-a-kind activities. The projects under the Leadership in Emerging and Industrial Technologies programme part are on track to deliver innovations (output involves demonstrators and pilots; an increase in technology readiness levels is visible) and bring clear market orientation. So far, the SME instrument (Phase 2) has produced more closer to market outputs per EUR 100 million compared to other types of action, followed by innovation actions: 3.6% of participating SMEs introduced innovations new to the market and 3.0% innovations new to the company. The integral coaching system of the innovation in SMEs programme has been an important enabling factor for these positive developments.

Example box: Proof of concept, demonstration and diffusion in

Horizon 2020

The European Research Council’s Proof of Concept Grant aims to explore the commercial and social potential of ideas arising from European Research Council grants. Since 2011 there have been around 540 Proof-of concept projects supported and 180 concluded. Of the first 140 projects around 20% of them spun-out a new venture. In November 2015 the European Business Angels Network awarded its first-ever prize for “Innovation in Science Venture Finance” to the European Research

Council as recognition of its efforts to bring frontier research closer to the market. 50

LEIT ICT projects aim at translating R&I into commercially viable undertakings, thus helping bridge the gap between research and the market. Ongoing projects include Demonstrating/Piloting Activities primarily relating to areas such as Content Technologies and Information Management, Robotics and Future Internet, Micro-and Nanoelectronics and Photonics. The Innovation Radar identified 274

innovations in Horizon 2020 Information and Communication Technologies projects 51 , the majority of

which are significantly improved products or new products which are going to be exploited either commercially (170 innovations) or internally by the organisations (61). For some of them (53) there are no plans for exploitation yet.

According to the assessment of the programme part dedicated to industrial leadership in Nanotechnologies, Advanced materials, Biotechnology and Advanced manufacturing and processing (NMBP), 75% of the projects aim at developing a new product; 60% a new process; 24% a new service 52 , and 4% an organisational or business model innovation. Particularly relevant are demonstrators on technology integration in an industrial environment, for example those from the dedicated Pilot Lines call, which include also open access pilot lines for SMEs. A total of 77 pilot lines have been developed so far. The work programme has set out specific requirements with regard to an initial description of the business plan already at proposal stage. This requirement stems from evidence that dealing with business plans at the end of the projects would be too late to be effective.

The Future and Emerging Technologies Innovation Launchpad is modelled after the European Research Council’s Proof-of-Concept scheme and seeks to give innovators and entrepreneurs freedom and flexibility to innovate from results of previous or ongoing projects funded under the Future and Emerging Technologies programme part. In order to create a wider and more diverse support base from which to take these innovations forward, the participation of new actors and of young and highpotential researchers and high-tech innovators is further encouraged in the Work Programme 2016-17 (already with success in 2014-15).

Under Societal Challenge 2 flagship projects are expected to create direct and indirect employment in some of the lagging regions of Europe. For example, the FIRST2RUN project is a flagship demonstration of an integrated biorefinery which is expected to revitalise local economies across Europe by reconverting old industrial sites and creating skilled jobs: an estimated 60 new skilled jobs will be created for every kiloton of bioplastics produced, taking into account the whole value chain, from agriculture to the end life of the final products.

Under Societal Challenge 7 the C-Bord project intends to develop and test a comprehensive and costeffective solution for the inspection of containers, and large-volume freight, in order to protect EU borders, These are tested by live field trials under real conditions at different border control points

Horizon 2020 projects already produce new knowledge, strengthen capabilities, and generate a wide range of innovation outputs including new technologies, products and services. The thematic assessments show that the portfolio of currently on-going projects is expected to produce, in addition to publications, new technologies, prototypes, tools, methods, databases, models, input into the development of industry norms and standards, new products, and new services. In all programme parts, the development of new knowledge and related

50 http://www.eban.org/eban-winter-university-2015-in-copenhagen-highlights

51 Data up to July 2016.

52 Indicating that these will play a role in the current tendency of European industry to introduce services.

learning effects are amongst the most frequent outputs expected from the projects. For private partners, and even more for SMEs, acquiring new knowledge and building research and innovation capacity are decisive.

Horizon 2020 is expected to increase the competitiveness of beneficiaries. This is a finding across the thematic assessments. The expected improvement mainly relates to access to new markets and the competitive position of partners internationally. There are clear indications that SME Instrument beneficiaries realise faster growth paths than control groups and the scale-up of their activities is more likely and/or more significant. The SME Instrument is intensively used by start-ups, especially the Phase 1 strand. However, the characteristics of Phase 2, in terms of e.g. Time-to-Grant and cash flow, discourage start-ups from participating.

Horizon 2020 has potential in terms of generating breakthrough, market-creating innovation. Innovation actions are some of key new actions introduced in Horizon 2020 to help bringing discoveries to the market. Initial findings indicate that a quarter of innovation actions have breakthrough, market-creating potential, and that companies and research institutions play a leading role in these initiatives. The SME Instrument focusses especially on product innovations, product performance innovations, and business model innovation. Service innovations, network innovations, and customer engagement innovations are less supported by the SME Instrument. More than half (53%) of Phase 2 beneficiaries report already having reached the market, or expect to do so in less than one year.

Example box: The Open Disruptive Innovation (ODI) scheme in the

Horizon 2020 SME instrument 53

The Open Disruptive Innovation scheme is the most popular topic within the SME instrument (one-third of proposals submitted). According to project participants, it contributes to the growth of highly innovative SMEs including start-ups. The most popular innovation fields of applicants include health, photonics and cloud computing. The case study interviews and desk research indicated that projects which implemented at least one Phase of the Open Disruptive Innovation scheme gradually increased their turnover and number of employees. Phase I supported in developing business market strategy which helped to expand their innovative product further. The turnover already increased slightly and the participants are expecting a gradual increase in the following years.

Many disruptive innovation products and services implemented under the Open Disruptive Innovation scheme have been commercialised and put to widespread use. For instance, after Phase I Global PERES, which offers an innovative device and mobile application designed to detect freshness of product and a risk of food poisoning, became popular in Europe and in the USA.

Project participants indicated that the Open Disruptive Innovation scheme supported their disruptive innovation to be further developed and expanded. Particularly Phase I was pointed out as essential. It supported SMEs to gain more knowledge and experience of entering to new markets and further helped to build a contact network for new potential clients. Project participants indicated that throughout the Phase I they all have established good networks in Europe.

Overall, the scheme is highly selective with a funding rate of 5,3% of the total submissions in Information and Communication Technologies. According to desk research unsuccessful proposals often fail due to the lack of a) market analysis to assess the competition and b) a robust and realistic

53 Source : CARSA (forthcoming), Support study for the interim evaluation of Horizon 2020 DG CONNECT

activities

emphasis on the commercialisation at the end of the project. 54 There is only a small amount of projects

which received grants for Phase II after the implementation of Phase I.

Support for breakthrough, market-creating innovation can be further enhanced. The programme has yet to make a significant outreach to young and fast growing innovative companies. A Commission call for ideas revealed that many stakeholders consider that important gaps still exist in EU support for disruptive, market-creating innovation. There is evidence that the programme should be able to better identify and support SMEs that are developing breakthrough technologies that cut across the boundaries between different

sectors, or support companies to scale up rapidly at EU level.. 55

The factors that have been identified as impeding full effectiveness in terms of fostering innovation with respect to market uptake and commercialisation are mainly technological, but relate also to the capacity of innovation ecosystems to address a range of issues, particularly for SMEs, from regulation and standards to technicalities such as the lack of a methodology to identify the dual-use potential of project results with a view to increase and diversify their market potential and access to finance, as well as a lack of customer acceptance of new solutions and a lack of access to a sufficient pool of end-users.

Stakeholder box: What do stakeholders say?

Overall over half of all stakeholders (53%) think that Horizon 2020 is fully or to a large extent

helping foster European industrial leadership, business being the most positive. A

comparatively low number of respondents (17%) agreed fully with this statement.

9.3. Progress towards achieving societal impact

Whereas FP7 was focused on specific domains, Horizon 2020 puts more emphasis on societal impact and aims at contributing through research and innovation to tackling the major societal challenges Europe and the world are facing. This means bringing together different technologies (including key enabling technologies such as digital), sectors, and scientific disciplines to find new solutions to these challenges but also taking on a stronger role at global scale for tackling these challenges. Progress is expected to depend on the typical results of research and innovation projects (e.g. scientific outputs, innovations) in domains of societal relevance.

As a continuation to the Science in Society programme in FP7, a dedicated programme part on "Science with and for society" is also included in Horizon 2020. The overall aim is to build effective cooperation between science and society, to recruit new talent for science and to pair scientific excellence with social awareness and responsibility. In parallel, gender equality, responsible research and innovation, and social sciences and humanities became cross-cutting

54 European Commission (2015), SMEs Instrument: ODI – Information Communication Technology – Implementation Update and Next steps. Presentation made by DG Connect

55 For supporting evidence, see section 8.2.3 of the In-depth Interim Evaluation of Horizon 2020, and Annex 1,

Part K (Fast Track To Innovation) and Annex 2, Part B (FET), I (SME Instrument).

issues promoted throughout the programme.

Summary box: Key conclusions on the progress towards achieving societal

impact

 Most Horizon 2020 projects, not only from the 'Societal challenges' pillar but also from the 'Excellent science' and 'Industrial leadership' pillars, are expected to generate key discoveries and technologies and cross-cutting societal impacts.

 The portfolio of Horizon 2020 projects selected under the 'Societal challenges' pillar and their progress are so far in line with the objectives set.

 Horizon 2020 projects already produce numerous results like publications, patents, prototypes, products, processes and methods in domains of societal relevance.

 Horizon 2020 has not yet met the targets for expenditure on sustainable development and climate action but it is expected that they will be achieved by the end of the Programme through decisive action in the remaining Work Programme 2018-2020.

 Stakeholders are less convinced about the role of Horizon 2020 in the resolution of societal challenges than in the achievement of knowledge-related objectives, which seems to call for better involvement of end-users and communication with citizens on the contribution that research and innovation can make to tackling societal challenges.

 Progress is made with respect to promoting gender equality under Horizon 2020 but data quality concerns remain.

 Results are encouraging in terms of the embedding of social sciences and humanities and Responsible Research and Innovation in Horizon 2020, even if highly uneven across the programme.

 A substantial majority of stakeholders (70.1%) agreed fully or to a large extent that Horizon 2020 is helping to support science with and for society.

Most Horizon 2020 projects, not only from the 'Societal challenges' pillar but also from the 'Excellent science' and 'Industrial leadership' pillars, are expected to generate key discoveries and technologies and cross-cutting societal impacts. Whereas projects under the 'Health' Societal Challenge are rather challenge-specific, projects in other Societal Challenges and also LEIT are expected to generate more cross-cutting impacts. Project coordinator’s survey data point to strong relevance of bioeconomy projects for the achievement of environmental objectives, to strong complementarity between energy and transport projects, and to strong relevance of health projects for the achievement of social

objectives. 56 The expected contribution of 'Excellent science' and 'Industrial leadership'

projects to the societal challenges is strong and evenly spread, with many technologies

developed with a view to provide solutions to a set of challenges. 57

The portfolio of Horizon 2020 projects selected under the 'Societal challenges' pillar and their progress are so far in line with the objectives set. Detailed assessments of progress

56 For underpinning evidence, see Part K of Annex 2.

57 For underlying evidence, see section 8.3 of the In-depth Interim Evaluation of Horizon 2020.

for each Societal Challenge indicate that the portfolio of projects selected and their progress are in line with the expectations set in the Work Programmes in accordance with the legal

base. 58

The main focus of funding so far under each area has been, respectively: the treatment and management of diseases; sustainable and resilient production and consumption systems and rural empowerment; low-cost, low-carbon energy supply; resource-efficient transport that respects the environment; the protection of the environment, sustainable management of natural resources, water, biodiversity and ecosystems and a sustainable supply of raw materials; inclusive, innovative and reflective societies, migration, societal polarisation and culture; and improved cyber security. More detailed information and specific examples of how Horizon 2020 projects contribute to tackling societal challenges can be found in the Indepth Interim Evaluation and the thematic annexes (Annex 1).

Horizon 2020 projects already produce numerous results like publications, patents, prototypes, products, processes and methods in domains of societal relevance. The projects funded under the 'Societal challenges' pillar have so far generated 809 peer-reviewed

publications 59 , mostly from the health, food/bioeconomy, energy and environment domains.

The majority of the 76 patent applications by, and 23 patents already awarded to, Horizon

2020 projects under the Societal Challenges pillar 60 is coming from the energy domain,

followed by health and transport. More than half of the 600 prototypes and testing activities developed are also coming from energy projects, which are also the strongest contributor to the launch of 106 new products, processes and methods into the market.

Stakeholder box: What do stakeholders say?

Views from the stakeholder consultation suggest that Horizon 2020 is perceived as

contributing less to addressing major societal challenges than to other objectives like

supporting growth and jobs. When looking at each Societal Challenge, more stakeholders were

of the opinion that Horizon 2020 is helping to foster a greater understanding of Europe,

providing solutions and supporting inclusive, innovative and reflective European societies

(SC6) (79% of agreement at least to some extent) and to improve the lifelong health and wellbeing

of all (SC1) (78% agree to some extent, but also 18% think the programme is not

helping at all). For all the other challenges, around 30% of the respondents do not know,

which is not surprising given the early stage of implementation.

Horizon 2020 has not yet met the targets for expenditure on sustainable development and climate action but it is expected that they will be achieved by the end of the Programme. Results of Horizon 2020 expenditure tracking for sustainable development and

58 See section 8.3 of the In-depth Interim Evaluation of Horizon 2020.

59 Given the low number of project completed so far and the length of the peer-review, publication and indexing processes, this number is expected to increase strongly in the following years of Horizon 2020.

60 Almost all patent applications (71) and patent awarded (22) in the Societal Challenges derive from SME-

Instrument projects.

climate action show that in the first three years of activity, the amounts spent did not reach the targets. For climate action, expenditure amounted to 27% against a target of 35% for the whole period of Horizon 2020. For sustainable development, expenditure amounted to 53.3 % against a target of 60 %.

Those figures represented a considerable increase in budget spent on research in those areas compared to FP7, however. For example, the 'Cooperation' Specific Programme of FP7 is estimated to have contributed EUR 2.4 billion to projects related to climate action, whereas for only the first three years of Horizon 2020 the equivalent figure (i.e. LEIT and Societal challenges together) is EUR 4.2 billion. The application of the OECD 'Rio Markers' methodology to bottom-up and thematic research funding still requires further optimisation and fine-tuning. A better alignment of the climate action and sustainable development tracking methodology with the Sustainable Development Goals (SDGs) would facilitate implementation by clarifying the scope of climate action and sustainable development in relation to globally-recognised goals. It is expected that the target will be achieved by the end of the Horizon 2020 through decisive action in the remaining Work Programme 2018-2020.

Progress is made with respect to promoting gender equality under Horizon 2020 but data quality concerns remain. Gender equality is implemented as a cross-cutting issue in Horizon 2020. One of the objectives of promoting gender equality is to ensure gender balance in decision-making. This is close to being achieved with 53% in advisory groups (33% in FP7) and 36.7% in evaluation panels. In addition, 6,022 experts (3,904 women and 2,118 men) declared having gender expertise in the Commission expert database in December 2016. Concerning the integration of gender into research and innovation content, the gender-flagged topics increased from 99 out of 610 topics in Work Programme 2014-2015 to 108 out of 568 topics in Work Programme 2016-2017.

Concerning the workforce, women represent 31% of projects’ coordinators, including 24.5 % of ERC Principal Investigators, 42.2% of MSCA Fellows and 26.9 % of scientific coordinators in other Horizon 2020 activities. It represents an increase compared with FP7, where women represented overall 28.5% of projects coordinators, 20% of ERC Principal Investigators, 36.5% of MSCA Fellows and 20% of contact persons for scientific aspects in other FP7 activities.

A qualitative analysis carried out by a dedicated Expert Group of a subset of 111 projects from gender-flagged topics showed that 53% of them included the gender dimension well or in part. Overall, there is room for improvement in the understanding of the notion of 'gender in research content' at applicant level, among evaluators and among Commission and Agency personnel. Among the 61 projects considered by project officers as having a gender dimension, only 35 projects actually included it. None of the 111 projects included gender training (eligible for funding in Horizon 2020) indicating that offering the possibility of having gender training as an eligible cost is not sufficient to generate take-up. Implementing gender in research content is challenging and calls for a better understanding of gender issues.

Results are encouraging in terms of the embedding of social sciences and humanities (SSH) in Horizon 2020, even if highly uneven across the programme. The integration of social sciences and humanities (SSH) is a cross-cutting issue that should be promoted throughout the whole Horizon 2020 programme. The quality of SSH integration is highly uneven across projects but almost half of the projects funded under SSH flagged topics show good or fair integration of SSH in terms of share of partners, budget allocated to them, and variety of disciplines involved. Societal Challenge 6 and its calls and topics attract many of the SSH disciplines. In the 2014-15 Horizon 2020 Work Programme, 37% of the topics have been identified as relevant for SSH researchers, and 41% in the Horizon 2020 Work Programme 2016-2017.

Contributions from economics, sociology, political science and public administration are well integrated while many other SSH disciplines are underrepresented, especially geography/ demography and philosophy/anthropology. The low participation of the humanities and the arts remains a challenge. Overall, EUR 433 million went to SSH partners in SSH flagged topics, representing 22% of the estimated total budget for the SSH flagged topics. In terms of countries represented, the SSH partners and coordinators in projects flagged as SSH relevant come predominantly from a group of 5/6 EU Member States. In the seven completed ERC frontier research grants calls under Horizon 2020, which operate on a 'bottom-up' basis without predetermined priorities, 1006 grants were awarded with 465 (21%) in the SSH domain, representing funding of over EUR 700 million.

The effectiveness of support to Science With And For Society has been limited due to the scale of the ambition, the relatively small budget allocation (less than EUR 450 million over 7 years) and the range of activities that are funded. Moreover, just a handful of projects are funded per topic, which spreads resources rather thinly, and there is an insufficient focus on areas where the greatest impacts are expected and on implementing sustainable institutional changes. Lastly, so far there has been a lack of clear objectives defined for all topics and projects; the under-representation of civil society and private companies in the funded actions; and the lack of focus on operationalizing ‘institutional change’, except for the gender equality lines (as an ERA priority).

Stakeholder box: What do stakeholders say?

A substantial majority of stakeholders (70.1%) agreed fully or to a large extent that Horizon

2020 is helping to support science with and for society. 21.4% agree to some extent and 3.3%

not at all. The most positive respondents are businesses and research organisations, whereas

the least positive are NGO and public authorities.

9.4. Progress towards achieving Horizon 2020's general objective

Summary box: Key conclusions on progress towards Horizon 2020's general objective

 Horizon 2020 is contributing to the creation of jobs and growth and the achievement of the priorities of the Juncker Commission.

 Through its focus on scientific, economic and societal impact,Horizon 2020 is projected to produce large-scale economic impacts.

 Horizon 2020 contributes to the achievement of a Digital Single Market.

 Horizon 2020 contributes to improved resource efficiency, a key vector of the Energy Union and climate Commisison priority.

 Horizon 2020 reinforces the European Research Area.

Through its focus on scientific, economic and societal impacts, Horizon 2020 is contributing to the creation of jobs and growth and the achievement of the priorities of the Juncker Commission. By pursuing its general objective of building a society and an economy based on knowledge and innovation, Horizon 2020 contributes to growth and jobs, to the implementation of the Europe 2020 strategy, to the priorities of the Juncker Commission and to other Union priorities. Having marked a definite shift towards innovation, Horizon 2020 has contributed significantly to the Innovation Union flagship, by improving and strengthening the framework conditions and facilitating access to risk finance for research and innovation.

Horizon 2020 is projected to produce large-scale economic impacts. It is difficult to assess the extent to which Horizon 2020 - which only represents a small proportion of total public R&D spending in the EU - is contributing to key performance indicators set to measure progress against the general objective (the 3% GDP target, innovation output indicator and share of researchers as part of the active population). Based on new macro-economic

modelling results 61 , Horizon 2020 is nevertheless expected to have a significant economic

impact in the medium to the long term. On average, the gain in terms of Gross Domestic Product compared to the reference scenario is estimated to amount to between EUR 24 and 35 billion per year during 2014-2030. The total gain in terms of Gross Domestic Product is between EUR 400 and EUR 600 billion: each 1 EUR of Horizon 2020 investment brings an increase in Gross Domestic Product of between EUR 6 and 8.5. In terms of employment creation, on average, during the period 2014-2030, the EU contribution through Horizon 2020 would increase the level of employment by between 110,000 and 179,000 units, including up

to 35,000 more jobs in research compared to the reference scenario. 62

Horizon 2020 contributes to the achievement of the Innovation Union and a Digital Single Market. Having marked a definite shift towards innovation, Horizon 2020 contributes to the Innovation Union flagship, by improving and strengthening the framework conditions and facilitating access to risk finance for research and innovation. About 30% of the Horizon 2020 budget so far has gone to actions promoting to some extent research and innovation in information and communication technologies in line with Digital Single Market Strategy for

Europe 63 , thus showing the enabling nature of such technologies, e.g. in helping address the

Societal Challenges.

Horizon 2020 contributes to improved resource efficiency, a key vector of Energy Union and climate Commission priority. Resource efficiency is being supported in almost all Societal Challenges and benefits from the large investment in sustainable development and climate, even if the respective expenditure target have not yet been achieved.

61 See section 8.4 of the In-depth Interim Evaluation of Horizon 2020

62 See section 8.4 of the In-depth Interim Evaluation of Horizon 2020. This section also contains a comparison with the projections of the ex-ante impact assessment.

63 See for instance the launch of the Cybersecurity contractual PPP, http://europa.eu/rapid/press-release_IP-16-

2321_en.htm

Horizon 2020 reinforces the European Research Area. It encourages the development of framework conditions to help European researchers remain in or return to Europe (e.g. through the provision of research infrastructures), to attract researchers from around the world (e.g. through the European Research Council and Marie Skłodowska-Curie Actions), to make Europe a more attractive destination for the best researchers (Euraxess, Resaver pan-European pension scheme, international cooperation), and to promote gender equality in research. Horizon 2020 also provides support to Member States and the main stakeholders in implementing the European Research Area reform agenda (e.g. through the Policy Support Facility).

• 10. 

  H OW COHERENT HAS H ORIZON 2020 BEEN INTERNALLY AND

  WITH OTHER (EU) ACTIONS SO FAR ?

This question involves looking at the extent to which Horizon 2020 actions work together, internally and with other EU interventions/policies. This section summarises the key findings on coherence. The supporting evidence for the key findings can be found in section 8 of the In-depth Interim Evaluation of Horizon 2020; specific assessments of the coherence of

individual Horizon 2020 programme parts can be found in the thematic annexes (Annex 1). 64

Summary box: Key conclusions on coherence

 The integration of research and innovation, the three pillar structure, the challenge-based approach, and the use of focus areas contribute to greater internal coherence of Horizon 2020 compared to FP7.

 Outside the 'Excellent science' pillar, Horizon 2020 is increasingly focused on research and innovation at higher Technology Readiness Levels, demonstration and deployment.

 The large number of European research and innovation funding instruments is difficult to understand for potential applicants and may lead to overlaps.

 Compared to FP7, greater efforts have already been made to increase the synergies between Horizon 2020 and other programmes, notably the European Structural and Investment Funds but these can be strengthened further.

 The Seal of Excellence is a prime example of the synergies established between Horizon 2020 and the European Structural and Investment Funds.

 There is great scope for synergies with European Fund for Strategic Investments, but there are also risks involved.

 There is a strong coherence between Horizon 2020 and international obligations..

 Horizon 2020 specifically aims to establish synergies with national programmes through the creation of long lasting collaborations between funding agencies and capacity building but do not seem to really influence the alignment of national strategies and policies in their current format.

64 See sections 6 of Annex 2 Parts A – R.

The integration of research and innovation, the three pillar structure, the challengebased approach, and the use of focus areas contribute to the internal coherence of Horizon 2020. The three pillar structure of Horizon 2020 and its focus on finding solutions to challenges (notably through focus areas promoting interdisciplinary solutions to multiple societal challenges) rather than being domain-oriented improved its internal coherence compared to FP7. The enabling nature of the LEIT part and the strategic roadmaps of cPPPs support underlying technologies for the next generation of solutions across societal challenges (health, energy, climate action, the circular economy). The Public Private Partnerships under LEIT-NMPB are specifically expected to have ambitious impacts with regard to energy and environment/climate. 71% of stakeholder consultation respondents agreed that combining different forms of support for research and innovation under one single programme better addressed stakeholder needs than having separate programmes. Internal coordination mechanisms are also put in place ensuring cross-fertilisation between services involved in the programme.

The use of focus areas (so-called cross-cutting activities) bringing together funding from the programme parts Leadership in Emerging and Industrial Technologies and Societal Challenges is particularly noteworthy and supported by stakeholders, allowing to channel funds into specific areas of interest for several parts of Horizon 2020 (e.g. Blue Growth, circular economy, Internet of Things, Smart and Sustainable cities, Digital Security) and increasing industrial competitiveness. However, some stakeholders are concerned that the use of too many focus areas leads to a complex reading and understanding of the Work Programme. In terms of actions, the newly introduced SME instrument is seen as being particularly complementary to other EU interventions through Horizon 2020, in particular Fast Track to Innovation and collaborative projects, providing a welcome addition to the Framework Programme toolbox.

Looking at the types of actions, approximately 75% of the funding goes to instruments

facilitating collaborative research and innovation 65 bringing organisations across countries

together. A quarter of the funding is allocated to single beneficiaries to support excellent science (European Research Council) or research and innovation projects of SMEs (SME Instrument). Based on data provided in the thematic assessments on Access to Risk Finance, Societal Challenge 1 and Societal Challenge 3, it is estimated that Horizon 2020 currently provides at least EUR 1 in financial instruments for every EUR 12 in grants.

There is scope for further streamlining the Horizon 2020 funding landscape. The Horizon 2020 funding landscape addresses the different objectives of the programme, but stakeholders point out that a large number of instruments and a complex funding architecture are difficult to understand and may lead to overlaps and prevent organisations from

identifying the instruments that best meet their needs. 66

Outside the 'Excellent science' pillar, Horizon 2020 is increasingly focused on research and innovation at higher levels of technology readiness, demonstration and

65 Research and Innovation Actions, Innovation Actions, Marie Skłodowska-Curie Actions International Training Network, and Coordination and Support Actions.

66 For more details, see Annex 1 Part B.

deployment 67 . Whereas the 'Excellent science' pillar focuses on more fundamental research

and, with the exception of e-Infrastructures, does not move beyond the stage of an experimental proof of concept, the rest of the programme is rather concentrated on higher levels of technological readiness, the majority of which are targeting product demonstration in both the Industrial Leadership and the Societal Challenges pillar.

The programme part dedicated to Future and Emerging Technologies plays a special role here, building new communities and innovation eco-systems and pushing new technologies up the scale of technology readiness towards innovation and impact. Different types of stakeholders regret that the programme parts related to Societal Challenges and Leadership in Emerging and Industrial Technologies do not invest more in lower levels of technology readiness for collaborative research, which is regarded as one key source of future breakthrough innovations, albeit longer-termed, in line with societal needs.

Efforts have already been made to increase the synergies between Horizon 2020 and other programmes, in particular the European Structural and Investment Funds and the European Fund for Strategic Investments, and these can be further strengthened. Both the Council and the European Parliament have highlighted the need for synergies and complementarities between EU funds for research and innovation, Cohesion Policy funds, and industrial competitiveness measures. Smart Specialisation strategies can provide a framework to develop complementarities through "upstream actions" to prepare regional research and innovation stakeholders to participate in Horizon 2020, and "downstream actions" to exploit and diffuse research and innovation results, developed under Horizon 2020 and previous programmes, into the market.

The Horizon 2020 and Cohesion Policy regulations now foresee the possibility to combine the funds in the same project, an alignment of rules for simplified cost options, and greater possibility to spend Cohesion Policy funding outside programme areas and the European Union. The Commission has produced guidance to relevant authorities on synergies and has proposed a further simplification to facilitate joint funding. Other initiatives to encourage further synergies include: (1) the Seal of Excellence (see below); (2) the Stairway to Excellence pilot project, which helps close the innovation gap between the EU regions by developing and exploiting the complementarities between Cohesion Policy, Horizon 2020 and other EU funding programmes; and (3) a Horizon 2020 Policy Support Facility Mutual Learning Exercise on Widening participation and ensuring synergies between the EU research and innovation programmes and Cohesion Policy.

The Seal of Excellence is a prime example of the synergies established between Horizon 2020 and the Structural Funds. In order to build upon the evaluations of high quality proposals under Horizon 2020 SME Instrument and Marie Skłodowska-Curie Actions, the Seal of Excellence initiative has been launched by the Commission to support synergies with national/regional initiatives by highlighting high quality projects that could not be funded by Horizon 2020. An increasing number of national and regional funding schemes are offering support to SME Instrument and Marie Skłodowska-Curie Actions proposals awarded with a

Seal. 68 However, comprehensive data on exact number of proposals for which these quality

67 Technology Readiness Levels are indicators of the maturity level of particular technologies. This measurement system provides a common understanding of technology status and addresses the entire innovation chain. The scale goes from TRL 1 – basic principles observed; to TRL 9 – actual system proven in operational environment.

68 See footnote 318 of the In-depth Interim Evaluation of Horizon 2020.

labels allowed applicants to secure other sources of public or private is not yet available. It is recognised that the initiative has not yet achieved its full potential, which would be possible, subject to the alignment of rules to further smoothen implementation.

There is a clear strategic willingness to ensure complementarity and synergies between Horizon 2020 and the European Structural and Investment Funds. However, strong evidence is lacking on how far this has materialised in practice yet. At project level there remains a need for a greater clarity of roles, responsibilities and coordination. Many beneficiaries point to differences in the rules governing Horizon 2020 and Cohesion Policy expenditure, notably in relation to eligibility rules, procurement and state aid rules. Particular attention needs to be given to the sustainability of investment in research infrastructures. These challenges are magnified in the context of interregional projects where expenditure possibilities outside the programme area are limited within the Cohesion Policy. These issues need to be addressed to streamline implementation.

Example box: Examples of coherence between Horizon 2020 and the

European Structural and Investment Funds (ESIF)

In the programme part dedicated to industrial leadership in Information and Communication Technologies (ICT) a number of cases have been identified where research activities under national programmes act as stepping stones to Horizon 2020 projects and, conversely, where FP/Horizon 2020 projects have led to research being funded by national or regional sources. However, survey results suggest that respondents had limited knowledge or experience regarding the synergies that could be developed by combining Horizon 2020 and other sources of funding. Survey findings also suggest that participation in Horizon 2020 does not seem to offer any competitive advantage for securing funding from other sources.

An analysis of FP7 programme part dedicated to industrial leadership in Nanotechnologies,

Advanced manufacturing and processing areas 69 showed that most regions have participated as

much as one would expect from their level of activity: regions with more R&D resources tend to participate more. The main factors for high performing regions are the track record and level of specialisation, but also the level of regional expertise. In this context, the creation of regional research centres, some of which were established in the 1980’s and 1990’s to diversify incentives to innovation, appears to pay off.

In the case of Spreading Excellence and Widening Participation the design of the programme entails synergies with cohesion policy in particular for Teaming where applicants are obliged to ensure appropriate co-financing for the infrastructure and equipment component of the centres of excellence from the ESIF or other sources. Beyond the mere financial dimension, the programme is well aligned with the overall objectives of cohesion policy notably to help less developed European countries and regions in order to catch up and to reduce the economic, social and territorial disparities that still exist in the EU.

In Marie Skłodowska-Curie Actions, investments from the European Structural and Investment Funds can be in support of COFUND, for instance in the form of investment in infrastructures, large equipment (European Regional Development Fund) or training and networking (mainly European Social Fund).

69 Study “Mapping the regional embeddedness of the NMP programme”, INNOVA et al., 2016. No significant differences between NMBP areas were detected, therefore these results are considered relevant also for Horizon 2020.

There is great scope for synergies with the European Fund for Strategic Investments, but there are also risks involved. The evaluation of the European Fund for Strategic Investments found that while there is a risk of competition between the European Fund for Strategic Investments and certain financial instruments (like Horizon 2020 InnovFin Large Projects), other Horizon 2020 instruments provided a strong boost to EFSI's support for SMEs (e.g. InnovFin SME Guarantee, InnovFin Equity). Loans are no substitutes for, but complementary to, grants.

There is a strong coherence between Horizon 2020 and international obligations, such as the implementation of the Sustainable Development Goals, for which investing in research and innovation is seen as crucial.

Horizon 2020 specifically aims to establish synergies with national programmes by providing top-up funding through the ERA-NET Cofund instrument and supporting Article 185 initiatives. Member State support to public-public partnerships has significantly increased

over the past years to more than EUR 700 million annually. 70 .

For ERA-NET Cofunds, based on the planning of actions resulting from the 2014-2016 Work Programme and past experience, an overall leverage effect on public investment of 3-5 can be expected (i.e. funding from Horizon 2020 is about EUR 280 million for pre-committed national budgets of about EUR 700 million, and additional calls without EU co-funding are planned).

The results of the ERA-NET Cofund evaluation highlight that the main added value of the ERA-NET scheme is the lasting collaboration between funding agencies. Capacity-building benefits are also perceived as very important, addressing not only research capacities, but also research approaches and improved research quality at national level. ERA-NET Cofund actions are relatively less perceived as strategic instruments that can influence national strategies and lead to the alignment of national policies among participating states and/or EU R&D policies. In addition, the coherence among ERA-NETs but also between the ERA-NETs and other joint initiatives is clearly underdeveloped and there is a lack of understanding of the financial aspects of the ERA-NET Cofund instrument.

As regards Joint Programming Initiatives, the overall level of ambition does not meet the initial expectations. The level of co-investment so far in joint calls and actions is no greater than for good ERA-NETs and the survey feedback does not indicate that this will increase significantly. Also, it seems that most countries are unwilling or unable to co-invest in the central executive resource that is needed to effectively implement the strategic agendas of the Joint Programming Initiatives. Whilst there are some notable exceptions, it seems that most countries are neither adapting their national research activities towards the strategic research agenda and its implementation plan nor the activities of the Joint Programming Initiatives.

An external study concludes that the Member State-led joint programming process is not sustainable, especially during times of severe economic austerity in many countries, without Union support; Member States often point out that, in the case of some countries (especially small), participation in an increasing number of Public-to-Public Partnerships causes

70 For more information,see Part H3 of Annex 2.

difficulties mainly due to the lack of administrative capacity and available national funding. Their large number also makes it difficult to understand for potential participants and national

authorities. 71

• 11. 

  W HAT HAS BEEN THE EU ADDED VALUE OF H ORIZON 2020

  SO FAR ?

This question aims to assess the value resulting from Horizon 2020 that is additional to the value that could result from interventions which would be achieved by Member States at national and/or regional levels. This section summarises the key findings on EU Added Value resulting from the interim evaluation. The underpinning evidence for the key findings can be found in section 9 of the In-depth Interim Evaluation of Horizon 2020. Specific assessments of the EU Added Value of individual Horizon 2020 programme parts can be found in the

thematic annexes. 72

Summary box: Key conclusions on European added value

 Horizon 2020 produces demonstrable benefits compared to national and regional-level support to research and innovation in terms of scale, speed and scope.

 Horizon 2020 increases the EU's attractiveness as a place to carry out research and innovation.

 Horizon 2020 is seen as improving the competitive advantage of participants.

 The additionality of Horizon 2020 is very strong – support is given to fund distinctive projects, which are unlike those funded at national or regional level.

 Stakeholders widely agree that Horizon 2020 has strong European added value.

Horizon 2020 projects are characterised by high additionality, which means that there are no substitutes. Horizon 2020 funds distinctive projects, which are unlike projects funded at national or regional level. Horizon 2020's additionality (i.e. not displacing or replacing national funding) is thus very strong. Survey results indicate that, on average, 83% of Horizon 2020 projects would not have gone ahead without Horizon 2020 funding. The figures are higher for Research Infrastructures (100%), Space (95%) and Future and Emerging Technologies (95%). The impacts of discontinuation of the programme are difficult to quantify, but are likely very large.

Horizon 2020's European added value derives from the economies of scale, scope and speed it delivers. Horizon 2020 produces demonstrable benefits compared to national and regional-level support for research and innovation in terms of scale, scope and speed, notably through the organisation of competitions at continental scale, the creation of cross-border, multidisciplinary networks, and the pooling of resources and creation of critical mass to tackle global challenges. It thus increases the EU's attractiveness as a place to carry out research.

71 See Part H.6 of Annex 1.

72 See sections 7 of Annex 2 Parts A – R.

The better network of Horizon 2020 supported European research units helped organisations attract more international talent.

Beneficiaries had more than twice as many researchers from other EU countries than similar non-funded teams. Moreover, beneficiaries’ research capacities and scientific outputs would have significantly decreased had they received national funding instead (i.e. higher effectiveness). This decrease would have been especially large in terms of their ability to collaborate with industry and business, transfer of knowledge, the number of participations in scientific conferences and the knowledge in new areas. Horizon 2020 also helped achieve results faster in 45% of the projects compared to what could have been achieved at national level.

Pan-European competition adds an additional layer of European added value to monobeneficiary parts of Horizon 2020. In particular, the European Research Council is recognised as a global brand synonymous with research excellence, with substantial structuring effects in the Member States. The SME Instrument is providing support to Europe’s potential highgrowth innovative SMEs that goes beyond the possibilities on offer through the innovation support measures of individual member states and regions. The combination with the Seal of excellence has been particularly appreciated, and a structuring effect on national and regional SME support schemes is emerging.

Example box: European Added Value in the fight against Anti

Microbial Resistance

Antimicrobial Resistance (AMR) is the ability of microorganisms to resist antimicrobial drugs. Various pathogens, including bacteria, viruses, fungi and parasites can evolve to be resistant to antimicrobial drugs due to gene mutations over time. Excessive and inappropriate use of antimicrobial medicines on humans and animals, and poor infection control practices, are both speeding up the evolution of resistant strains of microbes and transforming AMR into a worldwide public health threat. A subset of multidrug-resistant bacteria in Europe are responsible for about 25 000 of human deaths annually. 73

In addition to the avoidable deaths, this also translates into extra healthcare costs and productivity losses of at least EUR 1.5 billion each year. In 2007, infections caused by antibiotic-resistant bacteria resulted in approximately 2.5 million extra hospital days, which translated into EUR 900 million hospital costs. According to a report commissioned by the UK Government in collaboration with the Wellcome Trust, 700 000 people die of resistant infections every year. 74

In order to tackle Antimicrobial Resistance, the EU employed a "One Health" approach and also initiated coordination efforts between countries and international organisations. In 2011 the

Commission adopted an action plan against the rising threats of Antimicrobial Resistance. 75 Through

its research framework programmes (e.g. FP7, Horizon 2020) the Commission contributed to several of these areas by funding research activities in the fields related to antimicrobial resistance.

Research projects directly or indirectly related to Antimicrobial Resistance were conducted under different themes, including Health, Nanosciences, Nanotechnologies, Materials & New Production Technologies (NMP), Knowledge Based Bioeconomy (KBBE), Information and communication technologies (ICT) and others.

73 EMEA and ECDC Joint Technical Report. The bacterial challenge: time to react. 2009.

74 Tackling drug-resistant infections globally: final report and recommendations. The review on Antimicrobial resistance chaired by Jim O‘Neill. (2016).

75 Communication from the Commission to the European Parliament and the Council - Action plan against the

rising threats from Antimicrobial Resistance. COM (2011) 748 final i.

In order to promote adequate use of antimicrobial drugs, the Commission launched in 2015 a EUR 1 million challenge prize to develop a rapid diagnostic test for upper respiratory tract infections that can be safely treated without antibiotics. The prize was awarded to MINICARE HNL for a finger prick test that can diagnose in less than ten minutes a bacterial infection and identify if a patient can be treated safely without antibiotics.

To foster the engagement of industry in antibiotic research, several Antimicrobial Resistance related projects were launched under the Innovative Medicines Initiative. The Innovative Medicines Initiative was launched in 2008 and is currently one of the largest public-private partnership between the EU and the European Federation of Pharmaceutical Industries and Associations. Overall, the EU has contributed more than EUR 1 billion towards combating Antimicrobial Resistance over the years.

Stakeholder box: What do stakeholders say?

Stakeholders widely agree that Horizon 2020 has strong EU added value. Stakeholders find

that Horizon 2020 has higher added value than other programmes and that a possible

discontinuation of the programme would have strong negative impacts, which would extend

far beyond a simple reduction of R&I funding for their organisations. The cost of

discontinuation of the Programme was estimated to be in the order of over EUR 400 billion lost by 2030.

• 12. 

  C ONCLUSIONS

The results of this interim evaluation will help improve the implementation of Horizon 2020 in its last Work Programme 2018 – 2020, provide input into the report of the High Level Expert Group on maximizing the impact of EU Research and Innovation programmes, and inform the design of future Framework Programmes. This section summarizes the key findings and outlines issues for future consideration. A full list of lessons learnt for both the remainder of Horizon 2020 and the successor Framework Programme can be found in the Indepth

Interim Evaluation of Horizon 2020. 76

12.1. Strengths

The evidence presented in the Horizon 2020 interim evaluation has demonstrated that, overall, Horizon 2020 is an attractive and well performing programme. It has so far attracted more than 100,000 applications, representing a huge increase in the annual number of applications compared to FP7. It involves top level participants from the higher education, research and private sectors; from a wide range of disciplines and thematic fields; and from over 130 countries. 52% of participants are newcomers. Industrial participation has increased compared to FP7. 23.9% of the budget for industrial and enabling technologies and societal challenges goes to SMEs, far exceeding the target. Stakeholders are generally very satisfied with the programme.

76 See section 12 thereof.

Horizon 2020's objectives and rationale for intervention remain highly relevant and have been validated by, and are fully consistent with, recent EU and global priorities, such as the Sustainable Development Goals. The programme has also proven that it is flexible and can respond to emergencies (e.g. Ebola, Zika) and emerging needs.

Horizon 2020 is on track to be cost-efficient, achieving a very low administrative overhead, thanks to the extensive externalisation of programme implementation, the creation of a Common Support Centre, and the large-scale simplification of the rules for participation, in particular the funding model, which has reduced time to grant and lowered costs for participants, to the satisfaction of stakeholders and without reducing the level of co-funding by beneficiaries.

In terms of effectiveness, through its focus on scientific, economic and societal impacts, Horizon 2020 is on track to contribute to the creation of jobs and growth and the achievement of the priorities of the Juncker Commission. It strengthens the science base by involving the EU's and world's best research institutions and researchers; by training large numbers of EU- based researchers; by producing large numbers of world class open access scientific publications and data; by producing scientific breakthroughs; and by building cross-sectoral, inter-disciplinary, intra- and extra-European research and innovation networks.

It fosters industrial leadership by successfully involving the private sector and SMEs; by creating networks between the business sector, universities and research institutions; by providing businesses and SMEs with risk finance to carry out their research and innovation projects; by investing in demand-driven innovation; by producing high quality, commercially valuable patents and other intellectual property rights; by generating proofs of concept and demonstrators and supporting the deployment of innovation solutions; by producing new knowledge, strengthening capabilities, and generating a wide range of innovation outputs including new technologies, products and services; and by increasing the competitiveness of beneficiaries. It addresses major societal challenges by producing publications, patents, prototypes, products, process and methods. It is successful in spreading excellence and widening participation through dedicated instruments and as a cross-cutting issue throughout the programme. It achieves encouraging results in terms of gender equality and the integration of the social sciences and humanities.

Compared to FP7, Horizon 2020 is an internally more coherent programme. Synergies with other programmes and instruments are being strengthened.

Horizon 2020 has clear European added value in terms of speed, scale and scope and a strong additionality: 83% of funded projects would not have gone ahead without EU funding.

12.2. Challenges

At the same time, it is clear, however, that in the last three years of Horizon 2020, as well as in the next Framework Programme, efforts will need to be made to address a number of challenges that have been identified. The most important ones would appear to be the following:

• 1. 

  Horizon 2020 suffers from underfunding, resulting in large-scale oversubscription, much larger than in FP7, which constitutes an enormous waste of resources for applicants and of good proposals for Europe.

• 2. 

  While Horizon 2020 demonstrates potential in terms of supporting breakthrough, marketcreating innovation, such support needs to be strengthened substantially.

• 3. 

  There is a need for greater outreach to civil society to better explain results and impacts and the contribution that research and innovation can make to tackling societal challenges, and to involve them better in the programme co-design (agenda-setting) and its implementation (co-creation).

• 4. 

  While great efforts have already been made to increase the synergies between Horizon 2020 and other EU programmes (notably European Structural and Investment Funds), these can be strengthened further, particularly in view of R&I capacity building for lower performing regions.

• 5. 

  While Horizon 2020 has achieved a broad international outreach, international cooperation needs to be intensified and more efforts are needed to ensure that the programme fully delivers on its target for sustainable development.

• 6. 

  While compared to FP7, great progress has been made in terms of simplification, simplification is a continuing endeavour, which requires constantly identifying new candidate areas for improvements; at the same time, there is scope for rationalising the Horizon 2020 funding landscape.

• 7. 

  While Horizon 2020 has made great progress in terms of making openly accessible to the wider scientific community and public the scientific publications and data it generates, more can be done in this respect.

To conclude, so far Horizon 2020 is an attractive and well-performing programme, highly relevant for stakeholders. It goes in the right direction delivering value for money and is on course to meet its knowledge-creating objectives. Main areas for improvement are oversubscription; stimulating breakthrough, market-creating innovation, notably by SMEs, and scaling up to EU level; further alignment to policy priorities; and bringing results to citizens and involving them more.

Met de EU Monitor volgt u alle Europese dossiers die voor u van belang zijn en bent u op de hoogte van alles wat er speelt in die dossiers. Helaas kunnen wij geen nieuwe gebruikers aansluiten, deze dienst zal over enige tijd de werkzaamheden staken.

De EU Monitor is ook beschikbaar in het Engels.