COMMISSION STAFF WORKING DOCUMENT INTERIM EVALUATION of HORIZON 2020 ANNEX 1 - Hoofdinhoud

Council of the European Union

Brussels, 31 May 2017 (OR. en)

9787/17 ADD 1

RECH 211 COMPET 454 IND 142 MI 458 EDUC 272 TELECOM 148 ENER 255 ENV 551 REGIO 65 AGRI 289 TRANS 227 SAN 224

COVER NOTE

From: Secretary-General of the European Commission, signed by Mr Jordi AYET PUIGARNAU, Director

date of receipt: 29 May 2017

To: Mr Jeppe TRANHOLM-MIKKELSEN, Secretary-General of the Council of the European Union

No. Cion doc.: SWD(2017) 221 final - PART 2/16

Subject: COMMISSION STAFF WORKING DOCUMENT INTERIM EVALUATION of HORIZON 2020 ANNEX 1

Delegations will find attached document SWD(2017) 221 final - PART 2/16.

Encl.: SWD(2017) 221 final - PART 2/16

EUROPEAN COMMISSION

Brussels, 29.5.2017 SWD(2017) 221 final

PART 2/16

COMMISSION STAFF WORKING DOCUMENT

INTERIM EVALUATION of

HORIZON 2020

ANNEX 1

{SWD(2017) 220 final} {SWD(2017) 222 final}

• A. 

  P ROCEDURAL I NFORMATION

Lead DG: Directorate General Research and Innovation (RTD)

Agenda Planning number: 2015/RTD/005 Interim evaluation of Horizon 2020.

The requirement for the interim evaluation of Horizon 2020 derives from Article 32 of Regulation 1291/2013/EC i establishing Horizon 2020. This stipulates that "by 31 December 2017, and taking into account the ex-post evaluation of the Seventh Framework Programme to be completed by 31 December 2015 and the review of the EIT, the Commission shall carry out, with the assistance of independent experts, selected on the basis of a transparent process, an interim evaluation of Horizon 2020, its specific programme, including the European Research Council (ERC), and the activities of the EIT".

The interim evaluation of Horizon 2020 started in 2016 and has been guided by Terms of Reference adopted by the Commission after a vote by the Member States’ Programme

Committee 1 . An evaluation roadmap, summarising the design, purpose and scope of the Horizon 2020 interim evaluation, was published in May 2016 2 .

An Inter-Service Group (ISG) 3 gathering representatives of different Directorates-General (DG)

of the Commission was set up in early 2016 and held 7 meetings prior to submission of the Staff Working Document to the Regulatory Scrutiny Board in March 2017 (14 April 2016, 12 May 2016, 13 June 2016, 13 July 2016, 20 September 2016, 27 October 2016, 23 February 2017). A series of internal seminars were also organised between December 2016 and February 2017, to which all ISG members were invited, at which the emerging interim evaluation results were presented and discussed.

The interim evaluation was coordinated by the Evaluation Unit of the Commission's Directorate General for Research & Innovation (DG RTD) with inputs from several Commission services that, in turn, contracted studies or steered groups of independent experts. A cross-DG Working Group was established and held 13 meetings between March 2015 and November 2016. The evaluation is based on a wide range of sources comprising internal assessments by Commission services as well as external expert group reports, horizontal and thematic evaluation studies, the

results of the ex-post evaluation of 7 th European Research Framework Programme (FP7) and the

review of the European Institute of Innovation and Technology. The sources are systematically described and identified throughout the Staff Working Document.

A public stakeholder consultation on the interim evaluation of Horizon 2020 was launched on 20 October 2016 and closed on 15 January 2017. On 28 April 2017 a conference was organised by DG RTD in cooperation with the European Economic and Social Committee (EESC) to present the results of this consultation.

1 C(2016)5546.

2 See: http://ec.europa.eu/smart-regulation/roadmaps/docs/2015_rtd_005_evaluation_ie_horizon_2020_en.pdf

3 The ISG for the Horizon 2020 interim evaluation consisted of representatives from the following Directorates-General of the

European Commission: AGRI, BUDG, CLIMA, CNECT, EAC, ENV, ECFIN, ENER, GROW, HOME, JRC, MOVE, REGIO, RTD, SG.

In accordance with the feedback received from the Regulatory Scrutiny Board on 31 March 2017, the Staff Working Document has been revised as presented in Figure 1 These revisions were endorsed by the Inter Service Group during the meeting of 7 April 2017.

Figure 1 Modifications to the draft Staff Working Document based on comments received from the Regulatory Scrutiny Board

Comments from the Regulatory Scrutiny Actions taken for the Staff Working Document Board

• (1) 

  Expected vs actual results - key evaluation  The’ background to the initiative’ section has been questions completed to present the key evolutions from FP7

The report does not make enough use of the (novelties and continuity), and key features and available evidence to benchmark and compare expectations of Horizon 2020 compared to FP7.

results with what was anticipated.  The objectives pursued through Horizon 2020 have

The introduction should elaborate on the been clarified and the detailed intervention logic presentation of the programme. It should used for the interim evaluation has been included highlight the differences with its predecessor, (including the different types of expected outputs, FP7. On that basis, the report should clearly set results and impacts around which the effectiveness out the initial expectations of the programme. It analysis is structured).

should link these to a strengthened analysis of  Key questions for each evaluation criteria have the results obtained so far and the reasons for been included as well as a short explanation at the possible deviations. A clear intervention logic beginning of each section on the purpose of the should describe how the programme aims to analysis performed. achieve its intended effects, what were the

projections made and how those compare with  Under the effectiveness section, the structure of the the results achieved so far. Since this is an analysis of the progress towards impacts has been interim evaluation, the report should focus on further explained according to the channels used the key channels of the intervention logic which under Horizon 2020 for the generation of impacts.

can be checked at this stage. It should formulate  Comparisons with FP7 have been included where key questions around the five evaluation criteria. relevant and possible (availability of comparable The report should select the relevant data that data). A benchmarking table of the main

answer these questions and explain how the implementation data for FP7 and Horizon 2020 has

programme is performing. This is for instance been added. the case for the efficiency criteria where the

evaluation could benchmark the efficiency gains  The baseline scenario and expectations from against the forecasts of the cost-benefits study Horizon 2020 based on the Impact Assessment on the externalisation of the management of EU have been clarified for each evaluation criteria.

funds. In terms of coherence, the report should  In the efficiency assessment, comparisons with the provide a critical picture of how Horizon 2020 forecasts of the cost-benefits analysis of the and some of its parts (such as the financial externalisation of the programme management instruments) fit with other programmes in a have been added.

complementary manner.  The coherence assessment has been strengthened

to provide a more critical and comprehensive picture of the complementarity of the instruments

of Horizon 2020 with other instruments.

• (2) 

  From key questions to key conclusions  The conclusion section has been restructured to

The conclusions do not always clearly stem from present more clearly the key findings and areas for

the analysis. improvement, with distinctions made between improvements needed on the short term (e.g 2017-

The report should more systematically qualify 2020) and in the longer term (e.g next Framework its key findings in terms of positive and negative Programme). These include suggestions for further developments. It would clarify what issues will simplification.

Comments from the Regulatory Scrutiny Actions taken for the Staff Working Document Board

be addressed at different stages. It should put  The expectations from the programme have been more emphasis on possible areas for added under each evaluation criteria accompanied improvements in the remaining three years of by an overview box on the key conclusions from the programme. This would for examples the analysis. include adapting the SME instrument to support the most disruptive innovations, increasing the participation of third countries and reconciling its focus on excellence with capacity building in some countries. The report should also better identify the remaining areas for further simplification. As mentioned under point 1), for each of these, it should start from the intention of the programme, the actual results and suggest possible solutions.

• (3) 

  Prioritisation  The structure of Horizon 2020 has been clarified as

The report does not fully explain how activities well as the way the priorities were defined at time were given priority within and between the three of programme design. The strategic programming

pillars. process has been further explained as well as further references to the thematic assessments

The report should more clearly explain how (Annex Part 3) were introduced - where the specific topics are prioritised under Horizon allocations per topics are discussed in-depth. 2020. In particular the report should clarify

whether the programme properly addresses  Stakeholder views are used for contextualising the current challenges identified by stakeholders. findings. These are based on a public consultation Generally, it should break down stakeholders' questionnaire (app. 3500 respondents) and more views and distinguish between beneficiaries and than 300 position papers received. In addition, other parties. It could supplement those views horizontal studies and thematic assessments used with case studies, illustrating both good surveys of project coordinators, participants/nonpractices and challenges encountered. In participants and interviews of multiple addition, the report should provide further stakeholders as evidence base for their analysis. explanations on the implementation of priorities The wording has been revised to ensure the that are both pillar-specific and cross-cutting differences between the sources of information are (such as excellence or innovation). clear for the reader.

 Projects’ example boxes have been introduced throughout the document as illustrations.

 The text has been clarified in order to stress that the interim evaluation is not ‘pillar-based’ but covers the whole programme according to the objectives sought (e.g. excellence is supported under all pillars and innovation can emerge from ERC grants). Detailed assessments of each programme part are provided in Annex Part 3.

• (4) 

  Synthesis:  The extended summary has been reworked to

The structure and presentation of the report do clearly present the key findings from the interim

not convey a clear overview of key messages. evaluation, the strengths from Horizon 2020 and the remaining challenges to be addressed in the

The extended summary could serve as a basis (near) future as the main Interim Evaluation. In for a more synthetic Staff Working Document. order to ease the reading it follows the same The report should in a balanced manner take structure as the In-Depth Interim Evaluation. stock of the achievements and difficulties encountered so far. This would correspond to the

Comments from the Regulatory Scrutiny Actions taken for the Staff Working Document Board

evaluation's objective to inform the College and feed into the impact assessment for a future research programme. It would also pave the way for future general orientations on issues like innovation, basic research or support to SMEs.

• B. 

  S TAKEHOLDER CONSULTATION RESULTS

A public stakeholder consultation on the interim evaluation of Horizon 2020 was launched on 20 October 2016 and closed on 15 January 2017. The stakeholder consultation results were discussed at a conference organised by the European Commission, DG RTD, with the European Economic and Social Committee on 28 April 2017.This document presents an analysis of the responses received, structured according to the five evaluation criteria: relevance; effectiveness; efficiency and use of resources; coherence; EU added value.

The document concludes with the analysis of 296 position papers also received within the context of the stakeholder consultation.

B.1. Overview of respondents

B.1.1. Who are the respondents?

In total 3483 responses to the online questionnaire were received:

 49% (1721) from individuals;  5% (175) from representatives of "umbrella" organisations of EU interest; and  46% (1587) from representatives of a single institution or a company.

Among different types of organisations, the highest number of responses was submitted by businesses (687 or 20%), of these 65% (443) were SMEs.

Answers come from 69 different countries. However, the majority of the respondents come from EU15 countries with Spain and Italy being the most active. 65 respondents come from third countries.

Table 1 What type of organisation do you represent?

Type of respondent 1. I am responding Total % of respondents

As an individual n/a 1721 49.4%

Academia On behalf of a single institution/company 297 8.5%

On behalf of an “umbrella” organisation of EU interest 27 0.8%

Business On behalf of a single institution/company 664 19.1%

On behalf of an “umbrella” organisation of EU interest 23 0.7%

Non-Governmental On behalf of a single institution/company 88 2.5%

Organisation On behalf of an “umbrella” organisation of EU interest 37 1.1%

Public authority On behalf of a single institution/company 133 2.9%

On behalf of an “umbrella” organisation of EU interest 13 1.6%

Research organisation On behalf of a single institution/company 305 8.8%

On behalf of an “umbrella” organisation of EU interest 20 0.6%

Other On behalf of a single institution/company 100 3.8%

On behalf of an “umbrella” organisation of EU interest 55 0.4%

Total 3483 100.0%

Figure 1 Country of respondents

B.1.2. Which part of the programme have they participated in?

76% (2648) of the respondents received support from different parts of the Horizon 2020 programme. Most commonly, they participated in the Marie Skłodowska-Curie Actions (MSCA, 30% of respondents) or the Leadership in Enabling and Industrial Technologies (LEIT, 22% of respondents) parts of the programme. 30% (790) of the respondents who received support from Horizon 2020 are newcomers to the programme (not having participated in FP7).

Figure 2 Share of respondents that have participated in Horizon 2020 (2648) stating which part the programme they have participated in

Horizon 2020 Pillar Number of respondents Share of respondents

Societal Challenges 2227 36.6%

Excellent Science 2185 35.9%

LEIT 940 15.4%

Other 745 12.2%

B.1.1. Main reasons for not participating

24% (835) of respondents did not receive support from Horizon 2020. Besides not being funded, the main reasons for not participating in Horizon 2020 were:

• 1) 

  Success rates in Horizon 2020 are too low to be worth applying; and

• 2) 

  Limited financial/human resources to prepare a proposal.

Figure 3 Main reasons for not participating to Horizon 2020 (max 3 answers)

Out of the 134 respondents, who listed "Other", 58 were not researchers/ innovators and a few more quoted reasons for not participating that are already listed above ('lack of relevant topicsareas', for example, because topics are too broad and limited resources). 8 indicated they were in the process of applying or were awaiting results after a proposal submission, while 5 were still involved in ongoing FP7 projects that had prevented them from applying to the new programme. A few mentioned conflicts of interest or a desire to maintain an independent view of on the programme, for example consulting firms involved in evaluations of Framework Programmes (FPs).

Some respondents commented on Horizon 2020 requirements that had hampered their participation mostly because they applied from third countries (e.g. Swiss respondents). Among other reasons for not participating were the lack of incentives, lack of awareness of the Horizon 2020 programme, lack of experience in participating in such a programme, and the limited involvement of end-users in FP projects.

B.2. Relevance

B.2.1. Is Horizon 2020 tackling the right issues?

B.2.2. The relevance of Horizon 2020 given the challenges to address

When asked whether Horizon 2020 priorities address the current challenges confronted by the European Union (e.g. migration, terrorism, ageing population), 77% of the consultation respondents agree fully or to a large extent, and 8% judge that it is not the case at all. Academia and research organisations tend to be more positive (86-83% think it does at least to some extent) than business (71% think it does at least to some extent).

Figure 4 Do you think that Horizon 2020 priorities address the current challenges

confronted by the European Union (e.g. migration, terrorism, ageing population)?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=348.3

B.2.3. The relevance of Horizon 2020 to address European objectives

Almost all the consultation respondents agreed to some extent or more that Horizon 2020 is

contributing to support jobs, growth and investments (95%) and to foster the role of the

European Union as a stronger global actor (92%).

The vision of respondents on the contribution of Horizon 2020 to other EU priorities is more

nuanced: 74% agree to some extent or more that Horizon 2020 is contributing to achieving a

deeper and fairer internal market with a strengthened industrial base, 72% to promoting an

Energy Union with a forward-looking climate policy (25% do not share this vision at all, that is

the priority which sees the highest proportion of disagreement), and 66% to helping to create a

Digital Single Market (however 29% of respondents declare that they do not know).

Figure 5 Do you think that Horizon 2020 is contributing to the following priorities of the European Union?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.2.4. Does Horizon 2020 allow adapting to new scientific and socio-economic developments?

While the majority of consultation respondents thought that the programme’s thematic coverage was flexible enough to cope with changing circumstances (77% agree to some extent or more), the rate of full disagreement is however higher than for other statements (12% do not agree at all). In addition, NGOs tended to disagree more than the other categories of respondents did (16% of NGOs do not agree at all).

Figure 6 Do you think that Horizon 2020 thematic coverage is flexible enough to cope with

changing circumstances?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=3483

A high percentage of respondents agreed, to some extent or more, that Horizon 2020

supported the latest developments in research and innovation (93% of agreement rate). The

most positive respondents are business and public authorities.

Figure 7 Do you think that Horizon 2020 priority areas and calls support the latest

developments in research and innovation at the national/European and international level?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

Overall, consultation respondents think that Horizon 2020 is stimulating disruptive and

market-creating innovation but a large share think this is only the case to some extent (37%).

The most positive respondents on this question are SMEs, with 63% thinking that Horizon 2020

is fully or to a large extent stimulating disruptive and market-creating innovation.

Figure 8 Do you think that Horizon 2020 is stimulating disruptive and market-creating

innovation (a new process, product or service that upsets existing business models and

serves new set of customers)?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.2.5. Is Horizon 2020 responding to stakeholder needs?

More than 80% of the consultation respondents agree that the frequency of the calls and

their clarity are either “good” or “very good”.

However, the views regarding the transparency of the process of formulating the Work

Programmes and the ease of finding the right call for proposal differ. Many of the respondents

(67%) had a positive opinion on both these aspects. But some respondents (26%) found that the

transparency of the process of formulating the Work Programmes and the ease of finding the

right call “poor” or “very poor”. 45% of the respondents thought that the inclusion of Social Sciences and Humanities in the calls was “good” or “very good” and many (39%) did not know.

Figure 9 Please rate the following Horizon 2020 implementation aspects – Work

Programme and calls

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=2648

The comments to the open responses elaborate on the observed quantitative results. Some

respondents highlight difficulties in finding and identifying the calls documents. For instance,

they mention a lack of clarity in the calls and the dispersion of documents and information in

various places. They also ask for explanatory documents, as it is judged difficult to find

aggregated and clear information on the programme.

In addition, through an open question, consultation respondents were asked to outline the main

reasons for participating in Horizon 2020. Three main reasons stand out from the analysis,

pointing to the type of needs Horizon 2020 is able to address:

 Respondents highly value the financial support provided by Horizon 2020 (with a few

respondents underlying the long-term and reliable nature of Horizon 2020 funding streams);

 Respondents value the access to new knowledge and know-how, mostly through exchanges

of experiences and skills with partners, that allow them to build new competences and

capacities;

 Respondents underline that participation provides unique opportunities for collaboration

with European or international partners and for contacts with key players that are often

the best in their field. Respondents value the opportunities to strengthen partnerships inside

existing networks as much as the ability to meet new partners or build new networks.

Interdisciplinary work and the opportunity to work with other types of actors (businessacademia-research

organisations- governments- end users) also stand out.

Among the other reasons for participating in Horizon 2020 that are underlined by respondents, it

is worth mentioning the following: products, solutions development and commercialisation

(mainly quoted by businesses); internationalisation, visibility and enhancement of the

participants’ research profile (mainly quoted by academia); the ability to advance global

knowledge and solve societal challenges such as climate change and health; and the ability to

perform or have access to high-profile research. Some business respondents also mention growth opportunities and a better or secured position on markets, as well as the ability to develop

innovation faster.

30% (790) of the respondents who received support from Horizon 2020 are newcomers to

the programme (not having participated in FP7). 87% (2310) of respondents who received

support from Horizon 2020 are cooperating with a new partner(s) in Horizon 2020. The

main reason for collaboration with new partners in Horizon 2020 is to include specific

expertise from another discipline. Out of the 134 respondents, who list "Other reasons", 52

explained that they cannot choose only one main reason and that various or all reasons apply. 28

were approached by other organisations and were not themselves engaged in finding new

partners. Other reasons include: accessing new contacts, larger networks, expertise or

information on local specificities for product development, and benchmarking organisations'

practices (for public authorities).

Figure 10 Why did you look for a new partner (one main reason)?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=2310

When asked whether the different forms of funding provided through Horizon 2020 are relevant

to their needs or not, 76% of consultation respondents agree that grants for collaborative

projects are “fully” or “largely” relevant to their needs, while 49% do so for grants for single

beneficiary projects. Grants are therefore considered by consultation respondents to be the most

relevant forms of funding provided through Horizon 2020, followed by co-funding actions,

prizes, financial instruments and public procurement.

Compared to other respondents, SMEs value more the financial instruments and the grants

for single beneficiary projects. However they seem to be less aware of prizes than other types

of stakeholders.

It is also worth noting that some 8% of the respondents who did not participate in Horizon

2020 underline that they lacked an adequate type of financial support for their work and

15% mention that the programme lacked a relevant area/ topic for their needs (see Figure

2). This therefore suggests that despite increased interest from newcomers, there is still room for

attracting more participants.

Figure 11 Are the forms of funding provided through Horizon 2020 relevant to your needs?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

Additional comments provided by survey respondents in the open boxes corroborated these

results. Collaborative grants and the European Research Council stand out as being particularly

relevant to respondents. Some respondents specify that they find grants more relevant than

financial instruments (this applies to research organisations and academia as well as to business

respondents). Some respondents indicate that Horizon 2020 is too costly and the process is too

slow and complex to efficiently meet their needs.

57% of the respondents find the balance between small and large projects in calls for

proposals “good” or “very good”, but 24% of them find it "poor" or "very poor" and 19%

of respondents do not know. In their open comments, consultation respondents ask for more

opportunities for small projects (although some respondents are in favour of more support for

large-scale demonstrators), more prescriptive calls (to decrease the number of applicants); and

more funding opportunities for SMEs.

Figure 12 Please rate the balance between small and large indicative project sizes in the

calls for proposals

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=2648

A high percentage of respondents agree, to some extent or more, that Horizon 2020

addresses the main citizens’ needs (86% agreement rate), however 37% agree only to some

extent. The least positive respondents are NGOs.

Figure 13 Do you think that Horizon 2020 addresses the main citizens' needs in terms of

research and innovation?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

Among the issues listed in the consultation as needed to further maximize the socio-economic

impact of the EU framework programme for research and innovation, four items stand out (i.e.

meaning that more than 30% of respondents strongly agree): i) more room for bottom-up

proposals; ii) more focus on the support for the exploitation of research results; iii) better

access to the programme for newcomers and iv) increased focus on fundamental research.

Figure 14 To what extent do you agree that the following issues are needed to further

maximize the socio-economic impact of the EU framework programme for research and

innovation?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

Academia strongly agree with the statement that suggested increasing the focus on bottom-up

research and fundamental research (53% of the total number of academia respondents “strongly agreed”), whereas 48% of business respondents “strongly agree” with an increased focus on

support to closer-to-market activities, 38% with an increased focus on demonstration and 43%

with an increased focus on supporting the exploitation of research results. 40% of research

organisations also “strongly agree” that more needs to be done with respect to the exploitation of

research results.

Figure 15 To what extent do you agree that the following issues are needed to further

maximize the socio-economic impact of the EU framework programme for research and

innovation? Specific issues

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

The consultation’s respondents were asked to share a short, telegraphic testimonial on Horizon 2020. The results were analysed using a word cloud. The most common words used by stakeholders to express what Horizon 2020 means to them are ‘research’, ‘innovation’, ‘funding, ‘opportunity’, ‘collaboration’, ‘new’, ‘international’, ‘cooperation’ (see below).

Figure 16 Please share with us a short, telegraphic testimonial. What does Horizon 2020 mean to you? What is its main feature?

Wordle®, Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=1704

B.2.6. Key points / Areas for improvement

To conclude, the respondents to the stakeholder consultation generally:

 Agree that Horizon 2020 priorities address the current challenges confronted by the European

Union and are relevant for achieving European objectives (e.g. supporting jobs, growth and investments and fostering the role of the European Union as a stronger global actor).

 Agree that Horizon 2020 supports the latest developments in research and that the

programme’s thematic coverage is flexible enough to cope with changing circumstances.

 Participate in Horizon 2020 mainly to access funding, knowledge and expertise, and to

collaborate with European or international partners.

 See grants for collaborative projects as the most relevant form of funding for their needs,

compared to financial instruments and public procurement.

 Referred to the complexity and length of the funding process.

B.3. Effectiveness

B.3.1. Progress towards achieving Horizon 2020's objectives

Stakeholders were asked about the progress of Horizon 2020 in achieving its objectives. The

figure below provides an overview of the results for each of the eight objectives. A more indepth

analysis for each objective is presented in the subsections underneath.

Figure 17 Do you think that Horizon 2020 is helping to:

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=3483

B.3.1.1. EU world-class excellence in science

94% (3279) of the public consultation respondents agree, at least to some extent or more,

that Horizon 2020 helps to foster excellent science. The contribution of the programme to this

objective i s assessed very positively, since 36% (1242) of the respondents agree “fully” with this

statement, which is the highest result scored by the statements that were proposed in the

questionnaire.

Figure 18 Do you think that Horizon 2020 is helping to foster excellent science?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.3.1.2. Fostering European Industrial Leadership

84% (2927) of the public consultation respondents agree, to some extent or more, that Horizon

2020 helps foster European industrial partnerships. Businesses agree more with this statement

(94% of agreement rate) when compared to academia or research organisations (83%).

The contribution of the programme to this objective is assessed positively by a large majority of

respondents, but a comparatively low number of respondents (17%) agree “fully" with this

statement. This is less than the number of respondents who do so for the contribution of the

programme to fostering excellence in science. Also a comparatively large share of respondents

(12%) "don't know" about the Horizon 2020 contribution to this objective..

Figure 19 Do you think that Horizon 2020 is helping to boost industrial leadership?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.3.1.3. Spreading Excellence and Widening Participation

65% of the public consultation respondents agree fully or to a large extent that Horizon

2020 helps spread excellence and widen participation (and 91% agreed at least to some

extent) in research and innovation across Europe. The agreement level is similar for EU15

and EU13 respondents, but respondents from third countries (72%) and associated countries

(67%) are even more positive. The most positive types of stakeholders are SMEs (73% think it

does fully or to large extent) and individuals (63.4%). NGOs are slightly less positive.

Figure 20 Do you think that Horizon 2020 is helping to spread excellence and widen

participation?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.3.1.4. Generating Science with and for Society

70% of the public consultation respondents agree fully or to a large extent that Horizon 2020 is

helping to support science with and for society (92% agreed at least to some extent). 3.3% do not

agree at all. The most positive respondents are businesses and research organisations, whereas

the least positive are NGO and public authorities.

In addition, 87% (2310) of the public consultation respondents who were funded by Horizon

2020 cooperated with new partners thanks to Horizon 2020 projects and 11% of them did so in

order to involve potential users of the results.

Figure 21 Do you think that Horizon 2020 is helping to support science with and for

society?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=3483

B.3.1.5. Generating Science for Policy

87% (3018) of the public consultation respondents agreed, to some extent or more, that Horizon 2020 helps developing and implementing EU policies, yet a comparatively low number of respondents (18%) agreed “fully" with this statement, which is less than the number of respondents who did so for the contribution of the programme to support science with and for society.

B.3.1.6. Integrating the knowledge triangle of higher education, science, and education

96% (3279) of the public consultation respondents agree, to some extent or more, that Horizon 2020 helps building a society and an economy based on knowledge and innovation.

87% (2310) of the respondents who were funded by Horizon 2020 cooperated with new partners thanks to Horizon 2020 projects, and 1037 of them (45%) declare they have done so to include specific expertise from another discipline. This result underlines the importance of interdisciplinary work.

B.3.1.7. Addressing the Major Societal Challenges

The results of the consultation suggest that Horizon 2020’s contribution to addressing the major societal challenges is assessed more negatively by respondents than its contribution to the other objectives.

Horizon 2020 scored higher on its contribution to fostering a greater understanding of Europe, providing solutions and supporting inclusive, innovative and reflective European societies (Societal Challenge 6), with 79% of respondents agreeing at least to some extent, and on its capacity to improve the lifelong health and well-being of all (Societal Challenge 1) (78% agree to some extent, but also 18% think the programme is not helping at all). For all the other challenges, around 30% of the respondents do not know, which is not surprising given the early stage of the programme's implementation.

24% of respondents think Horizon 2020 is not helping at all to address the challenge of securing sufficient supplies of safe, healthy and high quality food and other bio-based products (Societal Challenge 2). A comparatively lower number of respondents agreed “fully” with the statements that were provided and more respondents expressed their disagreement.

Figure 22 Do you think that Horizon 2020 is helping to address major societal challenges?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.3.2. Contribution of Horizon 2020 to Growth, Jobs and Investments

60% of the survey respondents agree "fully" or "to large extent" that Horizon 2020 is

supporting jobs, growth and investments (95% of the respondents think so at least to some

extent). Only 1.7% entirely disagreed.

Figure 23 Do you think that Horizon 2020 is contributing to the following priorities of the

European Union? Supporting jobs, growth and investment

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.3.3. Contribution of Horizon 2020 to the Europe 2020 Strategy

62% of the survey respondents think that Horizon 2020 is helping fully or to a large extent to

‘implement the Europe 2020 strategy, the EU’s strategy for jobs and smart, sustainable and inclusive growth’ (90% of the respondents think so at least to some extent). Only 2% do not

share this view at all. In addition, 72% of the respondents think that Horizon 2020 is helping

fully or to a large extent to build a society and an economy based on knowledge and innovation.

For both options, the least positive respondents are umbrella organisations representing research

organisations and NGOs.

Figure 24 Do you think that Horizon 2020 is helping to implement the “Europe 2020” strategy, the EU’s strateagy for jobs and smart, sustainable and inclusive growth?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=3483

Figure 25 Do you think that Horizon 2020 is helping to build a society and an economy

based on knowledge and innovation?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.3.4. Contribution of Horizon 2020 to the achievement and functioning of the

European Research Area

75% of the respondents to the stakeholder consultation think that Horizon 2020 is fully or to a

large extent ‘helping to support the development of the European Research Area, a unified area open to the world, in which scientific knowledge, technology and researchers circulate freely’

(94% think so at least to some extent). Only 2.2% do not share this view at all. The least positive respondents are umbrella organisations representing businesses and NGOs.

Figure 26 Do you think that Horizon 2020 is helping to support the development of the

European Research Area, a unified area open to the world, in which scientific knowledge,

technology and researchers circulate freely?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.3.5. Key points / Areas for improvement

To conclude, the respondents to the stakeholder consultation generally:

 Agree that Horizon 2020 contributes to achieving the Europe 2020 strategy and the European

Research Area.

 Agree that the programme is effective in achieving its own objectives, for instance in

fostering excellent science.

 Agree to a lesser extent that Horizon 2020 is boosting European industrial leadership,

compared to other objectives.

 Agree that the programme is having at least some impacts, but the rates of disagreement

increase when asked about the programme’s contribution to addressing a set of societal challenges

B.4. Efficiency and use of resources

Satisfaction with the programme is high among respondents: 78% (2732) state that they are very satisfied or satisfied with the programme. Comparatively, a higher number of NGOs are dissatisfied with the programme (20%) and a higher number of businesses (25%) are very satisfied with the programme. EU13 countries express a higher level of dissatisfaction (18%), while 25% of third-country respondents are very satisfied with the programme. The satisfaction rate reaches 88% among the participants in Horizon 2020, but decreases to 49% for the respondents who have not participated in the programme.

Figure 27 Overall are you so far satisfied with Horizon 2020?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

B.4.1. Programme's management and use of resources

B.4.1.1. New management modes

For 73% (1927) of the respondents, the support provided by the EC services (including agencies) during grant preparation and implementation was either “very good” or “good”.

Figure 28 Please rate Horizon 2020 implementation aspects for support provided by the EC services (including agencies) during grant preparation and implementation

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=2648

The analysis of the open responses also reveals some testimonials of good working relationships with European Commission project officers. However some of the respondents who described this relationship underline the delays they experience in receiving answers to their requests from the project officers, while a few others ask for more personalised support from the agencies.

Additionally, a few respondents specifically comment on "New management modes" in their open responses to questions on the efficiency and implementation of the programme.

B.4.1.2. Use of resources

89% of respondents “strongly agreed” or “agreed” that an increased budget was needed for financing research and innovation at EU level.

Figure 29 To what extent do you agree that increased budget for financing research and innovation at EU level is needed to further maximize the socio-economic impact of the EU framework programme for research and innovation?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

For 21% of respondents the frequency of use of a two-stage procedure in evaluating proposals is

“poor” or “very poor”.

Furthermore, in the open comments, some stakeholders call for a more competitive selection

process at the first stage of the two-stage application process. Given the competitiveness of

Horizon 2020 funding, they feel high-quality projects are not being funded, and this could reduce

the n umber of proposals submitted at second stage and mitigate the risk of “wasting” time in

developing proposals. In this respect, a large number of open comments deal with

oversubscription and the low success rate caused by the high number of (good) proposals given

the limited amount of funding. This is further illustrated by the fact that, out of the 835

respondents who did not participate in Horizon 2020, 194 explain that the main reason is the

success rates that are too low to be worth applying. This item is the most common explanation

for non-participation for respondents who have never applied for Horizon 2020 funding (see

Figure 2).

Figure 30 Please rate the following Horizon 2020 implementation aspects: frequency of use

of 2-stage procedures in evaluating proposals

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=2648

B.4.2. Programme's implementation

B.4.2.1. The impact of simplification and the new funding model

Out of the 835 respondents who did not participate in Horizon 2020, 106 explain that the main reason is that the Horizon 2020 project implementation rules are cumbersome.

The analysis of open answers confirm this result. While some respondents (participants as well as non-participants) acknowledge that progress has been made, many mention that further simplification is needed. Many note that more could be done in terms of simplification, for instance regarding cost reimbursement and further simplification of the process and acceptance of organisations’ accounting practices. However, 65% (1732) of the survey respondents to a closed question note that the acceptance of organisations’ accounting practices in the programme is “good” or “very good” and 18% (475) view it as “poor” or “very poor”.

Some respondents also note that the rules are different from one call to the other and recommend more standardisation. Others point to a proliferation of funding and instruments that hamper their ability to grasp the broader picture and apply for the funding that is most tailored to their needs.

Some additional ideas that were identified during the analysis of the responses to open questions concern the need to diminish the administrative burden experienced by participants of the process and to promote more flexibility. For example, by allowing for some adjustments during the implementation of the projects (e.g. one respondent noted it is not possible to work with a third party who was not a formal project partner at the project start).

The respondents also elaborate on the imbalance between the need for control and the importance of trust. Some argue that the European Commission needs to focus on the quality of project outcomes rather than paperwork, while others propose that past participations in the FPs or a track record at the national level should be used as a proof that participants can be trusted. Many comments deal with the high amount of time spent on reporting. Despite these comments, a majority of respondents find the balance between control and trust of beneficiaries (71.5%, 1894) and the mechanisms for reporting and monitoring (79%, 2091) “good” or “very good”.

B.4.2.2. Mobilisation of stakeholders

This topic was covered through consultation questions that relate to the efficiency, transparency, clarity and flexibility of the processes to attract participants.

More than 80% of the respondents agree that the time taken to sign a grant agreement and to evaluate the proposal is either “good” or “very good”. 21% to 22% find that the communication activities to attract applicants are “poor” or “very poor”. Furthermore, 62% (1647) of the respondents assess the quality of the feedback from the evaluations as “good” or “very good”, while 34% (905) judge it is “poor” or “very poor” (which is the highest score reached by the “poor” and “very poor” categories compared to the other items related to the implementation aspects of Horizon 2020 on which respondents were asked questions).

Figure 31 Please rate the following Horizon 2020 implementation aspects

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=2648

In their feedback to open questions, many respondents asked for more transparency and an

improved quality in the feedback they received. Some respondents complained that not enough

details were provided, that the quality of the feedback varied greatly from one evaluation panel

to the other, and that discordant views could be provided to the participant. The selection of

experts for proposal reviews was also questioned by a few; with some participants stressing that

expertise in the field was not always available. Some mentioned that evaluations should not only

take place remotely.

B.4.3. Geographical dimension

The geographical dimension was covered in the survey questions that related to the non EU-

countries' and nonassociated countries’ participation. The figure below suggests that the

majority of respondents from these countries were rather satisfied with the communication on

Horizon 2020 in their countries, with 69% (42) having “agreed strongly” or “agreed” that

communication activities helped them find out about the programme and that it was easy to find

calls which were relevant to their area (strong agreement or agreement of 62% (38) of the

respondents). 45% (27) felt that it was easy to find calls that encourage the participation of non

EU and non-associated country partners and 43.4% (26) “disagreed” or “disagreed strongly”

with this statement.

Figure 32 If your organisation is established in a non-EU, non-associated country, to what

extent do you agree with the following statements?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=61

In their open comments on the relevance of and the issues at stake for the programme, some

respondents from third countries as well as respondents from EU-countries explicitly referred to

the need to increase the possibility for third countries to take part in Horizon 2020.

B.4.4. Cost-benefit analysis

The consultation’s respondents were asked about the costs of participating in Horizon 2020

compared to previous or other international programmes. The interpretation of the results has to

take into account the high percentage of respondents (over 30%) who declared they could not

respond due to a lack of knowledge of previous or other programmes. This set aside, the results

suggest that slightly more respondents think that the costs of participating in Horizon 2020

compared to FP7 had decreased rather than increased with the simplification measures that have

been implemented by the European Commission. 20% (521) of the respondents shared the view

that the costs of participating in Horizon 2020 are lower than in the previous FP7, 14% (364) felt

they are higher and 36% (950) felt they are similar. A more detailed analysis indicates that

comparatively business have a slightly better opinion of the costs of Horizon 2020 than research

organisations. While 20% of research organisations found the costs of Horizon 2020 higher than

FP7, only 10% of the business did so (and 7% of the SME respondents).

Figure 33 Level of costs of participating in Horizon 2020 compared to the 7 th Framework

Programme as a result of the simplification measures

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

Keeping in mind that a high percentage of respondents (34%) declare they could not respond due

to a lack of knowledge of other programmes, the majority of those who responded assess the

costs of participating in Horizon 2020 as similar to other international research and innovation

programmes (see Figure 33). Going into further detail, 25% of research organisations say that the

costs of Horizon 2020 are higher than for other international programmes, while only 15% of

business (and 14% of the SMEs) do so. More specifically there are slightly more SMEs that,

overall, find that the costs of participating to Horizon 2020 are lower than other similar

international research and innovation programmes (19%) than SMEs judging these costs higher

(17%). 21% of the respondents from associated countries share the view that Horizon 2020 is

more costly than international programmes. Overall, the results seem to suggest that EU13

respondents and newcomers do not feel that the costs are higher compared to other respondents.

They even seem to have a more positive opinion about the costs of the programme (e.g. 18% of

EU15 respondents and only 11% of EU13 respondents find the costs of Horizon 2020 higher

than the costs of other international programmes).

Figure 34 Level of costs of participating in Horizon 2020 compared to those of other similar

international research and innovation programmes

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=2648

B.4.5. Key points / Areas for improvement

To conclude, the respondents to the stakeholder consultation generally:

 Are satisfied with the programme. The support provided by the European Commission is

appreciated, although there is some criticism of the externalisation of grant management to

executive agencies. Some stakeholders reported delays in getting in touch with project

officers and asked for more personalised support and an improvement in the quality of

evaluation feedback.

 Agree that an increased budget was needed for financing research and innovation at EU

level.

 Assess the cost of participation to be lower than in previous programmes but noted there is

room for further decreasing the costs. Simplification measures are welcomed (processes are

efficient) but the administrative burden is still high for some respondents. Further

simplification (in terms of cost reimbursement for instance) is an area for improvement.

 Note that there is room for improvement in the standardisation between the different calls

and the information and communication activities to attract applicants (dispersion of

information, lack of explanatory documents).

 Prefer a ‘real’ two-stage application processes (in which proposals would be thoroughly

selected at the first stage) to address the oversubscription issue.

B.5. Coherence

B.5.1. Internal coherence within the Framework Programme

B.5.1.1. Coherence between the implemented actions

71% of the respondents agree that combining different forms of support for research and

innovation into one single programme better address stakeholder needs than having

separate programmes.

B.5.1.2. Coherence between Horizon 2020 intervention areas

76% of the respondents agree that the increased use of calls for cross cutting activities and

interdisciplinary is a positive feature in the programme (see Figure 35). 66% find that the

different parts of Horizon 2020 complement each other but only 46% agree that there is more

coherence and synergies in Horizon 2020 than in the FP7 (a large share of stakeholders (44%)

don't know). More academia and research organisations subscribe to these last two statements

than businesses and NGOs.

Figure 35 To what extent do you agree with the following statements regarding the internal

structure of Horizon 2020?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of Horizon 2020, October 2016-January 2017, N=3483

The majority of respondents agree that to increase the socio-economic impact of EU Framework

Programmes for research and innovation, there is a need for more cross-cutting calls (16.8%

disagree), more focus on capacity-building activities for R&I (15% disagree) and increased

coordination/synergy with other programmes (23% disagree). More than 30% of NGOs and

public authorities are in favour of more cross-cutting calls. However only 23% of research

organisations, 22% of academia and 16% of business feel this is needed.

Figure 36 To what extent do you agree that the following issues are needed to further

maximize the socio-economic impact of the EU framework programme for research and

innovation?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=3483

B.5.2. Coherence with other EU initiatives

The results of the consultation survey are difficult to interpret as most respondents feel they were

unable to answer because of their lack of familiarity with other initiatives, which explains the

high number of respondents having ticked the “I don’t know” box.

B.5.2.1. The European Structural and Investment Funds

Of the respondents who were able to provide an opinion, 15.6% find that Horizon 2020 and the

European Structural and Investment Funds complement each other and 12% judge that they work

in synergy (despite the existence of overlaps for 2.7% of the respondents) (cf. Figure 37).

B.5.2.2. The European Fund for Strategic Investments (EFSI)

For respondents who were able to provide an opinion, 10.4% find that Horizon 2020 and the

European Fund for Strategic Investments complement each other and 6.7% judge that they work

in synergy (despite the existence of overlaps for 1.8% of the respondents) (see Figure 37 below)..

B.5.2.3. Other EU initiatives

Among other programmes, Erasmus+ is assessed as the most complementary to Horizon 2020

(28% of respondents assess positively the complementarity between the two programmes) and

8% of respondents judge that they work in synergy (despite the existence of overlaps for 2% of

the respondents).

Figure 37 The European Commission implements several funding programmes.

How would you describe the linkages between Horizon 2020 and the following

programmes?

Source: Replies to stakeholder consultation questionnaire launched in the framework of the Interim Evaluation of

Horizon 2020, October 2016-January 2017, N=3483

Additional comments provided by the respondents suggest that the funding architecture is seen

as too complex and prevents organisations from identifying the calls and instruments that are

best fitted to their needs. Promoting synergies at project level is said to be very difficult and not

always realistic, given the fact that the rules and procedures are not standardised across different

EU funding programmes. Some recommendations include a joint funding of projects by different

instruments or funding of cross-project networking activities.

B.5.3. Key points / Areas for improvement

To conclude, the respondents to the stakeholder consultation generally:

 Agree that combining different forms of support for research and innovation into one single

programme is better for addressing their needs than having separate programmes.

 Agree that the programme should increase the use of cross-cutting activities to further

maximize its socio-economic impact.. Respondents also note that the coherence between the

different parts of Horizon 2020 improved compared to the previous Framework Programme.

 Indicate that more could be done to simplify the funding landscape and make it easier for

participants to identify the call(s) that best fit their needs.

 Lack knowledge of other, complementary funding opportunities at the EU level which

indicates that synergies with other EU programmes could be very limited.

Met de EU Monitor volgt u alle Europese dossiers die voor u van belang zijn en bent u op de hoogte van alles wat er speelt in die dossiers. Helaas kunnen wij geen nieuwe gebruikers aansluiten, deze dienst zal over enige tijd de werkzaamheden staken.

De EU Monitor is ook beschikbaar in het Engels.