Silly talk: nonsenses by the evaluators of EU funded projects

Hands up who has never received an evaluation of a project with a mistake by the evaluator. Unfortunately, the opinions of the evaluators in practice cannot be appealed, and after losing weeks to write a good project, it hurts to be charged with nonexistent pitfalls. In many cases the lower score assigned to the project for nonexistent pitfalls makes it not funded.

The evaluators are human beings, and as such can make mistakes. They often work in a hurry, on subjects sometimes they know little and with little supervision. EACEA and National Agencies should select and supervise them better, get rid of those who work badly, and allow the applicants  to signal mistakes  in assessment and, why not, review the lists of approved projects for the most serious assessment mistakes. A full re-evaluation of the project by others evaluators should be prescribed expecially when the evaluator reads the form in a hurry or proves no to know the program. In both cases all the evaluation can be biased.

This page, unfortunately bound to grow, reports a series of mistakes of evaluators on projects I know directly because I submitted them. If it happened to you too, you can report mistakes of evaluators that will be published on this page. Please note I have had also many projects approved, that received excellent evaluations, I only publish this article for the benefit of the community of project writers, to campaign for an improvement of the quality of evaluation in European projects.

Post scriptum 1: I could tell you of many more cases, but I should need to describe every project in detail, so I don’t list them here.

Post scriptum 2: my impression is that with evaluation carried out at national level (as in Strategic partnerships Erasmus+) mistakes and oddities by evaluators increased. When all projects were submitted to Brussels the group of evaluators was probably smaller, so more expert evaluators were selected compared to now with selection carried out at national level. Additionally (but this is a long time problem) if I am right previous experience on European project evaluation, project writing, project management is not considered as a criterion on selection of evaluators.

17. Error category: logical errors in the evaluation process
Type of program: Erasmus + Strategic partnership
Project identification number: 2015-1-PL01-KA202-016398
Assessment criterion:  Relevance of the project. Under ‘weak points’ the evaluator writes: It is also important to explain how partners will find 8000 people as users of the project and check their activity’.
Why it is nonsense: because this is an observation related to the criterion Impact and dissemination, and it should have be placed under Impact and dissemination. By placing his/her criticism under Relevance of the project the score of Relevance is wrongly reduced. This ends up altering the relative weights among  the different criteria.

16. Error category: the evaluator does not know the program and questionable opinions
Type of program: Erasmus + Strategic partnership
Project identification number: 2015-1-IT02-KA204-014880
Assessment criterion:   Quality of project design and implementation. The evaluator writes: ‘The cost of multiplier events (final national conferences) organized in every partner country at the end of the project is overestimated.’
Why it is nonsense: because in Strategic partnerships the funding of multiplier events is calculated at unit cost based on the number of participants (100 € per participant resident in the country where the event takes place, and 200 € for each participant non-resident, see Erasmus + program guide p. 117). The Guide only sets a maximum of 30,000 € for every project, that in my project has not been exceeded (if you exceed 30,000 € the application form gives back an error code, so it is impossible). The cost of the events cannot therefore be overestimated, except perhaps in the exceptional case in which you organize one only event for 300 people asking € 30,000 of funding, but it was not my case. In the project, I planned to organize 5 conferences for a total of 300 people, one with 100 participants -funding required 10,000 €- the other four with 50 participants each -funding required for each € 5,000. Events organized in a professional way require publicizing the event, renting of the premises, printing materials for distribution to participants, the buffet, sound and lighting, hostesses and stewards to welcome the participants, wardrobe, the shooting of the event; therefore the unit cost estimated by the Guide is justified, and in any case the evaluator should stay with the parameters set by the Guide.

15.A. Error category: questionable opinions and lack of supervision
Type of program: Erasmus + Strategic partnership
Project identification number: 2015-1-IT02-KA201-015003
Assessment criterion:  Quality of project design and implementation. Evaluator 1 writes: ‘The budget is well balanced, taking in consideration the length of the project. However, the costs of financial guarantee requires attention.’ Evaluator 2 writes: ‘The ratio costs- benefit is quite disproportionate. Only some of costs under the budget heading exceptional costs  (external evaluation and financial guarantee) are eligible and proportionate.’
Why it is nonsense: Evaluator 1 approves the budget, but disapproves the cost of the financial guarantee (‘requires attention’).   Evaluator 2 approves the cost of the financial guarantee, but disapproves the overall budget (it is ‘ quite disproportionate’). Now, it can be acceptable  two evaluators  have some slight different points of view on some aspects of the same project, however, when like here the disagreement is sharp and related to fundamental aspects like the overall budget, there is a clear problem: one of the two evaluators is incompetent and both lacked appropriate supervision. Additionally, Evaluator 1 should explain what he/she means with ‘requires attention’. The cost is not eligible? The amount is too high? (by the way, exceptional costs are reported at real cost, so it is impossible to inflate them). It has to be explained. By writing a obscure ‘merits attention’ the Evaluator criticizes a feature of the project (and gives low score)  without assuming responsibility for it.

15.B. A similar case in the project -1-IT02-KA204-014880. Evaluator 1 writes: ‘The proposal (…) is based on a authentic and sound needs analysis.’  Evaluator 2 writes: The proposal is not based on a detailed needs analysis, rather on the desire to favor the learning of the disadvantaged adults.’ More: Evaluator 1 writes: [the proposal] ‘demonstrates added value at European level because the results could not come out by activities carried out in a single country.’ Evaluator 2 writes: ‘the proposal demonstrates only partial added value at European evel. The results could come out by activities carried out in a single country, however they could benefit by a international approach, notwithstanding this aspect has not been adequately stressed in the proposal.’ Cases like these  (divergent points of view among evaluators, even on important features)  unfortunately are not so uncommon.

14.A.  Error category: the evaluator does not know the program
Type of program: Erasmus + Strategic partnership
Project identification number: 2015-1-IT02-KA201-015003
Assessment criterion:   under  Impact and dissemination Evaluator 2 writes: ‘The dissemination activities are appropriate, however they are not innovative’
Why it is nonsense:  because according to Erasmus + program guide, Version 3 (2015): 14/11/2014 p. 114 innovativeness is not a requisite requested to dissemination. According to the Guide, dissemination has to be effective. Innovativeness is instead request to the criterion Relevance (p.113).

14.B. The same in a project approved in France. The Evaluator writes: ‘Le management du projet reste assez classique.’ Again, innovativeness is not a requisite requested to the management of the project.

13.A. Error category: questionable opinions
Type of program: Erasmus + Strategic partnership
Project identification number: 2015-1-IT02-KA204-014880
Assessment criterion:   Overall comment to the applicant. Evaluator 1 writes: ‘The cost for the external evaluator  is not justified.’ This is to say the external evaluation is unnecessary. By the way, this section of the evaluation sheet should be used to point out the main results of the evaluation described in the previous sections, not to insert, as here, additional feedback.
Why it is nonsense:  because recourse to an external evaluator during the life of a project  is usually considered a must.

13.B. The same in the project 2015-1-RO01-KA202-015013.

12. Error category: the evaluator does not know the program
Type of program: Erasmus + Strategic partnership
Project identification number: 2015-1-RO01-KA202-015013
Assessment criterion:  Under Impact and dissemination the evaluator writes: ‘No funds are provided for achievement of the 1000 multilingual brochures describing the project.’
Why it is nonsense: Strategic partnership the cost of printing brochures describing the project is already included in the funding for Project management and implementation.  In detail, the Erasmus + program guide, Version 3 (2015): 14/11/2014 p. 116  under the costs funded under Project management and implementation quotes ‘information, promotion and
dissemination (e.g. brochures, leaflets, web information, etc.)’

11. Error category: the evaluator does not know the program
Type of program: Erasmus + Strategic partnership
Project identification number: 2014-1-IT01-KA202-002481
Assessment criterion:  Quality of project design and implementation. The evaluator writes:  ‘cofinancing has not been inserted’.
Why it is nonsense: because in Strategic partnerships co-financing is included in unit costs and is no longer shown in the budget (with the only exception of exceptioal costs), it is impossible to ‘insert’ co-financing  (see Erasmus + program guide, Version 3 (2015): 14/11/2014, p. 237)

10. Error category: questionable opinions
Type of program: Erasmus + Strategic partnership
Project identification number: 2014-1-ITOl KA202-002481
Assessment criterion:  Impact and dissemination. The evaluator writes: ‘The reference to the creative commons license gives no added value if it is not specified the specific version of license.’
Why it is nonsense: because all the Creative Commons licenses provide to third parties other than the authors the possibility to download and share original works (see Wikipedia ), and the reference to the Creative Commons licenses suffices to ensure the free use of the products of the project and to meet so its evaluation criteria set out in p. 114 Version 3 (2015): 14/11/2014. It is no coincidence that of the projects on Strategic Partnerships I submitted in 2014, this observation was made in only 2 projects assessed by this assessor.

9.A. Error category: the evaluator does not know the program and questionable opinions
Type of program: Erasmus + Strategic partnership
Project identification number: 2014-1-IT02-KA204-003582
Assessment criterion:  Relevance of the project.  Evaluator 1 writes: ‘The proposed project is relatively consistent with its objectives and priorities of the action in question [adult education]. (…) The project objectives are clear, but not particularly relevant to the program. The theme is relatively innovative and spills over in the development of VET tools. The evaluator 2 writes: ‘The project proposal is borderline regarding the goals of adult education, its features are most appropriate for the VET action, and includes in its objectives the priorities of VET’. Mostly based this, the criterion was rated low (15.5 / 30).
Why it is nonsense: because, as explained on p. 10 Erasmus + program guide Version 3 (2015): 14/11/2014: ‘Erasmus+ aims at going beyond these programmes, by promoting synergies and cross-fertilisation throughout the different fields of education, training and youth, removing artificial boundaries between the various Actions and project formats.’ So a key feature of Erasmus+ is the loss of the rigid division among target groups / sectors  of the previous  LLP Lifelong Learning programme. With the new approach, if a project is recognized overlapping with additional target group(s) it should not be penalized. However, the project was penalized because although the logic of Erasmus+ is profoundly changed from the earlier LLP, the Italian National Agency remained structured in three different agencies, each of them dealing  with specific target group / sector. So an organizational issue (the inability to merge the 3 agencies previously in charge for  LLP and Youth for Europe) distorts the approach  of the new Erasmus+. In addition, I disagree also the project was more relevant to vocational training. The project aimed to develop in adults a series of transversal skills that make possible the entrepreneurial competence, such as Decision making, Creativity / problem solving,  Setting goals / planning, Listening / Negotiating etc. The only technical skills, one of 10, was Budgeting / Financial monitoring. As 9/10 skills did not have a professional content the project in my opinion was firmly grounded on adult education.

9.B.  The same project in point 3.A., resubmitted in 2015 in Germany under VET. The evaluator writes: ‘The reference to vocational training is not sufficiently justified. (…) At this point, the question arises whether the project would be better placed in another program area of Erasmus+. As It has numerous points related to issues of general [i.e. adult] education.’ So the project is rejected again. It is like the the Game of the Goose when you go into the space ‘Go back  to the starting point’ (in this case the space is ‘Resubmit the project’). However, the general problem is that, despite declarations of principle contained in the Programme Guide, in Erasmus+ the division into target groups / sectors of the previous LLP remains rigid.

9.C. A confirm of the still rigid division into target groups / sectors results by the penalization of transversal projects, that is to say projects addressed to more than one target group / sector. The rate of approval of transversal projects is consistently lower of the rate of not transversal projects. For example in Italy the rates of approval of transversal projects in 2014 was: adults, school, university 4%, VET 3%, youths 0%. The rates of approval of not transversal projects was respectively  40%, 11%, 4%. At European level the rate of approval of transversal projects is 11% against 36% of the not transversal (source: my  elaboration by data provided by EACEA).

8.A. Error category: the evaluator reads the form in a hurry
Type of program: Erasmus + Strategic partnership
Project identification number: 2014-1-EN01-KA202-002476
Assessment criterion:  Impact and dissemination The evaluator writes: ‘it is not clear  how the number of  15,000 people involved via the web has been calculated.’
Why is nonsense: because at p.53 of the form I have included a detailed description of the number of people involved and their mode of involvement: ‘

  • Newsletters 500 addressees x 6 countries = 3,000 informed every time / National and European level
  • Information about the project partners websites on 50 hits / month x 12 x 6 = 3600 informed / National and European level
  • Project 1000 website hits in the last 12 months of the project in 1000 informed / National, European and international level
  • etc. (I will not reproduce all entries).

The total is 15,000.

8.B. The same occurred in project 2015-1-PL01-KA202-016398. Under weak points the evaluator writes:  ‘It is also important to explain how partners will find 8000 people as users of the project and check their activity.’  The explanation is found on pages 51 and 52, but apparently he did not read it.
In the same project, the evaluator complains ‘In the application it has not been described so exactly, if and how the analysis of needs for the goal groups has been carried out.’ Contrary to what the evaluator wrote, needs analysis prior to the presentation of the project is described on p.24.
In the same project, the evaluator writes: ‘The proposal does not include newcomers to this action.’ A newcomer  is an organization that has never participated in a European project and in this kind of projects ensures additional score. The assessor did not realize that at p. 25 of the form is written  (in capital letters just to make it more evident) P3 HAS NEVER PREVIOUSLY PARTICIPATED TO A PROJECT EU. The same information comes out by the description of P3  on p. 11. In all applications submitted this year I wrote this information in capital letters because last year it happened to me the same case, but I see that even writing in all capital letters is not enough.

8.C. The same occurred in project  2015-2-PL01-KA205-017366. Under ‘weak points’ the evaluator writes: ‘The partnership are organizations that have already benefited from funding as part of the Strategic Partnership. In dealing with requests priority is given to projects that have not yet received funding.’ This is wrong, because P3 has never participated to a EU funded project before, as it results by the table at p.13 and as explained all in capital lettes at p. 29:  P3 HAS NEVER PARTICIPATED TO A EU PROJECT BEFORE.

7. Error category: the evaluator reads the form in a hurry
Type of program:  Justice Daphne
Project identification number:  JUST / 2013 / DAP / AG / 4000005586
Assessment criterion:  the evaluator writes (project evaluation of Daphne was communicated to us not divided by criteria): The cost-effectiveness is good, but apparently translation costs have been forgotten. This should be checked and possibly amended.’
Why it is nonsense:  because contrary to what the assessor says translation costs were  budgeted in the cell H204. The description of the cell is as follows: ‘Translation on subcontracting of texts (Tool box corresponding to 25 pages of text, add on 1 page, Apps 6 pages, website 10 pages, text of 6 videos 12) in FR, DE, SP. Total pages 54 x 3 languages = 162 at 35 euros for page. All the page have been budgeted under the same WS and subcontracting budget assigned to only one partner so to get bulk rates in translation. Subcontracting care of Carlsen T Project Coordinator, trainer, permanent staff, private law contract.’

6. Error category: logical errors in the evaluation process
Type of program:  Justice Daphne
Project identification number: JUST / 2013 / DAP / AG / 4000005586
Assessment criterion:  overall assessment of project (the one that has been  transmitted) is as follows:
‘This proposal addresses Priority 2.2.6. References to EU policies, UN-decisions and recommendations of the Council of Europe are numerous and relevant. The proposed activities are relevant to the Call and the Priority. The consortium aims at reducing gender stereotyping impact of media on youth by means of electronic tools that help to identify and decode sexist stereotypes, providing also the possibility to tag not gender-biased media content. These tools are completed by an educational toolkit for teachers, which will be piloted with a sample of 1800 students. Provisions have been made that such a large-scale exercise in six European countries will be possible. Europe-wide dissemination, also in other than the project languages has been planned and sustainability has been considered – as far as this is possible in the online world. The cost-effectiveness is good, but apparently translation costs have been forgotten. This should be checked and possibly amended.’
You see the only critical element highlighted in the proposal is the lack of budget for translation (actually there was, see previous case above), which led to a penalty of 3 points on 10 available on the Cost-effectiveness criterion (so  the Justice Directorate in charge of the call wrote us). However, the project received only 80.5 points out of 100 available instead of 97/100 (100 – 3 points for the alleged lack of budget for translations).
Why it is nonsense:   A project on which the evaluator highlights no pitfalls should always get a scoore of 100/100. Based on the principle of transparency of assessments, when assessors assign scores below the maximum they ALWAYS SHOULD EXPLAIN THE REASON WHY. Instead, assigning scores below the maximum without explanation is unfortunately quite common.

5. Error category: the evaluator reads the form in a hurry
Type of program:  Erasmus + Strategic partnership
Project identification number: 2014-1-AT02-KA205-000199
Assessment criterion:  the evaluator writes (I translate from German) Youth Pass will not be used, and this is a big problem for a project submitted on Youth in action, whose goal is to develop soft skills.
Why it is nonsense:  because the use of Youth Pass It is listed at p. 47 Every participant that successfully finishes one unit of the E-course  will receive a certificate issued by the country partner describing the unit(s) attended and the learning outcome(s) reached, structured coherently with Youthpass’ and again p. 57: ‘The training is structured in learning outcomes, will validate the learning outcomes (…) and grant a final certificate structured according to Youthpass.’

4. Error category: the evaluator does not know the program
Type of program:  Lifelong Learning
Project identification number:  527533-LLP-2012-DK-GRUNDTVIG-GMP and 526534-LLP-2012-IT-GRUNDTVIG-GMP
Assessment criterion:  Cost-benefit ratio. In the first project, the evaluator writes ‘WP9 foresees a Grundtvig Training Course, but no  resources are allocated for this outcome.’  In the second project, the evaluator writes: ‘the budget does not foresees resources for (…) the Grundtvig Training Course.’
 Why is nonsense:  because in the old program Lifelong Learning implementation of Grundtvig Training Courses  was funded by participation fees provided directly to participants by the Commission.

3. Error category: the evaluator does not know the program
Type of program:  Lifelong Learning
Project identification number:  143721-2008-IT-KA3-KA3MP and 502608-LLP-1-2009-1-IT-GRUNDTVIG-GMP
Assessment criterion:  Cost-benefit ratio. In the first project, the evaluator writes ‘It is not comprehensively explained why the projet manager wants to meet the EU Commission 4 times during the project-lifetime.’ In the second project, the evaluator writes: ‘There is duplication (4) of trips to Brussels’.
Why is nonsense:  because both evaluators were not aware that the Commission had established (it was written in the Program Guide) that all projects were to include budget for two trips a year to Brussels for meetings with the Commission.

2. Error category: questionable opinions
Type of program:  Lifelong learning
Project identification number:  566330-LLP-1-2012-1-IT-GRUNDTVIG-GMP, 526400-LLP-2012-1-IT-GRUNDTVIG-GMP and 526766-LLP-2012-IT-GRUNDTVIG-GMP
Assessment criterion:  Cost-benefit ratio. In the first project, the evaluator writes: ‘The consortium has incorrectly calculated travel costs with one single average price for all return journeys no matter who is travelling from where to where.’ In the second project, the evaluator writes: ‘The budget includes only average costs for return journeys no matter who is travelling where which makes it difficult to estimate whether  the budget is overestimated or underestimated. Hence, the travel costs should be estimated for each partner individually to give a more precise picture of the actual costs involved.’ In the third project, the evaluator writes: ‘Travel and other costs are reasonable, but the average return journey in table 3 is the same no matter where the project meeting takes place.’
Why it is nonsense:  the projects are written in the Spring, then if approved they begin in late fall or January of the following year. 8-10 months in advance (time lag between the date of writing the project and the date of purchase of tickets for the first project meeting; for the other project meeting tikets will be bought at a distance of even 2 or 3 years) it is almost impossible to get a realistic estimate of the cost of tickets. Some companies do not provide the cost of tickets so in advance, the cost of those that provide them is constantly changing, not necessarily in a linear way. Additionally in the few months before the trips some companies will start new routes or suppress existing ones. For these reasons, the choice of an average cost is the most appropriate, considering also the project cost have to be reported at ‘real costs’. Indeed, this criticism was only done on 3 projects on dozens I have submitted over the years, all with average cost. Indeed, the new Erasmus +, which replaced the Lifelong Learning programme, uses a system based on standard average costs.

1. Error category: the evaluator has a superficial knowledge of the sector / subject area to which the program is focused
Type of program:  Leonardo da Vinci 2003-2004 Project identification number:  IT / 04 / C / P / TH-81409
Assessment criterion:  Relevance of results and potential impacts. The evaluator writes ‘Results and impact appear limited – much of the described material is already available’.
Why it is nonsense:  the project involved the development of a resource center on the web collecting and describing  materials used to carry out career guidance. All resource centers (housed in a physical location or on the web, including that foreseen by my project) by definition collect materials already produced. The website and the review of the materials existing guidance in several European countries would have allow spreading them further and a better use. With his/her  comment, the evaluator proved ignorant of what resource centers are and what is their function.

 

Continues (unfortunately).

This article should be read together with Why good projects are rejected? and What is the halo effect and how it distorts the assessment of European funded projects

Mr. Leonardo Evangelista is a expert of European projects, in different roles: project writer, project evaluator, trainer on project writing.

Author © Leonardo Evangelista.  See Informativa privacy, cookie policy, copyright e limitazione responsabilità. Copy and reproduction by any mean are forbidden.

Rispondi

Translate »
%d blogger hanno fatto clic su Mi Piace per questo: