SETIS

Strategic Energy Technologies Information System
Menu

Recent News

Analysis of Uncertainties

Available data do not allow to directlyassess current R&D investments dedicated to SET-Plan priority technologiesfrom industry and public bodies. The methodology applied in this report aims atcompensating the missing information to the extent possible and to reduce theoverall uncertainty in providing figures at EU level, even though the necessaryassumption-based approach is associated with some uncertainties. In thefollowing, the largest error margins stemming from the lack of data and thesubsequent approximation procedure are quantified.

Uncertaintiesassociated with estimates for corporate energy R&D investments

The analysis of corporate R&Dinvestments by technology includes a number of uncertainties, the level ofwhich depend on whether exact figures could be obtained, official data wasavailable as a starting point, or 'educated guesses' had to be made.

Figures with a 'very high accuracy' (or very high confidence level) could beobtained for companies

  • forwhich the R&D investment is known through annual reports or the EUIndustrial R&D Investment Scoreboard, and that are active exclusively inone technological field. Here, it was assumed that 100% of the well-known totalR&D investment is allocated to the respective technology;


  • thatprovided the exact breakdown of their R&D investments either through directcontact or in official publications.

The uncertainty associated with figures onR&D investment for this group of companies is estimated to be in the orderof ±2%.

Figures with 'high accuracy' (or high confidence level) are those that may beslightly inexact, but the probability of missing at least the right range isvery low with an estimated uncertainty range around ±10%. They relate to

  • companiesfor which the estimates made in the present report were refined through directcontact, but which did nevertheless not provide exact figures on their R&Dinvestments by technology;


  • companiesfor which the present estimates (or range) are supported by figures from otherstudies.

Figureswith 'significant uncertainties'

relate to estimates that had to be made on the allocation of the total R&Dinvestment to individual SET-Plan priority technologies. This is the case forcompanies that are active in various fields at the same time and for which noneof the above two points applies. Here, the R&D expenditures have beenassessed with the methodology described above, which relies on a number ofassumptions based on indirect indicators (such as the number of staff or patents;total sales by division), press released and expert guesses. Whenever possible,different approaches have been combined in order to control the uncertaintiesrelated with one approach. For example, an analysis based on the number ofR&D employees working in this specific technology was cross-checked with aparallel assessment based on patents or turnover. Nevertheless, the estimatesmade for the R&D investments for this group of companies are approximatevalues only and we assume an uncertainty range that may reach ± 50% of thecentral estimate.

Overall, the allocation process thus provesto be the greatest source of uncertainty in the approach. Enhancing the levelof certainty of the outcome would require a more systematic research. A morecomprehensive analysis will require an intensified direct contact to companies,a more systematic assessment of the companies' patent registrations, anassessment of the business areas and a closer look into press announcementsthat may reveal plans for future development and thus allow some conclusionsregarding R&D priorities (see also chapter "Methodology").

Finally,there have been a few companies for which the lack of information did not evenallow a rough estimate. However, this does not concern any of the major R&Dinvestors and would thus not distort the aggregated result to a large extent.The total lack of data typically occurred for some small companies active inone technological area only, such as biofuels or H2/FC. It can also refer tocompanies for which the total R&D expenditure is available from the EUScoreboard but the lack of any information did not allow any allocation toindividual technologies.

Figure 22 illustrates the distribution ofresults with the various levels of accuracies in terms of both the number ofcompanies and the total R&D investment. 

Analyses of Uncertainties 1

Figure 22: Number of companies and share of corporate R&Dinvestments by level of uncertainty of the analysis

Source: Own analysis

Applying the uncertainty ranges of ±2%,±10% and ±50% to the overall results, the overall uncertainty in the totalcorporate R&D investment could amount to a maximum of ±€568 million, roughly ±30% of the total1. This figuredoes, however, not include any uncertainties that stem from the fact that somecompanies are not considered due to lack of data.

The number of companies associated to thedifferent levels of accuracy and with it the uncertainty ranges vary acrossindividual technological sectors. In the areas of wind energy and CSP with anelevated share of specialised companies, the R&D investments can beestimated with a very high accuracy for more than 75% of the companiesconsidered. This share decreases for PV (around 40%) and even more so forsectors such as CCS and smart grids, in which most of the companies consideredare active in multiple business fields, thus necessitating an assumption-basedbreakdown that decreases the level of accuracy.

Whilethe above description applies for the estimates made for the year 2007, the uncertaintiesassociated to the rough estimates provided for the year 2006 are larger as thelatter are partly derived from the hypotheses made for the year 2007. This isdescribed in more detail in box 3.  

Box 3 - Methodology and uncertainties for approaching the 2006 R&D corporate investments

The scope of this report lies in the estimation of selected R&D investments for the year 2007. As an assessment of a one-year snapshot, however, bears a risk of giving too much weight to one-off events or data mavericks, it was decided to also include an annual average of the public national R&D expenditure between 2002 and 2007 (see section 2.3.2). Even though data scarcity does not allow for a similar approach on corporate R&D investments, a rough estimation of the corporate R&D investments for the year 2006 has been carried out.

Nevertheless, the accuracy of the 2006 figures remains below the 2007 estimates. This is due to the fact that a simplified approach has been used for estimating 2006 corporate R&D investments, partly derived from the assumptions made for 2007. Depending on the data availability for an individual company, an estimate has been produced on the following basis: 

  1. The exact figure has been taken from official reports, the EU R&D scoreboard, etc. where available for 2006.
  2. The total R&D investment was provided for 2006 but there was no information about how it was spread over the SET-Plan priority technologies. In this case, the same share (in terms of percentage of the total R&D) has been assumed as for the year 2007. This extrapolation is the source of some additional uncertainties given that the assumptions were originally 'tailored' for the year 2007.
  3. It has been assumed that the 2006 R&D investments are equal to 2007 figures. This is mainly assumed for small companies for which no information was available for 2006. This requires an idea on whether significant changes in the R&D activities from 2006 to 2007 have occurred for the company considered.
  4. An estimate of the 2006 R&D expenses has been obtained on the basis of the number of R&D employees in the year 2006. This approach was mainly applied for companies specialised in a particular technology (e.g. fuel cells small companies).


All in all, the results for 2006 are associated with higher uncertainties than the 2007 figures, impeding a direct comparison between them. Nevertheless, we consider that the accuracy allows a qualitative indication of the trends in corporate R&D investments, without being able to quantify it with a high degree of precision.

Uncertaintiesassociated with estimates for public energy R&D investments

As the assessment of national publicR&D investments of EU Member States largely draws on the IEA RD&Dstatistics and refers to the GBOARD only as a reference for cross-checking onthe aggregated level, the following assessment of uncertainties focuses on databased on the IEA database.

Uncertainties in the IEA figures mainlyoriginate from the differences in the extent to which individual Member Statesinclude regional funding, institutional budgets and support to demonstrationactivities in their original data. Such discrepancies limit the accuracy of adirect comparison across Member States.Furthermore, even for a given Member State, this may changeover time, adding some uncertainty when assessing the R&D trends over time. The mismatch betweenIEA members and EU Member States as well as the lack of data for some IEAmembers for a certain year and technology makes it difficult to derive an aggregated figure for the EU Member States'public energy R&D investment. As this is nonetheless needed for the presentanalysis, it had to be approximated by applying a gap filling procedure and byexcluding some Member States from the analysis of public national R&Dinvestments (unless official data could be obtained through the consultationprocess with Member States). The impact of the above limitations on the presentresults is discussed below.

Accordingto the IEA questionnaire (IEA, 2008), federal R&D budgets should becomplemented by regional (e.g. provincial) R&D spending when significant. Evenso, this does not seem to be the case for many countries, while it is includedfor others (e.g. Belgium).In the case of Germany,for example, R&D support through regional governments (Länder) is not partof the data underlying the IEA statistics. The regional support to non-nuclearenergy R&D of the 16 German Länder amounted to around €96 million in 2006(Schneider, 2007), equivalent to considerably more than one third of theequivalent federal budget in 2006. At the aggregated EU-level, the regionalGerman funds would be in the order of 4% (including both nuclear andnon-nuclear energy R&D as one may assume that regional funds directedtowards nuclear R&D are of limited nature). Unfortunately, theunder-estimation on the aggregated EU level that stems from the non-inclusionof some regional funds cannot be further quantified as it would require an in-depthassessment of the national data included in the IEA RD&D statistics, which hasbeen outside the scope of the present work. However, it must be considered thatregional R&D funds take an important role only in a limited number ofMember States, foremost all Germany,for which an estimation of the uncertainty could be performed. One may thereforeconjecture that the total uncertainty stemming from potentially missingregional funds should not exceed some ±10-15% of the total.

The IEA data focuses on energy-relatedR&D and as such excludes basic research 'unless it is clearly orientedtowards the development of energy-related technologies' (IEA, 2008; section 2.1). Often, this implies that the national data relate to a nationalenergy R&D programme, thus missing additional energy-related R&Dspending that stem from other programmes (such as defence or general researchprogrammes). At the same time, parts of the institutional funding included mayin practice cover research of a more basic nature. The extent to which suchdata are included can not be further quantified. It is expected to vary acrossthe Member States, influenced by the structure of their national energyresearch programmes and institutional set-up, and must be taken into accountwhen comparing Member States' data one with another.

As explained in section 2.1, the data included in the IEARD&D statistic shall capture public national support to demonstrationactivities in addition to their R&D support. However, most of the IEAmembers do not include or display this data. The share of demonstrationactivities thus remains small in general, yet differs between countries andtechnologies (Figure 3). This needs to be kept in mind when comparing dataacross countries and technologies.

Data gapsmake it difficult to assess the trend of R&D investments over time (such asthe one shown in Figure 10). This is due to changes in the methodology, thegeographical coverage etc. For example, the German data prior 1992 do notinclude the new Länder. Other Member States have provided only partialinformation for few years. For Belgium,data for the years 2000-2006 are missing. France recently changed themethodology applied for calculating its national public research anddevelopment expenditure on energy (DGEMP 2007; MEEDDAT, 2008). Public budgetswere re-calculated officially in accordance with the new methodology back tothe year 2002 and match the IEA figures for those years. The figures of the IEAdatabase prior to 2002 relate to the previous methodology. Any trends over timeneed to note this break in series, in particular considering the discrepancybetween the two approaches (e.g. the results differ by a factor of 1.9 for theyear 2002) and the fact that Franceaccounts for around one third of the EU Member States' aggregated budget.Despite this risk of distortion it was decided to not manipulate the IEA datafor Franceprior to 2002, but to restrict the analysis to the data directly available fromthe database in order to ensure comparability with other sources.

Asmentioned in section 2.3.2, only 19 of the 27 EU Member States are IEA members.This implies that the database systematically contains no data for Bulgaria, Cyprus,Estonia, Latvia, Lithuania,Malta, Romania, and Slovenia. A comparison with theenergy R&D budgets according to the GBAORD database, which includes datafor most Member States, reveals that the mistake made in the EU-aggregate thatis caused by the lack of data for some Member States remains limited (see Table5). The aggregated R&D budgets of the Member States covered by the IEAdatabase account for around 99% of the overall EU-27 energy budget according toGBAORD data, notwithstanding that the contributions of the missing MemberStates may be higher for individual technologies. 

The aggregated EU Member States' nationalpublic R&D budgets would be more distorted by the lack of data on R&Dinvestments that occur for some IEA members for a single year - often the mostrecent year; here: 2007 - and technology. At the time of downloading theinformation form the IEA RD&D statistics in January 2007, information ontheir 2007 energy R&D investments was lacking for 10 of the countriesassessed. Consequently, the aggregated figure for the year 2007 would havesummed up to €1237 million only. Due to the exchange of data with a number ofMember States, official national figures could be obtained to fill these gapsfor three countries (and adapt the figures for two others); for the others, asimple gap filling procedure has been applied. For entries missing for 2007, the value fromthe latest available year was applied down to the year 2003. Overall, once the 'datagaps' are filled, public national energy R&D investments in 2007 are almosta factor of two above the aggregate that was based on the 'raw data', and are wellin line with the levels found for the years before and the GBAORD figures. Thisresult justifies the data manipulation applied in the present report.

Thedistortion caused by the gap filling procedure is limited. Gap filling withvalues from previous years has been done for three countries with a totalR&D investment of €257 million. If one assumes that the maximalannual changes of their energy R&D investments do not exceed the relativelyhigh value of 20%, the mistake caused by the gap filling would be €57million, equivalent to 2.4% of the total aggregated figure over all EU MemberStates considered. Of course, for some technologies, a more drastic gap fillingprocedure has been necessary. Nevertheless, given that the main interest ofthis report lies on the aggregated EU figures, the gap filling approach seem appropriateand the related distortions could be limited due to the direct exchange of datawith some Member States. 

In total, we assume that the potentialerrors made in the estimation of the aggregated public R&D investments ofEU Member States should not exceed ± 13% to 19%, notwithstanding that it may belarger for individual technologies.

Main uncertainties associated with theassessment of EU R&D funds under FP6results from the biunique allocation of individual projects to one group ofSET-Plan technologies and the assumption of an even split of the investmentsover the entire duration of FP6. The latter seems fully justified for thepresent work as it levels out annual fluctuations due to the project cycles.

In order to avoid double-counting of projects,as a general principle the funding of an individual project was allocated tonot more than one technology. Considering that a number of projectssimultaneously undertake research in fields related to different groups (e.g.CCS and hydrogen production), this leads to an uncertainty associated with theaggregated EU FP6 funds by SET-Plan priority technology. This is most elevatedfor the sector hydrogen and fuel cells: if all related projects were accountedfor in this sector instead of removing those that have their research focus inother technological fields, the total FP6 funds would amount to €318 millionover the period 2002-2006 instead of the central figure of €279 million used inthis report (i.e. ±13%). Also for CCS, biofuels and to a lesser extent CSP thebijective allocation process generates some distortion. At the aggregated levelover all technologies, however, these uncertainties level out unless they occurbetween SET-Plan priority and non-priority technologies (such as betweentransport biofuels and bioenergy). We estimate this error to not exceed ±5%.

Combineduncertainties

Not all uncertainties of the presentanalysis can be quantified. In particular, the present assessment of corporateR&D investments tends to be an under-estimation of total industrialresearch efforts in this area, given that a number of companies could not beincluded in the present assessment due to either the lack of data or theirmissing inclusion in the list of relevant companies by technology. Furthermore,important up-stream research activities that are realised in the supply chaincould only be captured to a limited extent.

Keeping in mind that the overall figurestend to be an underestimation, an upper range of the  uncertainties related with the R&Dinvestments that are included in the present assessment can be roughlyquantified. Assuming from the above reasoning an overall uncertainty of notmore than ± 30% for the estimates on the corporate R&D investments, ±19%for public national investments and ±5% for the EU funds, the cumulative erroron the total R&D investment in SET-Plan priority technologies (€3.3billion) would not exceed ±€784 million, or 24% of the total.

------------------
1 An uncertainty analysis performed at the level of individual companies indicates a smaller uncertainty range of ±18% but contains some methodological problems and has thus not been further pursued.