2. METHODOLOGY

2.1 Literature inventory and selection of studies for inclusion

The present meta-analysis utilized a critical literature review of ex-post economic impact assessments to obtain benefit values for aggregation in the numerator of the CGIAR benefit-cost ratio. Studies were sourced from recognized peer-reviewed books and journals, as well as publications directly produced by individual IARCs. The initial selection of publications for review was based upon a comprehensive inventory of impact assessment literature, through searches in relevant literature databases, such as Agricola, Agris and EconLit. In addition, the publications lists of each of the CGIAR Centres was reviewed, and citations lists from relevant literature were perused, in order to obtain as comprehensive a selection of benefit estimates as possible.

During this search process, global or macro-regional benefits studies were more heavily sought than smaller-scale studies, and a minimum cumulative ex-post benefit estimate of $50 million was required for initial inclusion. The minimum cut-off value was a product of the significant value of the total CGIAR investment to date, as benefit estimates below $50 million (0.7% of the funds invested) have little impact upon the aggregate benefit-cost ratio. Although necessary for the efficiency of the review process, it should be noted that this criterion alone was heavily restrictive, and excluded the vast majority of published IAs.

Only studies published after 1989 were included in the initial document pool, as lag periods between impacts and data collection (often 3-4 years), as well as between research activities and impacts (commonly more than a decade), mean that studies from prior to this year encompass little time for the effects of CGIAR activities to have become evident, or that such studies are likely to include significant effects from pre-CGIAR research investments. While this temporal criterion may have resulted in the exclusion of some significant documented impacts, such restriction is necessary to ensure that estimates are conservative, and do not mostly reflect the prior contributions of other research institutions.

To restrict benefits broadly to target regions, only analyses covering lower and middle income countries were included. CGIAR participation or connection to the analytical process was the final criterion for inclusion in the initial study pool, as the System’s institutional contribution to impacts cannot be reliably determined without obtaining programmatic information from the IARCs.

2.2 Overview of studies in initial document pool

Most of the studies in the initial selection have been independently produced by individual Centres. Thus, there is significant diversity in methods employed, and key assumptions differ significantly among the studies (Cooksy, 1997). Since the purpose of the present study is essentially to derive a credible minimum aggregate benefit level for the CGIAR investment, it is essential that a high degree of confidence may be placed in included values. To do so requires “filtering” for only the most reliable studies. As a result, a critical review process was utilized to disaggregate by characteristics of the methodologies employed after the initial pool of documents had been selected (Figure 3).

Figure 3. Diagram depicting the overall process used in the present meta-analysis to select for “plausible” economic impact studies

2.3 Data collection for review process

Once an initial literature selection was made, key aspects of the methodologies of each of the studies were recorded in an Access database. This database then allowed for qualitative assessment of the methodologies employed against criteria enumerated below. A list of database fields is included in Appendix I.

2.4 Description of “best practices” in economic impact assessment

In order to develop criteria for assessing and including reviewed impact assessments, a general impact assessment model of “best practices” has been developed in the present study through literature review. Although it is recognized that research effects are diverse in nature and that no single linear model can be extrapolated to analyse all possible means of impact, the transparency of the review process is enhanced through specification of a set of standards from which review criteria were drawn. Furthermore, while it is recognized that no real-world study could completely satisfy all of the described conditions, this model of best practices provides the conceptual grounding for categorizing the studies to be included in the present analysis. Even though this model is technically an intermediate result, it is included here, as it comprises a necessary step for the subsequent review of included studies.

For the purposes of the present analysis, ex-post impact assessments can be characterized as typically composed of five interlinked subcomponents:

1. Description of the focal institution’s (the institution for which impact is being assessed) participation in innovation development and the impact pathways of the research process.

2. Empirical estimates of observed trends in technological adoption and productivity.

3. Attribution of observed productivity trends to different relevant causal factors.

4. Development of a hypothetical counterfactual scenario, which supposes the course of action without development and adoption of the innovation (when contrasted with observed productivity trends with the innovation, this forms the basis of impact claims).

5. Economic valuation of the benefits attributed to the research process.

2.4.1 Impact pathway elucidation

Impact pathways may be defined as the conceptualized connections between an intervention and effects on the broader social and physical environment. When impact pathways are defined and described in ex-post IA, a theoretical framework is set for the broader analysis. The research process upon which the IA focuses and the IARC or System role in this process should be delineated in detail, along with relevant innovations produced. Thus, the development of the analysed innovation should be described precisely, so that the proposed impact pathway can be reliably credited, and contributors attributed. It is particularly important that the intended advantages of the developed product be presented, as such will essentially establish conceptual boundaries for the assessed impact pathways. According to Anderson (1997) “to link...elements of agricultural development...to enhanced productivity and welfare...in a consistent framework for measuring productivity is task enough” in and of itself!

2.4.2 Estimation of adoption and productivity trends

Productivity effects can often be collapsed into two factors - adoption and increased efficiency. Both aspects require significant amounts of field data to derive meaningful results. In the absence of a comprehensive and representative empirical basis for claimed impact levels, benefit levels estimated, while potentially plausible, have little true meaning, as realistic assumptions for one agro-environment may not be necessarily extrapolated across agro-ecological conditions. Hence, it is essential that representative sampling procedures be utilized, which are adequately inclusive of the range of conditions and resource endowments under which the innovation is employed. As Maredia et al. (2000) note in their Tour of Good Practice “estimates of research benefits should be disaggregated by commodities, production environment, or geographical basis if the parameter estimates are different for different components of a research programme.” The sample size should be sufficient for deriving reasonably (given time and resource constraints) precise and repeatable figures, and the population should be randomly sampled, to prevent bias.

The data gathering process should ideally encompass the concept of multiple-source-verification or triangulation, which is defined by Baker (2000) as a set of “procedures that permit two or more independent estimates to be made for key variables.” In other words, the results of one data collecting process may be validated through the application of other methodologies, which can produce the same kinds of data. Similarly, Maredia et al. (2000) recommend that impact assessments “combine technical, scientific and economic information from a number of sources.” For example, adoption data derived from seed sales estimates may be validated through field surveys, and yield information derived from experiment station data should be corroborated with surveys or on-farm trials, as even relative yield gains may be greater in the breeding environment than in farmers’ fields.

2.4.3 Attribution of causal factors

It is critical that an impact assessment attempt to demonstrate causality, rather than proceed on the basis of assumed relationships. Correlations do not establish causation when presented in isolation, so the data gathering process must illustrate that the target innovation is a primary causal factor. To do so requires that mitigating influences be identified, and the role of such in affecting productivity changes be methodically assessed. For example, often adoption of a technological innovation acts as a catalyst for adopting other related technologies, by making investment in complementary inputs more profitable (e.g. semi-dwarf varieties eliminate lodging, thereby allowing higher rates of fertilizer application). Thus, the respective roles of complementary contributions to yield increases, should be assessed empirically, rather than arbitrarily supposed.

Furthermore, it should be recognized that many factors interact to influence productivity trends, and that these factors may be exogenous to the agricultural sector. Macroeconomic and trade policies, infrastructure, human capital development and property rights regimes may all catalyse changes in agricultural output and performance, since farmers adaptively optimize production techniques according to external conditions. Given this context, it is a crucial challenge for impact assessment to isolate changes in the productivity frontier from changes in factor use within the same technological boundaries. In so doing, it is particularly important to accurately represent how shifts in the boundaries of the productivity frontier transpire, as inaccurate representations of these shifts have seriously skewed results in the past (Nin et al., 2003).

2.4.4 Counterfactual^[6] development

According to the World Bank’s Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners “to ensure methodological rigor, an impact evaluation must estimate the counterfactual, that is, what would have happened had the project never taken place or what otherwise would have been true... [as] determining the counterfactual is at the core of evaluation design” (Baker, 2000). In many cases, even in the absence of CGIAR activity, it is likely that the NARS would still be producing research products, which would be generating a certain degree of impact. Thus the additionality of the CGIAR ought to be systematically estimated, so as to avoid crediting the CGIAR System with NARS products or vice-versa. This should be based on empirical assessment, and econometric methods offer one possible means to do so. Even in the absence of NARS research efforts, it is likely that yield gains would have been achieved through private sector research, farmer innovation, changes in crop management or factor substitution, and the counterfactual should include these elements. Ideally, such should be based on empirical analysis of farmer substitution opportunities and patterns, and should be illustrated as the “next best” option for maximising utility. The counterfactual, though often only implicitly developed, is equal in importance to the derivation of observed productivity gains for determining the accuracy and precision of results, as the difference between the productivity estimates under the two scenarios constitutes the basis of claimed benefits. Since this subcomponent is of such high relevance for the accuracy of benefits assessed, the methodology for counterfactual derivation should be presented in extensive detail.

Preferably, the counterfactual should be based on empirical analysis of comparisons between a “control group” and roughly equivalent populations of beneficiaries (Baker, 2000). Through socio-economic mapping, in combination with maps of ago-ecological conditions and factor-endowments, it should be possible to isolate groups of non-adopters and adopters with similar characteristics, and trace productive profitability over time. In addition, it should be possible to trace marketing channels to analyse supply changes induced by new technologies, and the effects of these for consumers.

2.4.5 Economic valuation

Once productivity impacts have been estimated as the difference between the observed adopter estimates and the counterfactual, they should be assigned an economic value. Establishing a proper price for the change in production is often impeded by market distortions, such as monopoly buyers, competition with subsidized exports, export taxes, or by poor infrastructure. “Economic prices that are appropriately adjusted to reflect policy distortions in the output market” should be utilized for valuing productivity changes, according to Maredia et al. (2000). The most basic price for utilization is the world market price, and this is an effective means of valuing export commodities, since price distortions are most likely to depress, rather than inflate international market prices.

However, crops primarily cultivated for subsistence use are much more difficult to value, especially when there are few possibilities for substitution. The most representative price for such commodities is probably the domestic price. To maintain credibility, it is best to be conservative in estimating prices, so as to not overvalue productivity increases. In addition, the price effects of supply changes should be assessed through empirically derived demand and supply elasticities. Such are necessary for the development of meaningful analyses, which allow for the distribution of benefits to be assessed. Furthermore, if secondary research impacts are to be estimated, such as reductions in deforestation, it is necessary that the price effects of supply changes be calculated. It is also potentially valuable to spatially disaggregate price effects of supply changes, as marginal rural areas served by poor infrastructure may experience different levels of price change than do major port cities, and such has substantial implications for the distribution of benefits.

Furthermore, when changes in the use of factors with significantly distorted market prices are analysed, adjustments should be made so that the social costs of alterations in input use are included in benefit assessments, through the application of shadow prices. For example, changes in pesticide levels applied to cultivars should be valued at the pesticide price plus incurred additional external costs related to human health or the natural environment. Similarly, Alston et al. (1996) recommend that “total benefits [when accompanied by externalities] are given by deducting the amount of the increased external cost from producer benefits.” Once all cost factors are converted to shadow prices reflective of social values, complementary effects may be considered in economic terms.

2.5 Criteria for study selection

After “best practices” in impact assessment have been described, it was possible to develop a systematic qualitative review process to assess the degree to which such practices have been realized in specific studies. As mentioned previously, the reviewed studies vary greatly in methodological transparency and sophistication. Consequently, since the reviewed studies are of such variable standard, this process was needed to determine the level of confidence and conservativeness with which results may be used.

To develop a structure within which the studies could be reviewed, a hierarchal framework from principles to criteria to indicators was developed (Table 1). In certain cases, criteria and indicators were merged into a single metric. The two overarching principles for the review of assessments were 1. transparency and 2. demonstration of causality, while the former of which was a necessary condition for the later.

2.5.1 Transparency: criteria and indicators

Since the ability to understand the basis of derived results is a requisite condition for placing confidence in findings, it is imperative that credible studies are characterized by transparency. For the purposes of this study, transparency as a principle was represented by three broad criteria (Figure 4):

1. clearly derived and explained key assumptions
2. comprehensive description of data sources
3. full explanation of data treatment.

Figure 4. Hierarchical relationship between criteria and indicators for assessing the transparency of reviewed studies

Clearly defined key assumptions

This criterion was represented by two qualitatively assessed indicators - explicitness of key assumptions and substantiation of key assumptions. The explicitness indicator refers to how openly the study defines which aspects of the analysis have been derived from expert opinion or presumption by the author(s). Substantiation refers to whether a logical basis or citation has been provided to authenticate these untested assumptions.

Comprehensive description of data sources

Under this criterion, four indicators were enumerated, including description of data sources for extent of adoption (when relevant), productivity effects, costs associated with adoption and prices for valuing productivity changes. For each of these factors, it was noted whether all apparent sources of data were specifically cited. General or vague references, such as “productivity data were cited from studies,” with no in-text citations listed, or “interviews were conducted,” with no sample size presented, were rated poorly, while precise citations for all specific data utilized were rated highly.

Full explanation of data treatment

Four indicators were derived for this criterion, in a similar manner to those of the attribution criterion. These include explanations of adoption, productivity, costs and valuation (including discounting/deflation). The ideal against which studies were evaluated was the provision of sufficient information to allow repetition of the methodology used for processing each of these kinds of information.

2.5.2 Demonstration of causality: criteria and indicators

For a study to make a credible claim of impact, it is essential that a causal linkage be established between the intervention and claimed effects. To address the degree to which the reviewed study demonstrated causality, five criteria were developed:

1. Reresentative data set utilized
2. Appropriate disaggregation
3. Adequate consideration of mitigating factors
4. Plausible counterfactual scenario developed
5. Precise institutional attribution.

Figure 5. Hierarchical relationship between criteria and indicators for assessing the demonstration of causality within reviewed studies

Representative data set utilized

This criterion refers to whether the data set utilized for generating adoption and productivity estimates was likely to accurately and precisely represent target populations, and was represented by two indicators - reliability of data set utilized and comprehensiveness of data set utilized. Reliability refers to whether the methods applied for generating observations were likely to encompass significant bias or accurately represent analysed trends. Triangulation, with multiple-source validation applied was most highly regarded, while expert opinion as a sole basis was given the lowest score. Under comprehensiveness of data set utilized a rating was given for the sample size used, both in terms of geographic and temporal coverage, particularly with respect to the basis of claimed productivity effects.

Appropriate disaggregation

To assess the degree to which impact heterogeneity is considered in the reviewed studies, two indicators were used for evaluating the fulfilment of this criterion - disaggregation by agro-environment and disaggregation of surplus generated (between and among producer and consumer groups).

Adequate consideration of mitigating factors

Numerous causal factors apart from CGIAR derived research outputs may explain observed trends in productivity. For the purposes of this review, a single indicator denotes whether major relevant classes of mitigating influences (such as infrastructure, policy and crop management) were incorporated. The degree to which the relative contribution of these other factors was contemplated in estimating impact causality has been assessed for each of the studies.

Plausible counterfactual scenario developed

Two indicators were used to assess the counterfactual scenarios developed in the reviewed studies - counterfactual plausibility and explicitness of counterfactual. Plausibility of implicit or explicit counterfactuals indicates the degree to which the assumed course of events in absence of the innovation represents a realistic “next best” course of action. The relative explicitness of the counterfactual was used as a proxy indicator for the precision with which the counterfactual has been derived in the reviewed analyses. In utilising this factor as an indicator, it was assumed that explicit counterfactuals can inherently give a more precise representation of the changes that would be likely to occur in the “without” scenario, than can implicit counterfactuals based on technological contributions to changes between “before” and “after” scenarios.

Precise institutional attribution

This criterion was represented by a single indicator, which attempted to capture the plausibility of the attributive basis for crediting the involved IARC, for those studies that attempt to do so. When a study did not attempt to attribute the relevant Centre contribution to research products, it received the lowest score. Such should not be interpreted as indicating that this factor renders these studies of low reliability, but it does reduce the reliability of any IARC attributed values derived from these analyses, since the present analysis applied assumed attributive coefficients to derive the CGIAR portion of benefits estimated in such studies.

Table 1. Principles, criteria, indicators and rating examples for evaluating benefit-cost studies

2.5.3 Rating of studies and aggregation scenarios

For each of these indicators, qualitative review produced ratings, which were aggregated and averaged for the two principles to derive a transparency rating and a demonstration rating. In the present study, two main levels of conservativeness were applied, and these are termed “significantly demonstrated” and “plausible,” respectively. The primary difference between the two main standards concerns the degree to which impacts must be proven before being quantified, as opposed to being assigned a reasonable value based on limited evidence. From these two main standards, one additional derivative of the “significantly demonstrated” scenario and two additional derivatives of the “plausible” scenario were drawn, for a total of five scenarios (Table 2; Figure 6).

Table 2. Characteristics of five scenarios under which benefits were assessed in the present analysis

1. Scenario of “empirically attributed and significantly demonstrated” studies

Most of the reviewed studies did not take the step of attributing research products to different institutions via empirical means, as such has not been a commonly recommended practice in impact assessment (Maredia et al., 2000). In lieu of such empirical attributive criteria, plausible assumptions were required to define the portion of collaborative research benefits that resulted solely from CGIAR activities. However, assumed attributive adjustments are speculative, and reduce confidence in results when only the isolated activities of CGIAR Centres are to be assessed. Thus, to be highly conservative, only those “significantly demonstrated” studies that take the extra step of empirically attributing the IARCs should be included in aggregate estimates. Since this scenario is also predicted on classification as “significantly demonstrated,” this highly selective standard only incorporates studies that are both rated highly for transparency and rigour, as well as inclusive of an empirical basis for partitioning institutional credit.

2. Scenario of “significantly demonstrated studies”

The “significantly demonstrated” scenario is conservative, and requires that substantial evidence supports impact claims before they are included in aggregate figures. Requisite conditions for this classification include high ratings for transparency and impact demonstration. Fulfilling the criteria for the “plausible” scenario is a necessary but insufficient condition for the “significantly demonstrated” scenario.

3. Scenario of “plausible” studies

In this analysis, the “plausible” standard could be conceptualized as derived from an a priori position that CGIAR impacts are substantial, when such is supported by expert opinion, and that the role of IA primarily concerns quantifying the value of these effects. Meeting a substantive rating for transparency and a limited rating for demonstration qualifies studies as “plausible.”

4. Scenario of “plausible” studies extrapolated to the present

Since the plausible scenario contains several single year benefit estimates for very large impacts of breeding research, which are truncated to the terminal year of the study period, benefit levels for these research areas may reasonably be expected to continue after the analysed year(s). Significant empirical evidence supports the contention that the benefits of varietal research have not dropped off significantly since these estimates were made, so this could be regarded as a plausible assumption (Reynolds et al., 1999; Sayre et al., 1997). For these scenarios, benefits were estimated to continue at the real rate of the final estimate year through 2001. In cases where such extrapolation could potentially cause benefits to be double-counted, such as when several sequential studies for the same research product are utilized, the more highly-rated estimate was used to cover the period.

5. Scenario of “plausible” studies extrapolated through 2011

The lag periods between investment and the realization of agricultural research impact mean that investment taking place at present will not often begin to have substantive social/economic effects until at least a decade from now. Furthermore, agricultural research products have a useful lifespan that may often exceed ten years. This implies that many of the productive effects of recent investments will not be yet realized, and that future benefits should be estimated to fully capture the effects of investments to-date. If it is assumed that research outputs from such investments are at least sufficient to maintain the level of benefits estimated by the reviewed studies for the next decade, such benefits may be plausibly extrapolated through 2011. Thus, in what may be regarded as the most optimistic scenario, the plausible real benefits from those research products which have been characterized by stable or rising benefit values in the late 1990s were carried forward at a constant level (prior to discounting) from the final study year through 2011. While this scenario helps to offer a fuller estimate of benefits than do the other four, the results are highly speculative, and are subject to a very high level of potential error, as previously unassessed research products are implicitly included. Thus, this is not truly an ex-post scenario, although it only attempts to include the effects of past research activities.

Figure 6. Additive inclusion of benefits in the five scenarios of the benefit-cost meta-analysis

2.6 Aggregation of results

2.6.1 Attribution between NARS and CGIAR

In many of the included impact studies, no attempt has been made to partition the benefits generated between NARS and IARC efforts. This presents a significant hurdle for the reliable estimation of efforts attributable to the CGIAR investment alone, as an explicit, detailed and substantiated counterfactual is necessary to precisely discern relative contributions. However, since such a large share of the included studies did not take this attributive step, it was also not viable to exclude non-attributive studies from most of the scenarios. As a result, plausible and conservative assumptions were made as to the relative contribution by CGIAR institutions. If available, the IARC proportion of the total research budget for the innovation was utilized as an attributive percentage. When attribution was not conducted in an included study, and there was no empirical basis on which attribution could easily be assessed, a blanket attribution level of 50% was utilized. This value was selected because this approximate percentage has been reflected in empirical assessments of the catalytic value of IARC efforts to the total number of outputs produced through NARS-IARC collaboration, as well as in observed average proportions of genetic content in collaborative breeding for major commodities (Evenson and Gollin, 1997; Byerlee and Traxler, 1995). The blanket assumed 50% attribution level offers an implicit counterfactual scenario that in the absence of IARC participation, 50% of the observed research benefit would have been realized.

2.6.2 Deflation and discounting

Once adjustments were made to estimate IARC-attributable benefit levels in U.S. Dollars, nominal deflation/inflation of currency values was calculated via the U.S. Producer Price Index, so as to establish a common base-currency year of 1990 for all included benefits. This was performed independently for each of the studies, as most annual benefit levels were already calculated according to the base currency years of each of the studies, while a few presented annual benefits in nominal values. Once nominally adjusted, benefits from the included studies were aggregated to produce total annual benefit streams, and these total annual benefit streams were discounted using a 2% real social discount rate, with sensitivity analyses lowering the rate to 0% and raising it to 10%. This range of rates was chosen because it represents a realistic range of potential returns to very long-term private-sector alternative investments. While a higher set of rates may be more appropriate for relatively short-term investments, such is a plausible rate of return over several decades or more.

Although the included benefits estimates only cover a small sample of CGIAR activities, they were set against comprehensive cost estimates for every activity of the System, with the benefits from all other actions of the CGIAR omitted. Costs were estimated from total CGIAR System investments reported in Integrative Reports, Financial Reports and Annual Reports published between 1974 and 2001. For 1995, 1996 and 1997, the money awarded to IFPRI by the Asian Development Bank for its rice policy assistance to Vietnam was also added as a cost, since this activity was externally funded. In addition, rice breeding benefits for Latin America and the global value of modern spring bread wheat varieties included benefits derived from research by IRRI, CIMMYT and CIAT that predated the establishment of the CGIAR. To incorporate the costs of additional early research efforts leading to these benefits, investments in IRRI, CIMMYT and CIAT from 1960, 1966 and 1966, respectively, were added to investments after System establishment. These costs were deflated to a 1990 base year, and then discounted according to a 2% real social discount rate.

To express the aggregation process algebraically:

TV = total value of benefits assessed
u = scenario under which estimate is generated
t = year (1990, the base year of the study, equals 0)
s = start year of benefit period
n = end year of benefit period
i = particular study included
z = total number of studies reporting benefits for year
B = benefit value reported in study (in 1990 US dollars)
a = attributive coefficient (if B is empirically attributed this equals 1, otherwise 0.5 is used)
r = real discount rate
TC = total costs of CGIAR and related investments
f = first year of IARC investment associated with outputs of the CGIAR
j = most recent year of CGIAR investment
c = IARC receiving investment
q = number of IARCs receiving CGIAR-related investment
K = investments in IARC
BCR = benefit-cost ratio

2.7 Limitations of the B-C Meta-Analysis

The validity and accuracy of the present meta-analysis’ approach is contingent upon a number of key assumptions. Perhaps most significant of these is the supposition that the presence of the CGIAR System has not resulted in any significant “poisoned wells,” or outputs with significantly negative impacts. Such may or may not be indeed completely accurate, as it is very possible that certain problems, such as the transmission of specific exotic pests, may have been potentially caused through individual System actions. Furthermore, according to Alston et al. (2000) “it is more likely that past R&D (particularly private R&D) has led to technologies that exacerbate, rather than ameliorate, the negative environmental consequences of agriculture, so the omission of these effects has given rise to generally overstated social rates of return.” However, no systematic effort to-date has attempted to analyse the impacts of unintended or inappropriate outputs within the CGIAR (such as accidental pest introductions), and it is likely that if such mistakes were indeed made, they would be very difficult to accurately attribute to specific actions or actors. There have been attempts to comprehensively analyse negative externalities associated with practices accompanying the adoption of IARC-fostered innovations, but these have only been on a qualitative basis (Maredia and Pingali, 2001). Furthermore, they generally conclude that research, or the generation of new knowledge, is difficult to attribute as the source of negative externalities, since these are largely the products of accompanying practices, not the CGIAR output itself. Due to these problems of attribution and quantitative data availability, it is not within the limited scope of the present analysis to attempt to account for negative impacts.

While the present analysis can offer meaningful insights into minimum aggregate levels of benefits generated through the efforts of the CGIAR, such can by no means be considered comprehensive. Most of the System’s impacts have not been subjected to thorough assessment, and many do not lend themselves to easy quantification. Impact assessment at the System level is a relatively young activity. As a consequence, the magnitude, comprehensiveness and methodologies for impact assessment differ significantly among Centres and research activities. Furthermore, there is presently a paucity of methods for quantifying longer-term mandate-level impacts on the mutidimensional problems of poverty, and this is necessary for evaluating true progress towards System goals. Many types of research pursued in the CGIAR also have impact pathways that make attribution especially difficult (such as policy research), or lead to benefits that are difficult to quantify (such as certain kinds of natural resource management research). Therefore, it should be stressed that the absence of quantified benefits for most research areas should not be extrapolated to imply that the impacts generated by such have been insignificant. For this reason, current impact-assessment coverage is insufficient to allow for truly comprehensive analysis or significant allocative insights. Thus, while the present meta-analysis can demonstrate whether past investment in the entire System has been minimally justified by known and measurable benefits, the results should not be used to substantiate or inform future allocation among CGIAR activities. Rather, the significance of the benefit levels generated from such a small sampling of research outputs should offer meaningful proof of the productivity of the overall research portfolio.

Furthermore, while the present study was being conducted, there was only limited opportunity for interaction with the authors of included analyses. Thus, when the methodology of the analysis is not clear in the text of reviewed the studies, assumptions had to be made from the best available evidence presented. In some cases these assumptions may over or underestimate the methodological sophistication of the included analyses.