Since establishment in 1971, the CGIAR community has invested approximately seven billion dollars in various research and research related activities. In an era characterized by scarce development resources, it is relevant to ask: Do the documented benefits from CGIAR research justify the total investment in the CGIAR so far? Although the CGIAR System has been a world leader in documenting research impacts, no previous study has attempted to comprehensively address this question in a quantitative manner. Thus, this study, which has received strong support from a number of stakeholders, represents a first attempt to scale-up quantified economic impacts to a System level.
At various times, the overall efficacy of agricultural research as development assistance has been called into question. With this in mind, the present analysis is intended to resolve on a preliminary basis whether the entire investment in the CGIAR over time can be justified on the basis of the benefits derived from its proven (and agreed-upon) major successes. Prior impact analyses have been unable to directly address this issue, because such have focused on the costs and benefits only of research successes, while ignoring the costs of associated efforts that have not resulted in quantifiable impacts. The present analysis overcomes these constraints by compiling reliable estimates of large-scale benefits, and comparing such with the total investment in the System to-date, under a number of different explicitly stated assumptions. The reasoning is that if the accumulated, aggregate value of generally accepted and credible benefits from a group of CGIAR activities is at least equal to the value of the entire investment in the CGIAR, when an acceptable alternative rate of return to investment is used to discount/compound benefits and costs, then the investment is justified under the assumption that the sum of benefits from all other CGIAR projects is zero or positive.
SPIA retained a consultant, David Raitzer, to develop this preliminary assessment, which also serves as a basis for deciding whether a more detailed, exhaustive assessment would be worth undertaking in the future. SPIA recognized from the beginning that it was impossible to carry out a complete benefit-cost analysis (comparing total returns with total investment), since the benefits derived from many of the CGIAR outputs have not been measured and, indeed, probably cannot easily be valued in economic terms because: (a) most research monitoring systems are not set up with that need in mind; (b) many important impacts are difficult to attribute; and/or (c) many desirable effects are difficult to value in market price terms. Thus, while aggregate investment can be estimated with some confidence, a comprehensive aggregate benefit figure cannot be easily derived. Consequently, the approach SPIA followed is inherently conservative, and provides only partial estimates of the benefits that have accrued beyond the breakeven point, at different levels of confidence. In fact, it is widely accepted that there are other positive benefits from the rest of the CGIAR investment, even if not yet documented.
More specifically, the present approach involved: (a) identifying available economic impact assessments (IAs) of CGIAR investments showing significant net benefits, (b) synthesising the methodological literature into standards for ex-post impact assessment ‘plausibility’ (c) appraising the transparency and analytical rigour of the benefit estimates provided by identified studies; and (d) adding up the benefits from those studies that met certain standards of rigour, starting with the most highly credible group of benefit estimates, followed by more inclusive standards to see what the relationship was between the entire seven billion dollar investment and the benefits generated at each chosen level of plausibility.
This type of study does face a number of limitations. Such an approach does not allow for appraisal of the rates of return on incremental investments in the CGIAR, nor the comparative efficacy of individual programs or projects. However, the main issue addressed, the demonstrated efficacy of the aggregate CGIAR investment, is critical both for the future of the CGIAR, and for the future of agricultural research as development assistance. In addition, such analysis can help to highlight those areas of research that are proven sources of benefits, and can help identify critical gaps in the CGIAR’s IA portfolio.
To arrive at a collective credible minimum benefit figure, Raitzer spent much time and effort developing a hierarchical framework of principles, criteria and indicators for judging whether benefit estimates from individual studies are ‘plausible’ and could with confidence be included in the aggregate benefit values. These standards alone comprise an important output of the analysis, and will be built upon in the future as a tool for Centres and investors to use in designing and screening impact studies.
With review standards applied, aggregate benefit estimates for different levels of confidence were developed. The study concludes that even under the most conservative of the six plausibility scenarios, the widely-accepted benefits from a few highly successful CGIAR programmes easily justify the entire aggregate investment in the CGIAR. Although the overall result may not be surprising, since other major assessments of the CGIAR and some of its large-scale investments provide evidence of the high returns to CGIAR research, the holistic nature of these findings is novel and persuasive.[1] This study represents the first attempt to aggregate the documented economic benefits produced by the System, and, as such, it offers compelling evidence that the System’s investment has been a productive one.
While, in aggregate, the evidence is impressive, this study does identify a number of ways in which the persuasiveness of individual studies could be further enhanced. In particular, topical coverage by large-scale IAs is somewhat limited, and counterfactual development could benefit from additional attention. In addition, the present analysis notes that increased transparency would strengthen the confidence of results, and more reliable data sources would enhance precision.
One particularly interesting aspect of the aggregate benefit-cost ratios estimated relates to the sensitivity of results to simple assumptions applied in the analysis. For example, a relatively minor slackening of the standards required for demonstration of impact increases the ratio of benefits to costs from under 2 to over 17. Such susceptibility to minor variances in study standards underscores the need for greater consensus regarding minimum expectations for IA among target audiences. In this vein, SPIA is now considering a follow-up activity oriented towards establishing a greater degree of consensus from investors as to their expectations regarding ex-post IA. For the purposes of eliciting client opinions of different scenarios, studies and standards, a workshop is being planned with donors and impact assessment practitioners to supplement and build on a previous conference sponsored by SPIA and CIMMYT.[2]
SPIA is grateful for critical external input provided by a number of individuals during different stages of the study. Early discussions with Alex McCalla, Jim Ryan, and Bruce Gardner provided ideas for the study design and approach. Finalization of this report involved a three step process of review by SPIA Members; review by six external experts; and review and approval by the interim Science Council. In particular, SPIA appreciates the critique and valuable insights provided in the comments by Dana Dalrymple (USAID), Bruce Gardner (University of Maryland), Peter Matlon (Rockefeller Foundation), Mandi Rukuni (W.K. Kellogg Foundation), Meredith Soule (USAID) and Dunstan Spencer (independent consultant).
SPIA commends David Raitzer for a thorough, perceptive and innovative report produced on a limited budget. It is recognized that the results generated provide an excellent preliminary view and an opening for further, more detailed work on the returns to the overall investments in the CGIAR Centres. At the same time, this study provides the starting point for developing a set of standards and criteria for judging the plausibility and credibility of impact assessment related to agricultural research. As a result of this study’s findings, SPIA concluded that there is need for expanded IA coverage in specific programme areas, such as natural resources management. In fact, such activity is presently part of the SPIA portfolio.
Hans Gregersen
Chair
CGIAR Science Council, Standing
Panel on Impact Assessment
|
[1] Evenson, R. and D. Gollin (eds.). 2003. Crop Variety Improvement and its Effect on Productivity: The Impact of International Agricultural Research. Oxon, U.K.: CABI; and Gardner, B. 2003. The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research. Washington, DC, USA: World Bank, Operations Evaluation Department; Anderson, J.R. 1985. Summary of International Agricultural Research Centers: A Study of Achievements and Potential. Washington, DC: CGIAR Secretariat. [2] See Watson, D.J. 2003.
Why has impact assessment research not made more of a difference. Proceedings
of an International Conference on Impacts of Agricultural Research and
Development in San Jose, Costa Rica, 4-7 February, 2002. Mexico, DF: CIMMYT.
The new proposed workshop would focus more on CGIAR donors and their needs.
At present it is envisioned that participating donors would be asked to
present short summaries of content, strengths, weaknesses, and points
for improvement of specific CGIAR ex-post IA studies. This would help
to define in clear and definitive terms the needs and expectations of
some of the primary users of ex-post IA studies. Patterns of expectations
evident in the conference would be distilled into minimum IA standards
broadly acceptable to IA audiences. In turn, with these standards defined,
it will be possible to revisit the wide spectrum of estimates developed
in the present analysis, and ‘zero in’ on a more precise benefit
range. |
