4.1.1 Accuracy of benefit estimates
All scenarios produced significant net benefits and benefit-cost ratios well above one, even when a high discount rate is used and when extremely conservative criteria are applied, which restricted the included studies to a select few. This strongly suggests that the CGIAR has been a productive investment, and demonstrates a certain degree of robustness in the results. Sensitivity analyses, using a 10% and zero real discount rates, in addition to nominal adjustments, also maintain this result. The insensitivity to discount rate applied may initially appear counterintuitive, but is largely a product of the benefit distribution peaking during the middle of the period for the non-extrapolative scenarios, with costs evenly spread over the benefit duration (Figure 8).
With limited resources, impact assessors can either choose to invest in enhanced precision and detailed coverage of a specific and limited locality, or they can attempt to encompass larger temporal and spatial scales, but with lower reliability and significantly more reliance upon simplifying assumptions. Since the studies reviewed in the present analysis are all of large scope, all of these studies to some degree sacrifice precision in favour of scale. Encompassing greater scale allows for a more complete picture of impact magnitude, but by nature of lower precision, such estimates are more prone to error, and are vulnerable to inaccurate and unsubstantiated assumptions (Figure 11). This comprehensiveness-precision tradeoff applies both to individual studies included within the present analysis and to scenarios developed within the present study. The more comprehensive extrapolative scenarios encompass more of the “true picture” but do so with considerably less reliability than the most restrictive scenarios. To be sure, the application of different minimum standards for impact demonstration greatly affects results, as the most uncertain scenario produces a benefit value nearly nine times as great as the most conservative. Unless minimum standards for acceptance of impact claims are defined, it is difficult to select one of the scenarios for future reference. However, even the most comprehensive scenario included in the present study probably doesn’t encompass most of the System’s impacts, as temporal and spatial coverages are often not complete, and many acknowledged impacts have never been comprehensively assessed.
This limited IA coverage renders it likely that even the most generous of these results may be considered as somewhat conservative. Accordingly, the “significantly demonstrated” scenarios, though more conservative than the “plausible” scenarios, do not produce more accurate values, but represent a viable absolute minimum level of impact. The extrapolative “plausible” scenarios are probably more accurate (closer to the truth), but are less precise (less repeatable), as the methods applied to derive these values are presented in less detail, and rely more upon assumption.
Figure 11. Relationship between inclusiveness of benefit coverage and the error margins of estimates made
Although these values are quite robust, there is potential for over-estimation, even though this is very likely to be outweighed by the many other unquantified CGIAR-derived benefits omitted from the selection of analyses included. Over-estimation may have resulted from the inclusion of benefits derived from innovations that pre-date the establishment of the CGIAR. Modern semi-dwarf varieties of wheat and rice, which comprise the largest documented System research impacts, both had been well established before 1971. Thus, it is arguable that research adoption lags render many of the included benefits attributable to research pre-dating the CGIAR institutions. However, by including investments dating as far back as 1960 in Centres which later became affiliated with the CGIAR System, this potential pitfall has been somewhat addressed, unless research lags are assumed to be very long. The issue of accurately incorporating adoption lags has been long contended in impact assessment, and has great ramifications for analytical results, in terms of both magnitude and attribution. Accordingly, Alston et al., (1998a), have noted that rates of return to agricultural research fall from exceptionally high to comparable with other investments when an infinite lag structure is utilized. Yet, such potential criticism does not substantially apply with regard to the present study, because this study focuses on the marginal effects of applied research, which would not have occurred without the additionality of the assessed activities, even if the scientific basis for such is largely derived from previous innovations. Had the additional efforts of the CGIAR not been pursued, the vast majority of benefits enjoyed would indeed not have been realized, as the pre-CGIAR modern varieties are vastly outyielded by their descendants, which were derived as a result of CGIAR investment. Although adoption lags do present problems in temporal attribution of total effects, marginal additionality of specific actions is largely unaffected.
A second major potential source of error relates to the attribution of simultaneous complementary efforts by non-CGIAR institutions. Most of the included studies, even within the “significantly demonstrated” scenario, do not attempt to partition benefits to different entities, and assessed collaborative efforts holistically. This renders impact claims of specific institutions somewhat tenuous, as there is no empirical basis for assessing the additionality of particular efforts. Furthermore, even when empirical attribution is attempted, the rationale for such is somewhat simplistic, and based on fairly arbitrary assumptions regarding the contribution of genetic content, or reductions in research lag periods. Important contributory factors, including extension, are often not considered, although extension is liable to be of lower significance for the research programmes generating the bulk of assessed benefits (breeding and biocontrol) than for other areas of emphasis (such as crop management). Since the assumed attributive coefficients applied in the present study may have been too generous towards CGIAR institutions, these assumptions may be made even more conservative through further reductions in derived benefit values. To account for such potential for overestimation, if all benefit values reported in the “significantly demonstrated” scenario are halved, the benefit cost ratio still remains reasonably high at 1.88. In fact, if only 27% of the “significantly demonstrated” benefits or slightly more than half of the “empirically attributed” benefits have actually been realized as a result of CGIAR activity, the investment remains sound.
Since most of the reviewed studies use financial prices, without accounting for market distortions, there may be a third significant source of bias if significant social costs accompany these benefits. Indeed, there have been substantial interventions in most agricultural markets of both developing and developed countries during the period assessed. Consequently, there may be substantial scope for over-estimation of returns if significant government costs producing few benefits accompany each unit of production, such as occurs with price-support systems and output subsidies under conditions of surplus production (Oehmke, 1988). Although such interventions have been prevalent in most of the developed world (Western Europe, Japan and North America) during the analysed period, market distortions in much of the developing world are of a distinctly different character. In these countries agriculture was often penalized, rather than subsidized, through various combinations of explicit and implicit taxes. For example, monopsonistic government distribution and marketing agents often bought a significant share of production at below-market prices to offer cheaper food for sale to politically preferred clientele (Sran and Srinivasan, 1987). Export taxes, restrictions or quotas often accompanied these measures, to insure insulation of artificially-depressed prices from world markets, and, perhaps more significantly, macroeconomic policies, such as overvalued exchange rates, essentially imposed high levels of implicit taxation (Binswanger and Deininger, 1997). To compensate for the production-inhibiting consequences of such policies, inputs were often subsidized, and imports were restricted. Overall, agriculture, as a major portion of the economy, was often a net source of government revenue for investment in other sectors, rather than a net sink, as in developed countries (Bates, 1983). Under such circumstances, productivity increases, which raise the quantities of food supplied, and reduce market price equilibria, are less likely to increase government expenditures, although government revenues may decline under certain circumstances. However, from the mid-1980s onward, Structural Adjustment Programmes encouraged by the international donor community, along with ongoing processes of international trade liberalization, have fostered widespread reduction of such market interventions. Under these expanding conditions of liberalization, which should be prevalent for the foreseeable future, there is increasing convergence between equilibrial prices at the market and at the farmgate. As a consequence, the analysed benefit levels should become even less liable to this source of error as benefits are extrapolated to the future.
In all likelihood, the true value of benefits arising from the CGIAR is indeed much higher than any of the values presented here, as most impacts have not been assessed in a systematic and comprehensive manner. Just over half of the Centres are represented in the reviewed analyses, and even for those included, only a small number of research programmes account for most of the included benefit values. It is widely recognized that the CGIAR has achieved much more than enhanced germplasm of wheat and rice, as well as biocontrol of the cassava mealybug. The list of quantified intermediate impacts of the System alone is rather impressive, with thousands of developing-country researchers trained, thousands of studies published in peer-reviewed journals, and significant contributions made to agricultural sciences in the tropics.
Other evidence suggests that intermediate products have produced numerous substantial impacts for target poor populations in developing countries through a myriad array of complementary pathways, many of which have yet to be reliably assessed. For example, according to the “Borlaug hypothesis,” improvements in agricultural productivity induced through research, which have fostered commensurate reductions in commodity prices, have reduced the profitability of production in marginal environments. In turn, such has reduced incentives for the expansion of agriculture at the margin, and has helped to avert deforestation through “land savings.” Following this theory, Evenson and Rosegrant (2003) and Nelson and Maredia (1999) have shown that such impacts may run into the tens of millions of hectares of land saved from being cleared of natural biodiversity or stripped of watershed protection values. Furthermore, natural resource management research and forest policy and management research results should help to enhance such conservation impacts. However, it should also be recognized that the presence of negative environmental effects, which are potentially attributable to CGIAR research products, may mitigate some of these positive consequences (Maredia and Pingali, 2001).
To holistically assess benefits produced by a public-sector investment, the fungibility of funds invested should be considered in concert with the level of benefits produced, as the productivity of the venture does not resolve whether the private sector has been displaced. In the case of the CGIAR, the “global public goods” orientation of the System offers reason to believe that there has been mainly complementarity, rather than competition with other private and public sector research entities. Since the research products produced by the System, as with many other agricultural research organizations, are inappropriable and the beneficiaries do not have sufficient resources or the means for investment, such benefits would not haven realized without public-sector involvement. Moreover, the long time lags and high uncertainties involved in financing such research further discourage private-sector participation (Alston et al., 1998b).
Furthermore, significant evidence indicates that rather than compete with existing agricultural research efforts, the existence of the IARCs catalyses other research investment (Alston et al., 1998b). Evenson’s (1987) econometric study offers particularly strong evidence that investment in NARS was significantly increased, so as to capture economies of scale induced through IARC activities. Thus, not only has the CGIAR impacted target populations through research outputs, but it could be credited with many of the successes of national research systems, as well, since such would have probably been smaller in the System’s absence.
4.1.2 Proportion of benefits reaching target poor populations
To return to a theme mentioned in the beginning of the present study, and to put the benefits generated in a mission-relevant impact context, the proportion of benefits accruing to the poor should be considered to assess whether the CGIAR mission is being effectively fulfilled. This study cannot give a precise breakdown of benefits accruing to different groups of target beneficiaries, due to the simplicity of the analysis and the lack of relevant data in the included studies. However, significant evidence suggests that the poor have received a large portion of generated benefits, in marked contrast to some of the early critiques of the Green Revolution (GR). In many, if not most cases, adoption of improved technologies has not been disproportionately pursued by larger farmers, and in some instances, such as among rice producers in Bangladesh, smaller farmers within high-potential environments adopt innovations, including modern varieties, more frequently than do their larger counterparts (Hossain, 1998). Although it is clear that increased supply often lowers prices, and thereby may reduce revenues to nonadopters in marginal environments, in many cases these producers have benefited from increased employment opportunities in better-endowed areas, as well, through migration (Hazell and Ramasamy, 1991). Employment opportunities for these migrants are often accordingly improved through increased labour intensity, as has been repeatedly noted for Asian rice cultivation systems. Contrary to common criticism of GR technologies, the adoption of labour-saving technologies is not significantly catalysed by MV utilization, and bears greater influence from farm size and relative factor prices (David and Otsuka, 1994). Producer income increases generated through enhanced productivity also do not disappear with adopting beneficiaries, and rather filter through the rural economy, catalysing significant “multiplier effects” along the way. Consequently, it has been estimated that for each additional dollar generated in the farm sector, an additional 50 cents to one dollar is generated in the surrounding non-farm economy, as a result of increased rural demands for goods and services when farm incomes rise (Hazell and Haddad, 2001; Delgado et al., 1998).
Yet, changes in producer returns only comprise a small proportion of the social gains assessed, as the bulk of analysed research benefits have been realized through prices reductions resulting from supply increases enabled via boosted productivity. For example, approximately two thirds of rice germplasm enhancement benefits in Latin America result from such price declines, while only one third accrues to producers (Sanint and Wood, 1998). For modern varieties of spring bread wheat, similar trends have been observed, notably in Pakistan (Renkow, 1993). The predominance of this impact pathway implies that the poor have received considerable shares of benefits generated through the assessed research efforts, since poorer groups spend greater proportions of their income on food (Kerr and Kolavalli, 1999). Moreover, as incomes rise, food consumption patterns often shift from those basic staple commodities which have benefited from documented widespread productivity increases attributable to the research efforts of the CGIAR, to preferred substitutes, such as meat and vegetables (Barker and Dawe, 2002; Huang and David, 1992). These low expenditure elasticities for the analysed products of CGIAR research further help to ensure that the consumer benefits derived from productivity increases accrue in substantial proportion to poorer beneficiaries. As rural-urban migration increases the number of urban poor who have few opportunities for subsistence cultivation, the importance of this impact pathway will grow. In addition, cassava mealybug biocontrol, a substantial source of estimated benefits, can be expected to principally impact poor beneficiaries, as cassava is primarily a subsistence crop in Africa, which is rarely cultivated for export purposes.
It should be noted that the reviewed assessments of surplus distribution are largely dependant on the assumption of free and undistorted markets (with the exception of Ryan, 1999), although such suppositions do not hold true for many developing countries. Moreover, it is unclear how such distortions affect the distribution of research benefits, as much hinges on how developing-country policymakers adjusted distortional measures in response to increased productivity. To the extent that the below-market procurement prices of monopsonistic parastatal “marketing boards” remained static relative to equilibrial price declines, producers received greater shares of research benefits, as these penalising policies are brought closer to liberalized market conditions (Singh, 1988; Bates, 1983). Furthermore, if productivity increases allowed for additional output to be shifted into “parallel markets” for production in excess of marketing board procurements, both consumers and producers would reap substantial benefits, as producers could receive higher quasi-market prices for greater shares of production, and the “closed” nature of these economies would allow for significant price declines. On the other hand, if procurement prices fell as quickly as equilibrial prices, and procurement quantities rose as fast as supplies increased, research benefits would accrue almost exclusively to consumers and government coffers. However, as a general trend, explicit and implicit taxation of agricultural production for food price subsidization has declined, in part due to low commodity prices on the world market, which have been resulted from increased productivity (von Braun, 1988). This pattern implies that producers have reaped a substantial share of benefits stemming from the IARCs.
Based on the evidence described in this section, which indicates that producer benefits have been relatively equitably distributed, while the balance of benefits accrues through price reductions for basic staple foods, it is plausible (and probably even conservative) to assume that impacts generated should often reach the poor in at least equal proportion to their portion of the population. Such an assumption is also consistent with analyses of the distributional consequences of untargeted food subsidies, which have found that the poor generally receive a proportional or higher allocation of benefits derived from food price reductions for commodities with low expenditure elasticities (Ahmed et al., 2001; Alderman and Lindert, 1998). Consequently, if the 23% of developing country population (as of 1999) that subsists on less than one dollar per day (World Bank, 2003) are counted as the only beneficiaries, and are assumed to benefit proportionately, most scenarios still produce satisfactory results, as all “plausible” scenarios result in benefit-cost ratios over unity. Alternatively, if a further supposition ventures that the 56% of the developing world’s population that presently survives on less than two dollars per day are the only beneficiaries counted, the aggregate benefit-cost ratio is more than unity in the most conservative scenario. If the more inclusive extrapolation of plausible benefits to the present is used, this measure rises to four, and if benefits accruing into the future from present research are counted, this rises further to more than eight. One or two dollars per day are crude measures of extreme absolute poverty, and many beneficiaries with incomes over these arbitrary levels are also very destitute by Western standards. If the severely resource constrained are comprehensive counted, target poor populations may be much larger than either of these somewhat crude measures indicates, and the proportion of benefits reaching poor beneficiaries may be accordingly higher. To go even further, these proportions are recent and lower than those when the CGIAR was initiated, so the “true” proportion of estimated benefits reaching target beneficiaries may be higher still. When these results are combined with the fact that most benefits of the CGIAR probably remain unestimated, there is a clear indication that the CGIAR investment has been efficient and effective. It remains as a future challenge for impact assessment to more comprehensively represent these benefits.
During the review process it became apparent that economic assessments of impact within the System could benefit from enhancement of scope and methods applied. All of the studies reviewed could be regarded as financial, rather than economic analyses, as market prices are not adjusted so as to include external effects or to compensate for market distortions (through the application of social or shadow prices). Rather, the studies make the implicit assumption that price equals marginal values, even when such fluctuate drastically due to external influences. Many commodities’ markets are known to be highly distorted, due to external and internal government policies, so a social, rather than financial orientation may be much more appropriate and illustrative.
The fact that all of the reviewed studies take a financial, rather than social, approach runs counter to the recommendations of Alston et al. (1996) that “we ought to take into account the total effects on the welfare of all affected groups when we can.” When a structure for impact assessment at the System level was being initially formulated, a “need to cover both the intended and unintended effects of research at different stages from various forms of uptake of research results to their ultimate impact on target groups or objects (e.g., farmers, environment, society, economy, etc.)” was noted by the Impact Assessment Task Force (Özgediz, 1995). Similarly, since the 1970s, the World Bank has recommended application of the following techniques to facilitate social, rather than financial, analysis (Mosley, 2001):
None of these techniques has been fully applied within the reviewed analyses. To be fair, most multilateral agencies have failed to effectively apply these methodologies to the majority of benefit-cost studies commissioned. However, this failure does not eliminate the necessity of including such factors. While it may be exceedingly difficult to operationalize the inclusion of external costs, and utilize social pricing, little methodological progress can be made if such is not attempted. Simply ignoring these factors does not render them irrelevant, and can create somewhat misleading results, which reduce assessment credibility. For example, many of the most successful CGIAR innovations catalysed increases in input use (such as the initial introduction of semi-dwarf varieties of wheat and rice, which do not lodge under high doses of fertilizer), and the social costs of changes in input-use associated with adoption should be considered in a more meaningful manner. However, it should be noted that some progress has been made in assessing these external effects in separate small-scale assessments, although additional efforts will be needed to integrate findings into large-scale analyses (Pingali, 2001).
Unfortunately, few of the studies utilized an explicit counterfactual scenario. Those that do analyse prevention of losses due to pests or diseases, which is a type of benefit that makes the need for counterfactual scenarios particularly apparent. Virtually all major textbooks on economic project appraisal place a great deal of emphasis on counterfactual development, so it is somewhat surprising that such is attempted so rarely in IARC assessments. Although overt counterfactual postulates may seem somewhat arbitrary and contrived, implicit assumptions for the “without” scenario are even more so. Counterfactual plausibility should also be enhanced through an empirically-derived basis, such as through the identification of control populations. Every impact assessment contains a counterfactual, whether explicit or implicit, and, thus, methodological transparency demands that the assumptions behind such be overtly declared.
The lack of a counterfactual also may have contributed to the meagre consideration of mitigating influences apparent in the studies. While researchers may have an a priori understanding that the research outputs studied have been the prime causal factors for the impact trends observed, this causality is not so readily apparent to the “outside observer.” Arguably, in science it is much easier to disprove than to prove, so refuting the causality of other relevant factors would be quite effective for establishing causality. The somewhat simple approach of the reviewed studies leaves them vulnerable to the impact claims of alternative complementary interventions. For example, the findings of Chavas (2001) that essentially all yield growth may be explained by changes in input use, and that there has been no overall technological development in the agricultural sector are more difficult to refute if input use remains superficially considered in the bulk of assessments.
Since so few of the included studies attempt institutional attribution, potentially controversial conflict-prone claims to research impact are avoided. However, this also serves to undermine the accountability purpose of the IAs, as claims of collective credit do not necessarily offer strong arguments for allocation to individual research entities. Thus, some sort of universally acceptable means of including alternative research providers in counterfactuals, so as to partition impacts is necessary, in order to improve the potential impact of IA on allocative processes.
While these somewhat simple benefit-cost techniques, as presently applied, are valuable for demonstration of investment productivity, such do not have high potential for reliably feeding into priority-setting processes. This limited capability is largely a product of the simplicity of the analytical techniques commonly utilized, along with the paucity of reliable data available to allow the derivation of precise intermediate impact estimates. In the absence of high-quality, comprehensive empirical data on control groups, productivity, crop management, or input use, analysts have to make due with only very limited sets of information, from which only crude extrapolations may be made. With only rough results generated, few recommendations for priority setting can be reliably extrapolated. Since donors have been keen to request “lessons learned,” which are relevant to current programmatic implementation, more comprehensive and representative data sets will be necessary to dependably generate such lessons. To foster the collection of such data, additional resources will need to be devoted to IA within the System.
At an even more basic level, many of the reviewed studies appear to be characterized by a serious lack of methodological transparency, which undermines the accountability objectives inherent in the pursuit of economic impact assessment. Data collection methods employed in the reviewed studies are not frequently explained or cited. For example, key meta-data, such as survey sample sizes, are omitted, while major sources of information are often not explicitly mentioned. The treatment of data is also often inadequately described, so that discount rates, commodity prices and the aggregation of yield increase estimates are not specified. In certain cases such details are described in referenced unpublished internal publications, but such are not easily available to the casual reader. Such opacity reduces the reliability of derived estimates and may make results inadequate for establishing credible impact claims, since the basis for such is unclear. If impact assessors wonder “why has impact assessment not made more of a difference?,” such a lack of clarity is likely to be one viable answer.
In general, ensuring the quality of basic data utilized, rather than focusing principally upon data treatment, should become more of a focus for economic impact assessment efforts. Operationalising this suggestion would require significant changes to the manner in which impact assessments are pursued within the System. Since IA is pursued as research endeavour, rather than as a procedural requirement, the economists who conduct such studies have little incentive to undertake exhaustive and time-consuming data collection efforts, and instead focus on methodological innovation. The lack of a clearly defined role for IA does nothing to aid this situation, as the standards necessary for fulfilling different envisioned roles have never been identified, and allocative decisions do not appear to be tied to the quality with which impacts are analysed. As such, and somewhat ironically, the marginal value of investing resources in impact assessment work has not been proven, as compared with other social science research options. Consequently, research managers have little reason to prioritize impact assessment work, and the effectiveness of the CGIAR investment is not comprehensively represented, nor can “lessons” be effectively “learnt” to enhance future efficacy.
To move beyond the wide range of plausible estimates developed in the present analysis, and “zero in” on a more precise benefit range, a greater degree of consensus needs to be established for expectations from ex-post impact assessment. Assessing benefits with a fixed amount of resources generally presents a tradeoff between scope and reliability, and client preferences are needed to select an optimal allocation between these two attributes. To achieve consensus on such allocation, it is critical that the clients of IA research articulate minimum standards for impact claims. Such standards should not be established by SPIA, Centres or single experts in isolation, as the diverse views of investors and other stakeholders will not be comprehensively represented through such a proxy. It is also critical that dialogue be established among impact assessors and intended audiences, so that such expectations are effectively communicated, and such can realistically incorporate the constraints imposed by limited resources and competing priorities.
The IA methodological literature is replete with “best practice” manuals and guiding principles (Alston et al., 1996; Baker, 2000; Echeverría, 1990; Maredia et al, 2000) but a clear gulf persists between these idealized conceptions and operationalized assessments. Repeatedly, it is stressed that techniques, such as shadow prices be utilized to encompass environmental effects, that conversion factors should help to estimate distributional impacts, and that counterfactuals should be explicit and empirically-derived. Yet, as has been noted in the present document, such techniques are nearly never incorporated into large-scale assessments of research impact, and the simplistic techniques typically applied are rarely criticized for these omissions.
Simultaneously, documentation of impact is increasingly emphasized as a necessary requirement for programmatic accountability. Furthermore, in the methodological literature, impact assessment is often assumed to be a primary source of information for priority-setting procedures (Alston et al., 1996). However, there is no clear and consistent linkage between documented impact and resource allocation. Ironically, those research programmes for which impact has been most thoroughly documented (commodity breeding) have suffered declining budgets, while those for which there have been very few large-scale impacts assessed (natural resource management) have benefited from rising allocations. This dichotomy clearly presents a potential quandary for the accountability role of impact assessment.
The apparent low ability of impact assessments to influence allocative decisions may be principally caused by either the possible low persuasiveness of the current impact assessment portfolio or by overriding concerns unrelated to past performance, which may dominate funding processes. Since the latter has not been analysed in this study, the former is focussed upon in the activities recommended.
The criteria enumerated in the present meta-analysis for critically reviewing the included studies, and the range of benefits presented under the different scenarios may constitute a viable context for initiating improved stakeholder-assessor dialogue. It is recognized that this would not be the first attempt to gain better understanding of client needs, but it is possible that prior attempts may have had few meaningful results due to the absence of a specific context for eliciting responses. It is difficult to abstractly define minimum general data standards, when impacts are so diverse in nature, and the methodological approaches lack an accepted norm. However, if a response to a specific study is requested, it is much more likely than meaningful feedback may be provided. Thus, if a selection of studies encompassing a variety of degrees of entailed effort are to be reviewed by a panel of investor representatives, with the intention of producing a consensus evaluation, trends in expectations may become evident. To ensure that such a workshop is focussed on critical review, rather than synopsis, it would be beneficial for donors to both present and critique Centre assessments. Thus, each investor/stakeholder could bear responsibility for a short summary of content, strengths, weaknesses and points for improvement of a specific IARC impact study. In turn, each presentation would be followed by discussion intended to produce a consensus impression of study quality.
Patterns in expectations evident in the workshop could then be distilled into minimum IA standards, which are broadly acceptable to IA audiences. With these standards established, the studies reviewed in the meta-analysis may be revisited. To facilitate more comprehensive impact coverage in this second attempt at aggregation, the IARCs should be invited to submit additional and/or revised impact studies for inclusion. Once submission has taken place, review should centre on the client-derived standards, and evaluate to degree to which such are met by the reviewed assessments. The resulting aggregate benefit values should then be implicitly acceptable to target IA audiences, and insights from the critical review could be reliably utilized to identify needs for IA improvement.
