APPENDIX II INTERIM TERMS OF REFERENCE AND GUIDELINES FOR THE FIFTH EXTERNAL PROGRAMME REVIEW OF ICRISAT

TERMS OF REFERENCE FOR EXTERNAL PROGRAMME REVIEW OF ICRISAT^[6]

BACKGROUND

Context

1. The Consultative Group on International Agricultural Research (CGIAR) is an informal association of over 50 members that supports a network of 16 international research Centres in agriculture, forestry and fisheries. The CGIAR aims, through its support to the Centres, to contribute to promoting sustainable agriculture for food security in developing countries. Because the Centres constitute the core of the CGIAR, the effectiveness of each Centre is crucial to the continued success of the CGIAR as a System.

2. Each Centre is an autonomous institution operating within the mandate assigned to it by the CGIAR, and is governed by a legally constituted Board that has full fiduciary responsibility for managing the Centre. To ensure accountability in an essentially decentralized system, each Centre is expected to be responsive to the CGIAR, which provides financial support for its work.

3. The CGIAR has established a tradition of external reviews to provide a mechanism of transparency and accountability to the Members and other stakeholders of the CGIAR System. External reviews are conducted for each Centre approximately every five years. As each Centre is autonomous, external reviews provide a measure of central oversight and serve as an essential component of the CGIAR's accountability system.

4. Besides the External Reviews, Centre Commissioned External Reviews (CCERs) are undertaken at each Centre. These CCERs are commissioned by the Centre Boards to periodically assess the quality and effectiveness of particular aspects of a Centre's work. The terms of reference (ToRs) for each CCER are determined by the Centre.

5. External Reviews complement the CCERs by providing a CGIAR-commissioned and comprehensive external assessment of the Centre's programme and management, especially its future directions and the relevance and quality of its research. The ToRs for the external programme review (EPR) of ICRISAT are provided below. Guidelines for undertaking the review are issued separately.

TERMS OF REFERENCE

Objectives and Scope

6. The EPR seeks to inform CGIAR members that their investment is sound, or recommend measures to make it so. Members of the CGIAR and other stakeholders can be informed whether the Centre is doing its work effectively and efficiently. The EPR is both retrospective and prospective; and help ensure the Centres' excellence, relevance and continued viability, and the coherence of the CGIAR System's research agenda. The review is expected to be strategic in orientation and programmatically as comprehensive as the situation warrants.

7. The broad objectives of the EPR is to: a) provide CGIAR members with an independent and rigorous assessment of the relevance and quality of research and research-related activities and the contribution of the Centre they are supporting to its goals and those of the CGIAR; and b) to provide the Centre and its collaborators with assessment information that complements or validates their own evaluation efforts, including the CCERs.

8. The EPR Panel is specifically charged to assess the following:

a) The Centre 's mission, strategy and priorities in the context of the CGIAR's vision, priorities and strategies;

b) The quality and relevance of the science undertaken, including the effectiveness and potential impact of the Centre's completed and ongoing research;

c) The effectiveness and efficiency of programme management, including the mechanisms and processes for ensuring quality; and

d) The accomplishments and impact of the Centre's research and related activities.

9. The topics expected to be covered by the EPR are listed below.

TOPICS TO BE COVERED

A. Mission, Priorities and Strategies

• The continuing appropriateness of the Centre's mission and goals in light of important changes in the Centre and its external environment since the previous external review.

• The policies, priorities and strategies of the Centre, their coherence with the CGIAR's goals (of poverty alleviation, natural resources management, and sustainable food security), and relevance to beneficiaries, especially rural women.

• The appropriateness of the roles of relevant partners in the formulation and implementation of the Centre's strategy and priorities, considering alternative sources of supply and the benefits of partnerships with others.

B. Relevance and Quality of Science

• The relevance and quality of the science practiced at the Centre.

• The effectiveness of the Centre's processes for planning, priority setting, quality management (e.g., CCERs, peer reviews and other relevance and quality assurance mechanisms), and impact assessment.

C. Effectiveness and Efficiency of Research Leadership and Programme Management

• The performance of the Centre's Board in programme oversight, the effectiveness of leadership throughout the Centre, and the suitability of the organization's research culture to its mission.

• The adequacy of the Centre's organizational structure and the mechanisms in place to manage, coordinate and ensure the excellence of the research programmes and related activities.

• The adequacy of resources (financial, human, physical and information) available for planning and implementing Centre's research programmes and the effectiveness and efficiency of their management.

• The effectiveness of the Centre's relationships with relevant research partners and other stakeholders of the CGIAR System.

D. Accomplishments and Impact

• Recent achievements of the Centre in research and research-related areas.

• The effectiveness of the Centre's programmes in terms of their impact and contribution to the achievement of the mission and goals of the Centre and the CGIAR.

GUIDELINES FOR THE EXTERNAL PROGRAMME REVIEWS OF ICRISAT^[7]

INTRODUCTION

1. External Programme Review (EPR) of ICRISAT will be carried out in accordance with the process Guidelines outlined below and the Terms of Reference (ToRs) issued separately. The review is expected to be strategic in orientation and programmatically as comprehensive as the situation warrants. To be credible and acceptable, the review must strive to be objective, transparent and participatory. The reports must be direct, explicit and frank. These principles are observed throughout the review process.

2. Being a member of a review Panel is usually an interesting and rewarding experience. Moreover, Centre management and staff generally welcome the opportunity to discuss with Panel members their achievements, concerns and future plans. A healthy atmosphere of mutual respect and collaboration in the interchange of ideas is the key to the success of the review. It helps to ensure that the recommendations of the Panel are realistic, are well understood by the Centre management and staff, and will be willingly, or even enthusiastically, implemented.

GUIDELINES

3. The EPR is expected to maintain high standards of quality and rigor, and be conducted by an independent and objective Panel. The EPR is expected to assess the Centre in terms of its: mission and overall strategy, programme priorities and strategies; relevance and quality of its science; achievements and impact; and effectiveness and efficiency of programme management, as noted in the ToRs.

4. It is inevitable that the conduct of a review requires the collaboration of numerous individuals; as well as a process that enables the various participants to collaborate effectively in a complex assessment that has to meet high expectations and tight deadlines. The main participants in the EPR are: the EPR Panel Chair and members; the CGIAR Members; the interim Science Council (iSC) and its Secretariat; members of the Centre's Board, management and staff; the Panel's support team of external consultants and resource persons from the iSC Secretariat; and the Centre's many partners at the local, national, regional and international levels.

Roles and Responsibilities

5. The CGIAR, iSC and the iSC Secretariat. The CGIAR establishes external review policies for the System, and EPR is conducted on its behalf, in accordance with the ToRs and Guidelines. For each review, CGIAR Members are requested to propose Centre-specific issues for the Panel to consider, and receive the review report. Once the timing of the EPR has been determined, generally according to the 5-yearly schedule, the iSC and its Secretariat are responsible for the coordination and management of the EPR, and they provide guidance on matters of review design and Panel composition, in consultation with the Centre's Board and management.

6. A senior staff member of the iSC Secretariat will serve as a resource person throughout the review process, accompanying the Panel Chair and members during their visits to the Centre and on field visits. The iSC Secretariat resource person also serves as the Panel Secretary. Besides substantive briefings on technical, programme and programme management matters, the resource person assists the Panel on process matters, including the logistical aspects of report preparation and production. However, to help safeguard the EPR Panel's independence and objectivity, the Secretariat resource person is not normally expected to undertake substantive review, analysis or writing responsibilities on behalf of the Panel.

7. The Panel Chair, Members and Consultants. The leadership and task management skills of the Panel Chair are obviously critical, as are the expertise and experience of Panel members. The Panel Chair is appointed by the iSC in consultation with the Centre. The Chair's involvement begins early on, when he/she is consulted regarding Panel composition, and briefed by the iSC Chair and the iSC Secretariat about the review process and key issues and concerns regarding the Centre. Once the review is underway, the Chair is responsible for ensuring that the Panel undertakes its assessment and completes the task in accordance with the ToRs and Guidelines for this EPR. Given the magnitude of the task, the complexity of the issues, the fact that many Panel members may by unfamiliar with the CGIAR, the importance of maintaining dialogue with the Centre, and the need to produce a report that reflects the consensus of the Panel, the Chair's task is a demanding one.

8. Because the report should reflect the judgement of the whole Panel, all members of the Panel are expected to contribute to all aspects of the review report. The staff member provided by the iSC Secretariat assists the Panel Chair and members throughout the process, as appropriate. Consultants are also provided to the Panel, as needed, for limited periods of time, for assessment of specialized areas. While these consultants and resource person from the iSC Secretariat (and sometimes an iSC member) support the Panel's efforts as members of a team, ultimately the Panel is responsible for formulating the assessment and recommendations of the EPR report.

9. The Centre Board, Management and Staff. The Centre's Board, management and staff play a crucial role in the conduct of the review. They are heavily involved in planning the review, and subsequently in organizing the review and preparing for the Panel's visits to the Centre and to the field. Once the review is underway, it entails a significant degree of interaction between the EPR Panel and Centre staff, as part of a valuable two-way learning experience. Throughout the process, the collaboration and inputs of Centre management and staff are essential for the review to run smoothly and for the report to be credible and acceptable.

10. The Centre's Partners. Representatives of national agricultural research systems (NARS), regional fora, bilateral and multilateral agencies, NGOs and the private sector are important partners of CGIAR Centres, and their input is considered essential for the viability of the EPR review process. As part of the review, representatives of such organizations are consulted for their views on the Centre's long-term strategy, programme priorities and strategies and collaboration. This may be through Panel visits and/or meetings, as well as through questionnaires or interviews. The Panel may also visit or contact managers and researchers from other CGIAR Centres and other relevant institutions with which the Centre collaborates. Such consultations are valuable as a means of assessing the Centre's role in the CGIAR and in the global context. Given the vast number of collaborators or potential partners of a Centre, such meetings must be limited. Their outcome is considered important, however, and is expected to feed into the Panel's assessment of the Centre.

Panel Composition and Report

11. Panel Composition. The review Panel is composed of experts in research and research management areas relevant to the Centre being reviewed who can carry out a comprehensive assessment and give the CGIAR their best judgement about the past performance and future potential of the Centre. The Panel is expected to make an independent assessment based on its own observations and other information available to it, particularly the evidence provided through CCERs (see below).

12. The EPR Panel normally consists of about five members, including the Chair. Panel members are generally selected for their ability to focus on the institution-wide issues relating to the Centre's mission, long-term strategy, research priorities and strategies and programme management and oversight. To ensure adequate coverage of the ToRs, the Panel composition usually meets the following requirements: a) the Chair and at least two Panel members are familiar with the CGIAR; b) at least two Panel members have a technical background relevant to the Centre being reviewed; and c) at least two Panel members have expertise in research organization, management and oversight.

13. Panel Report. The EPR report is expected to present an accurate account of the outputs and what is known about the impact of the Centre during the review period. It is expected that in-depth reviews of particular programme or programme management or programme oversight would have been undertaken earlier through CCERs - and would not normally need to be undertaken by the EPR Panel. This enables the EPR Panel to concentrate on the important strategic issues rather than on specialized detailed assessments of each programme, project or activity.

14. Because research in the CGIAR System is a long-term undertaking, the problems the Centre is working on may not have visible outputs until several years. For this reason, the review report is expected to provide convincing evidence on the relevance and quality of the completed and ongoing research, and the efficiency with which the work is conducted, as a surrogate measure of the potential impact of the Centre's current programme of work.

15. Although the EPR report is expected to be comprehensive, the Panel has considerable leeway in deciding on what issues it would focus in depth. The review report highlights the most significant issues faced by the Centre and makes recommendations on how the Centre (or the CGIAR) could address them. It provides assurances and convincing evidence to indicate that other aspects of the Centre's programmes and management (i.e., those not covered by the Panel's report in depth) are effective and efficient. It also comments on the effectiveness of the Centre's internal review system on which the EPR was based, and on how well the Centre has addressed the recommendations of the other reviews commissioned by the Centre.

Integration with Centre Reviews

16. It is expected that some detailed high-quality CCERs would have been completed within 2 or 3 years preceding the main phase of the EPR. The CCERs are undertaken by specialized external consultants, assisted by members of the Centre Board and staff as resource persons (not participants). They are expected to cover at least portions of the Centre's main research programmes (including their relevance, direction, science quality, achievements, and, to the extent possible, impact) as well as aspects of Centre management (including programme governance, research organization and management, financial and human resource allocation for research and its management effectiveness).

17. The Boards would decide which programme related CCER reports are made available, at the time of their completion, to the iSC and its Secretariat. These reports, along with comments from the iSC and Secretariat staff, are made available to the EPR Panel, along with reports of the follow-up actions planned or taken by the Centre's management and Board. Other analytical papers - particularly internal assessments of programme performance and impact - and other background documentation prepared by the Centre are also provided to the Panel, at the discretion of the Centre. The Centre is responsible for providing this information in an easily accessible and usable form, so that the EPR Panel's conclusions can be based on a comprehensive and thorough review of all aspects of the Centre.

18. The CCERs - which are often very detailed and comprehensive - provide essential evaluative information to the EPR Panel on particular aspects of the Centre's programme and management. Their availability in advance of the main phase of the EPR helps create an integrated system of Centre- and CGIAR-commissioned reviews of each Centre, and enables the EPR to be forward-looking and to focus more on strategic, rather than operational, issues.

19. The EPR, then, can serve as a vehicle for analyzing, verifying, and synthesizing the information already available through CCERs and other reviews, and for making this information available to a wider audience outside the Centre. While the Centre's Board and management are responsible for ensuring that the internal evaluation system is sound (in terms of scope, coverage, quality and timeliness), judgements on the adequacy of a Centre's quality assurance system, including the processes for undertaking CCERs and other mechanisms of peer review, are the responsibility of the EPR Panel.

Review Design and Board Assessment Visit

20. Interactions between the Centre Board and the Panel form an essential component of every review, given the Board's important role in the CGIAR System. Hence, early in the process, prior to (or sometimes during) the first visit of the full Panel to the Centre (see below), the Panel Chair along with iSC Secretariat resource person and possibly one other Panel member or consultant attend a Board meeting, and interview Trustees concerning Board and Centre matters related to the overall research and programme strategy, programme oversight and management, research and research-related priorities and strategies. This design visit helps ensure the participation of the Board in the planning and design of the upcoming review, including the identification of key issues and concerns of relevance to the EPR.

21. The design visit also provides the Panel Chair and selected members or consultant an opportunity to review any documentation provided to the Board, interact informally with individual Board members, observe at least one formal meeting of the Board and its committees, and serve as an element in assessing the Board's effectiveness and operations in so far as these apply in assessing the relevance and quality of programmes and the future evolution of the Centre. The preliminary assessment of the Board is made available to the Panel (but not the Centre), and is modified as appropriate during the main phase of the EPR (see below).

22. In assessing Board effectiveness and operations, the Panel takes into account the key legal documents governing the Centre - particularly the Establishment Agreement, the Headquarters Agreement, and the Constitution of the Centre. It also keeps in mind the main provisions of the Guidelines for CGIAR Boards, particularly the guideline on the "Role, Responsibilities and Accountability of Centre Boards of Trustees", as they apply to programme oversight, leadership and management.

Panel Appointment and Briefing Phase

23. Following the Panel Chair's visit for discussions with Board members (or sometimes coinciding with it) and the issues identified, the full Panel will be appointed. Once appointed, the Panel will receive briefings from iSC Secretariat staff and Centre management on the recent developments in the CGIAR and the Centre being reviewed, and on the processes, quality and content of the CCERs made available to the Panel. The Panel will be briefed by the Panel Chair and Secretary in a virtual mode. Subsequently, the Panel will receive a virtual overview briefing on the Centre's current activities and future plans, and further elaboration of the strategic issues to be covered by the review team. The Panel will prepare preliminary drafts of key sections based on an agreed outline of the report which will be completed during the main phase several months later.

24. Briefings in a virtual mode by the iSC Secretariat resource person cover technical and programme management/oversight matters such as the CGIAR's mission, priorities, strategies, programmes and impact assessments as well as management matters including Board's programme governance. These briefings by the iSC Secretariat also cover the CGIAR's expectations regarding the scope and process of the review (as outlined in the TORs and Guidelines for EPR); as well as an overview of programme and programme management issues of relevance to the Centre being reviewed. The resource person from the iSC Secretariat also provide substantive and process-oriented support as requested by the Panel Chair.

25. The Panel then receives briefings, e.g., through documents and PowerPoint presentations shared with the Panel; a structured e-mail conference among Centre senior staff and Panel members; and tele- or videoconferencing, from Centre management and senior staff on the Centre's long-term strategy, research priorities and strategies, programmes, programme (research and research-related) management and research leadership. These briefings focus particularly on the Centre's recent developments and achievements, CCER findings and conclusions, and future plans. In addition, the Panel seeks additional information from other Centre staff, on a selective basis, as needed; and invites Centre staff members during main phase, either individually or in small groups, to voluntarily share their concerns, if any, regarding Centre-wide programme and research management issues.

26. To help ensure that these briefings and discussions are as comprehensive and up-to-date as possible, and to enable the Panel to obtain a comprehensive overview of the Centre's work, the Centre is expected to provide to the iSC Secretariat and Panel members, in advance, copies of the recent CCERs and other assessments undertaken, as well as other relevant Centre-related documentation (such as the latest Strategy document, Medium Term Plan, and other relevant policy documents or analytical papers prepared by the Centre). For the list of documents generally provided to the Panel by the iSC Secretariat and the Centre, see Attachment I. The Centre should prepare documents specifically for the review, and these should include those indicators of scientific quality as agreed by iSC, Panel Chair and Secretary: e.g., publications, breakthroughs, solutions to problems, new technologies and other products, awards and other recognition of scientists).

27. Once the briefings are completed, the Panel spends few days to prepare preliminary drafts and précis of sections based on the outline of the report agreed by the Panel and the writing responsibilities assigned by the Panel Chair. This ensures that the Panel undertakes a significant amount of preliminary drafting prior to the main phase of the review, and continue its assessment of the key issues and concerns during the period between the briefing phase and the main phase.

Field Visits

28. To help ensure that the EPR Panel's assessments are adequately grounded in the reality of the Centre's circumstances, the Panel members are expected to undertake country field visits, jointly determined by the Centre, Panel Chair and the iSC Secretariat. The field visits cover the major non-headquarters based operations of the Centre, so as to provide a realistic assessment of the Centre's field operations, working conditions, and interactions with NARS and others in the region. These visits by Panel members (as smaller "sub-panels", if necessary) are often few days each, and are undertaken before the main phase of the review.

29. A senior staff member from the Centre normally accompanies the (sub) Panel members on these field/country visits, but does not participate in substantive discussions with country officials or representatives of regional fora. The resource person from the iSC Secretariat helps coordinate the field visits and accompany the Panel members, as requested by the Panel Chair. These visits supplement any surveys of NARS and Centre staff, organized by the resource person from the Secretariat in advance of the main phase.

Main Phase and Report Writing

30. The EPR Panel visits the Centre for a period of about ten days to undertake the main phase of the review, and to bring its report to a semi-final draft stage. As noted earlier, the Centre is expected to have made available to the Panel, well in advance of this visit (through the Panel Chair and Secretary), copies of CCER reports and other relevant documents; and the Panel is expected to have completed the field visits and been adequately briefed by the resource person from the iSC Secretariat. The Panel is thus expected to be reasonably well informed about the Centre and be familiar with other detailed evaluations of its specific programmes and activities by the time it undertakes its own assessment of the Centre.

31. The EPR Panel's report is expected to focus on the four topics covered in the ToRs - namely, the entre's: a) mission, strategy and priorities; b) relevance and quality of science; c) effectiveness and efficiency of research leadership and programme management; and d) what is documented about accomplishments and impact. The report is expected to be succinct and written in plain language, focusing on strategic issues. It can, where relevant, propose forward-looking recommendations on overall direction and priorities (rather than on detailed programme content or operational management). The writing style is expected to be direct, explicit and frank.

32. Since descriptive material and detailed analysis is expected to be kept to a minimum, a report of about 50 pages - with suitable cross-referencing (not summaries) of the CCERs - is expected. However, if the CCERs available to the Panel are inadequate in quality, coverage or depth, the EPR Panel's report is expected to compensate for gaps through its own analysis and assessment.

33. The drafting of the EPR report is completed soon after the main phase visit, and the final draft chapters are shared with the Centre management to ensure their accuracy and completeness. The Panel Chair formally transmits the document to the iSC Chair. If convenient, the main findings and recommendations of the final EPR report are normally expected to be presented by the Panel Chair to the Centre Board, management and staff. Response and Follow-up.

34. The Board and management of the Centre under review are expected to submit a formal written response to the EPR report, addressed to the iSC. Then the iSC discusses the report in the presence of the Panel Chair and representatives from the Centre (including the Board Chair and Director General), and prepares a commentary, including recommendations for follow-up action by the CGIAR or the Centre. The EPR report, the Centre's written response, and the iSC commentary are then simultaneously distributed to and considered by ExCo and also posted on the Web where it is available to CGIAR member agencies and all interested stakeholders prior to the formal discussion by the Group at its annual meeting.

CONCLUSION

35. EPRs provide the CGIAR and other stakeholders very valuable information on the accomplishments and future prospects of each Centre funded by the Group. Because they undertake a comprehensive strategic assessment of all key aspects of the institution, such reports from an independent external Panel can provide much needed assurance to the CGIAR Members - as well as to the Centre's Board, management, staff and partners - about the Centre's direction and its institutional capacity to produce the desired research and research-related results. If significant changes in direction, scope, focus, or mode of work are required, these too can be made on a systematic and periodic basis, based on Board-endorsed EPR recommendations. In any case, the Centre and the System benefit from such reviews.

Attachment 1

LIST OF DOCUMENTS FOR THE EXTERNAL PROGRAMME REVIEW OF ICRISAT

The following is a list of documents for the Panels conducting the External Programme Review of ICRISAT. Copies of relevant documents should be sent to Panel members in advance by ICRISAT and the iSC Secretariat as indicated. Copies of all documents should be available at ICRISAT during the review.