• Home
  • Outcome Evaluation Approach – 5 Case Studies from FTA

Outcome Evaluation Approach – 5 Case Studies from FTA

Yordana Yawate, carries a sack of sago pith to be filtered on the banks of the Tuba river in Honitetu village, Maluku province, Indonesia. Photo by Ulet Ifansasti/CIFOR
Posted by

FTA communications

Two recent publications discuss how to effectively assess the impact of transdisciplinary (TDR) research and apply these methods to 5 case studies.

The creation of the CGIAR Research Programs (CRP) was aimed to increase the social, economic, and environmental impacts of research. These programs have intentionally developed broader and deeper partnerships with a wide range of policy and development actors (i.e., international conservation and development organizations, NGOs, policy actors, other stakeholders), as well as with other researchers and research organizations. These efforts mirrored a shift in the broader research environment toward more engaged, problem-centred research. Such research, known variously as Transdisciplinary Research (TDR), Mode 2 Research, and Sustainability Science, among other terms, actively involves stakeholders to help ensure the relevance of the research, incorporate a broader range of expertise in the research process, and promote the co-generation of knowledge with research users.

In theory, engaged TDR approaches should help address complex sustainability problems and contribute to more and better outcomes. However, the increased complexity of these approaches makes impact assessment even more challenging than for traditional research approaches. Research impact assessment is chronically challenged by the fact that the uptake and use of research-based knowledge is incremental, with multiple steps and other intervening factors, often with long time-lags. Measuring and attributing impact are difficult. CGIAR research impact assessment has typically attempted to measure the benefits of improved technologies generated by CGIAR research; this assumes that the main contributions of the research are bundled within an improved plant variety or other technology package. TDR deliberately aims to contribute to several impact pathways simultaneously, by supporting capacity-building and empowerment among partners, facilitating dialogue and political processes, co-generating knowledge that will be implemented directly by partners, as well as through more conventional research products. However, empirical evidence of whether and how transdisciplinary approaches contribute to (more) effective scientific and social outcomes remains limited.

CIFOR Senior Associate Scientist Brian Belcher and his team in the Sustainability Research Effectiveness Program (SRE) at Royal Roads University have developed methods to assess TDR. The SRE Program has also conducted a series of case studies of completed FTA research projects to investigate the link between transdisciplinary research and societal effects. They recently published two papers to share lessons from their work.

A refined method for theory-based evaluation of the societal impacts of research [pdf]
A refined method for theory-based evaluation of the societal impacts of research” (Belcher et al., 2020) provides a detailed description of concepts and a method for assessing the relationship between research processes, outputs, and outcomes. The Outcome Evaluation Approach uses an actor-centred Theory of Change as the analytical framework, and accounts for complexity by recognizing the role of other actors, context, and external processes in change. The article provides stepwise guidance on how to:

  • document a theory of change;
  • determine data needs and sources;
  • collect data;
  • manage and analyze data; and
  • present findings.

 

The paper responds to the need for appropriate methods to demonstrate (for accountability) and analyze (for learning) whether and how research projects contribute to change processes, in an effort to make research more effective in addressing complex sustainability challenges.

Linking transdisciplinary research characteristics and quality to effectiveness [pdf]
Linking Transdisciplinary Research Characteristics and Quality to Effectiveness: A Comparative Analysis of Five Research-for-Development Projects” (Belcher et al., 2019) reports lessons from outcome evaluations [1] of five FTA projects. The five projects:

  1. Brazil Nut Project (BNP)
  2. Sustainable Wetlands Adaptation and Mitigation Program (SWAMP)
  3. Fire and Haze Indonesia (F&H)
  4. Global Comparative Study on Forest Tenure Reform-Peru (GCS-FTR), and
  5. Support to the Development of Agroforestry Concessions in Peru (SUCCESS)

 

 

represent a wide range of research approaches, social and policy contexts, and outcomes. Each case study used the Outcome Evaluation Approach described in Belcher et al. (2020) to document the project’s Theory of Change and assess whether and how outcomes were realized. The analysis also used Belcher et al.’s (2016) Transdisciplinary Research Quality Assessment Framework (QAF) to characterize each project by the degree to which its design and implementation conformed with transdisciplinary criteria.

Each project had a deliberate focus on moving beyond knowledge production to influence policy and practice. To do that, the projects employed a variety of strategies that crossed disciplinary bounds and engaged a range of partners and stakeholders at different levels. The results demonstrate that projects employing more transdisciplinary characteristics make more diverse contributions as they tend to leverage more diverse mechanisms of change. The participation of various system actors contributed to projects’ relevance and strongly contributed to the uptake and use of the research. Projects that invested most in developing and facilitating participation (e.g., the Global Comparative Study on Forest Tenure Reform-Peru and the Support to the Development of Agroforestry Concessions in Peru projects) were the most successful in generating social learning and building coalitions. Projects that employed the most traditional scientific models (e.g., the Brazil Nut Project and the Sustainable Wetlands Adaptation and Mitigation Program) but still invested in outreach and engagement, were able to realize significant outcomes. Research project efforts to support social processes helped translate and broker knowledge outputs and made substantial additional contributions through capacity-building, initiating and supporting discourse, and relationship-building.

Given the results, it is clear that research aiming to influence policy and practice change should consider integrating and reflecting on TDR characteristics more intentionally from the early planning stages and throughout the whole research process. This new Outcome Evaluation Approach will help linking outcomes, outputs and TDR more effectively, justifying the need for more transdisciplinary science, with an increase in overall results and global benefits.

[1] Two individual  project outcome evaluation reports have been published (Brazilian Nut, SWAMP), while the others are forthcoming (F&H, GCS-FTR, SUCCESS).


FTA is the world’s largest research for development program to enhance the role of forests, trees and agroforestry in sustainable development and food security and to address climate change. CIFOR leads FTA in partnership with Bioversity International, CATIE, CIRAD, INBAR, ICRAF and TBI. FTA’s work is supported by the CGIAR Trust Fund.

  • Home
  • Ensuring quality of research for development: The MELIA system

Ensuring quality of research for development: The MELIA system

Posted by

FTA COMMUNICATIONS TEAM

The CGIAR Research Program on Forests, Trees and Agroforestry (FTA) is an integrated global research initiative that aims to enhance the use, management and governance of forests, agroforestry and tree genetic resources as a way to improve livelihoods and the integrity of the environment. To test methods, approaches, partnerships and engagement strategies, and to seek the most effective means of achieving positive change, the program uses an innovative system to ensure the quality of its research, to monitor, evaluate and assess the outcomes (defined as changes in technical, social and economic behavior) and impact (defined as changes in actual environmental quality and human wellbeing) of its work.

  • Home
  • Strengthening the impact culture of research on forests, trees and agroforestry

Strengthening the impact culture of research on forests, trees and agroforestry

CIFOR researchers assess the nature of economic losses likely to be incurred through displacement of chitemene fields from large-scale jatropha plantations. ©Center For International Forestry Research/Jeff Walker
Posted by

FTA

 

Photo: Jeff Walker/CIFOR

By Brian Belcher, Senior Associate, Center for International Forestry Research (CIFOR)

In the new phase of the CGIAR Research Program on Forests, Trees and Agroforestry, the previous work under “Monitoring, Evaluation and Impact Assessment” has been re-labelled as “Monitoring, Evaluation, Learning and Impact Assessment” (MELIA).

FTA is developing its MELIA unit as a core research and support unit with team members from each of the participating centers. In the new phase, FTA has a great opportunity to make significant theoretical, methodological and empirical contributions to help improve research quality and research effectiveness within FTA, the CG, and beyond.

The MELIA work will be organized in four main clusters of activities:

Foresight/ex ante impact assessment

This is a new area for FTA, in which we will develop tools and approaches for assessing strategic opportunities and estimating potential impact to help inform priority setting and planning. In 2017, MELIA will develop our approach, in collaboration with other CGIAR Research Programs such as Policies, Institutions and Markets and Roots, Tubers and Bananas and in conjunction with research priority setting work in the FTA Management Support Unit.

Ex post impact assessment

MELIA will use and adapt experimental and quasi-experimental methods to assess the impact especially of technical or policy interventions. In 2017, for example, FTA will undertake an ex post impact assessment of the Agroforestry for Food Security Program in Malawi with support from the CGIAR Standing Panel on Impact Assessment.


Read also: Caught in a good loop – how to assess the usefulness of research


Outcome evaluation

MELIA will continue to develop, test and refine theory-based research evaluation methodology to assess research contributions in complex systems and to build a series of outcome evaluation case studies and a longer-term comparative analysis of cases. In 2017, at least four evaluations will be completed and four new studies will be launched. For example, an outcome evaluation of a gender-specific project focusing on women’s participation in forest management in Uganda will help improve the sensitivity of MELIA tools and strengthen FTA’s theory of gender transformative change in FTA landscapes. MELIA will also explore opportunities for collaboration with CGIAR Research Program on Climate Change, Agriculture and Food Security(CCAFS) and other partners.

Indicators, monitoring & reporting

This work will support the ongoing development of indicators, indicator frameworks and data collection to support the five Flagships in their reporting of outputs, outcomes and indicators for monitoring and reporting. Overall, this work will foster an impact culture within FTA, help ensure that FTA’s work remains relevant and useful in rapidly changing and complex circumstances, learn lessons from the rich FTA experience and guide ongoing research, engagement and capacity development to maximize effectiveness. In 2017, MELIA will support a FTA-wide work to revise the set of FTA indicators and milestones.

Key outcomes of MELIA’s work in 2017 will be

  • improved capacity and improved design, monitoring, evaluation and learning (DMEL);
  • strengthened impact culture within FTA;
  • stronger empirical evidence of contributions of FTA research; and
  • better, more transparent estimates of potential impact leading to well-informed strategies and improved donor confidence.

Back to top

Sign up to our monthly newsletter

Connect with us

X