Tag: evaluation


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • FTA Highlight No. 17 – Monitoring, Evaluation, Learning and Impact Assessment

FTA Highlight No. 17 – Monitoring, Evaluation, Learning and Impact Assessment


Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
Posted by

FTA communications

Monitoring, Evaluation, Learning and Impact Assessment (MELIA) has been a key focus of the CGIAR Research Program on Forests, Trees and Agroforestry (FTA) over its decade of work.

FTA’s MELIA Program has adapted, developed, tested and refined a comprehensive set of concepts and methods. These include participatory theories of change and developing and refining outcome evaluation methods.

They also include systematically reviewing, defining and assessing the quality of research that crosses disciplinary boundaries; and developing a quality assessment framework suitable for research for development (R4D).

This decadal work has generated a body of research and valuable lessons about research design, implementation, monitoring and evaluation within FTA and beyond.

Download the volume! [PDF]
As part of “FTA’s highlights of a decade,” a new series focusing on its main results since its inception in 2011, FTA is now publishing the volume on Monitoring, Evaluation, Learning and Impact Assessment (MELIA).

The CGIAR Research Programs (CRPs) embodied a fundamental shift in the approach to research for development (R4D). CRPs assume shared responsibility for achieving economic and human development outcomes, a shift that required new ways of monitoring and evaluating research (R4D).

When the Forests, Trees and Agroforestry (FTA) program was established in 2011 it responded to this approach with a plan to develop and use theory-based approaches for monitoring and evaluating outcomes and impacts.

PHOTO GALLERY

Over its ten years FTA has recognized the importance of monitoring and evaluation, not only as an important management tool, but also as a field of research in its own right.

From the beginning, FTA’s Monitoring, Evaluation, Learning and Impact Assessment (MELIA) strategy included a core research component. FTA’s impact orientation and MELIA work were strongly supported by FTA governance. More engaged, transdisciplinary approaches to research involve stakeholders in order to deal with complexity and increase potential impact.

FTA’s research approaches aim to contribute to reduced poverty, improved food security and nutrition, and improved management of natural resources and ecosystem services through technical, institutional and policy innovation. Monitoring is a key element of FTA’s adaptive, learning-oriented approach, from the project level to the program level.

Research activities operate within a program-level theory of change (ToC), and each activity has its own particular context, design and implementation, and specific ToC.

This multi-level approach creates an excellent opportunity for learning how research contributes to transformative change within complex social and environmental systems.

MELIA has focused its work on five key global challenges.

  1. accelerating rates of deforestation and forest degradation;
  2. high prevalence of degraded land and ecosystem services;
  3. unsustainable land-use practices;
  4. persistent rural poverty, with increasing levels of vulnerability;
  5. rising demand for and need for nutritious food.

FTA has developed a set of user-friendly and time-efficient monitoring tools for use at the project scale. The tools facilitate systematic collection of data on engagement with stakeholders, knowledge generation, uptake and use, and progress toward higher-level outcomes and impacts.

FTA’s MELIA team also contributed to the development of the Quality of Research for Development (QoR4D) framework. The framework guides and enhances research at all levels, and contributes to outcomes and impacts.

Download the publication to learn more about FTA’s approach to monitoring, FTA’s integrated outcome evaluations, case studies, results and lessons learned. FTA’s work on MELIA sets the ground for future initiatives to build and design projects integrating theory-based evaluation towards a transdisciplinary model of research impact.

Published volumes until today include:

Latest post


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • How does FTA’s oil palm research in Indonesia measure up?

How does FTA’s oil palm research in Indonesia measure up?


Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
Oil palm plantation at the border of a intact forest. Jambi, Indonesia. Photo by Iddy Farmer/CIFOR
Posted by

FTA communications

The Forests, Trees and Agroforestry research program (FTA) of the CGIAR has just released an extensive evaluation of its research portfolio on oil palm in Indonesia, which assesses the influence of the work on policy, practice, partnerships, and research within the sector.

FTA Evaluation report on Oil Palm Portfolio [PDF]
Oil palm production in Indonesia is “in many ways emblematic of key economic, social, and environmental challenges and opportunities,” the report’s authors state. The country is a major oil palm producer, and the sector has experienced rapid growth in demand – and rapid expansion of oil palm plantations – in recent years.

While this growth has contributed considerably to the country’s income, it has also been linked to severe environmental impacts, such as increased instances of fires, deforestation and peat exploitation, resulting in carbon emissions and biodiversity loss. There are also social challenges as the industry grows and becomes more regulated, such as smallholder disenfranchisement.

In that context, FTA’s leading partner, the Center for International Forestry Research (CIFOR), has been carrying out a number of research projects in Indonesia since 2010 to better understand Indonesia’s governance of and policy process for oil palm management; the biophysical characteristics and ecological implications of oil palm production; and the social realities of oil palm expansion.

Now, FTA’s Monitoring, Evaluation, Learning and Impact Assessment team (MELIA) has released an evaluation of that research. MELIA chose four key projects, valued at around USD7 million of investment, to represent the portfolio: Supporting Local Regulations for Sustainable Oil Palm in East Kalimantan (EK), Governing Oil Palm Landscapes for Sustainability (GOLS), Oil Palm Adaptive Landscapes (OPAL), and Engendering Roundtable for Sustainable Palm Oil (RSPO) Standards (ERS).

The evaluation assessed the project design, implementation and outcomes of those initiatives. “The objective was to assess the extent to which CIFOR’s research on palm oil has influenced policy and practice in Indonesia – and to learn from that,” said evaluation manager and CIFOR Research to Impact team leader Jean-Charles Rouge.

Portfolio’s Composite Theory of Change

The team used a new Theory of Change (ToC) as the analytical framework for the evaluation. It combined key elements of the four projects to create a composite ToC, which sketched out the theoretical relationships and sequences of steps through which the portfolio aimed to contribute to particular outcomes and impacts. The portfolio’s ToC identifies a total of 21 outcomes at three distinct levels:

  • 6 end-of-project outcomes that are reasonable to expect and observable at the time of the evaluation, and therefore are testable;
  • 7 intermediate outcomes that relate to changes in knowledge, skills and relationships, and
  • 8 high-level outcomes to support the causal logic from end-of-project outcomes to impacts and project purpose.

The distinction between end-of-project and high-level outcomes is made because higher-level results are expected to require more time to manifest and depend on variables beyond the influence of the project. The evaluation then tested to what extent and how outcomes were realized, using document review and interviews with relevant stakeholders (see Table 1).

The evaluation constituted a significant undertaking in terms of data collection and validation. Over 200 documents were reviewed, 89 interviews were conducted, and a total of five sense-making workshops were held to validate and solicit additional input to the report with representatives from government, NGOs, partner organizations, researchers, and FTA.

“CIFOR had already used a theory-based evaluation approach in previous evaluations of research projects,” said Rouge. “But in this evaluation, for the first time, the approach was used for a portfolio of projects, which led to a number of methodological challenges – on top of navigating in a very politically sensitive sector. The evaluation team addressed them well and ensured that all stakeholders had an opportunity to voice their views”.

Rachel Davel, one of the evaluators from the Sustainability Research Effectiveness (SRE) Program at Royal Roads University, added that “this evaluation offered us an opportunity to work with a composite theory of change to represent the intended contributions of multiple research projects. This required additional validation from researchers of the four projects to ensure the composite accurately reflected the collective research efforts and outcomes of the portfolio. We have since transferred this learning to another study to document composite theories of change to assess the FTA program.”

Rachel Claus, another member of the SRE evaluation team, noted that the value of the theory-based evaluation method was the ability of the analytical framework to make explicit and test outcomes organized around collective research contributions. “Using a theory of change framework was helpful to identify where projects were well aligned to reinforce each other and build on outcomes, and where there was scope to be more strategic, coordinated, and intentionally designed for impact,” she said.

The evaluators found that the portfolio fully contributed to the realization of 12 of the 21 outcomes and partially contributed to six of them identified in the ToC. Some notable achievements include: shaping a provincial-level regulation in East Kalimantan to include high conservation areas; government partners building new expertise in oil palm topics; ally organizations using FTA findings to hold private palm oil companies accountable to their commitments; RSPO changing international oil palm certification standards to reflect gender considerations; skilling the next generation of young researchers; and advancing oil palm research.

Six of the seven intermediate outcomes, which related to changes in knowledge, skills, and relationships, were met. While there is an indication that the outcome related to the private sector learning from research was partially realized, the lack of evidence could not fully support this claim. The report explains how private sector engagement was a shortcoming of the portfolio. As a lesson, this could be a focal area for future research on oil palm that could support cross-sector engagement and progress toward sustainability in the sector.

Five of the six end-of-project outcomes were also realized, although some of the portfolio’s higher-level outcomes are yet to be met; many of these depend on factors and processes that are beyond the direct influence of the portfolio.

The portfolio mechanisms that proved to have most influence related to new knowledge production and the scientific reputation of CIFOR and its partners. For instance, the smallholder typology developed by GOLS contributed to raising awareness about the need to focus policy to give the right assistance to the right kind of smallholders. “Numerous interview respondents appreciated the neutral, credible, and research-based information that CIFOR can offer to support constructive dialogues and collective action in a contentious sector,” the report underlines.

“Overall, FTA’s research portfolio produced knowledge on diverse aspects of oil palm sustainability and contributed to notable changes in policy, partnerships, capacity-building, and the research agenda,” said Davel. The evaluation concludes that there is potential for future oil palm production in Indonesia to take place in ways that have both enhanced social benefits and fewer negative environmental impacts – if it is well-guided by government regulation, private sector commitments and research to inform sustainable and inclusive practices. However, the report also noted that there are currently a number of barriers to realizing this scenario, including competing political and economic priorities, and future legislation that could undermine sustainability initiatives in the sector.

Table 1. Summary of Outcome Assessment and Portfolio Contribution

  Expected Outcome Outcome Assessment and Portfolio Contribution
High-Level Outcomes Partner organizations use project research to inform planning decisions and project development H Realized, clear portfolio contribution
Smallholders and women have improved representation in policy-making around oil palm M Partially realized, clear portfolio contribution. Relies on some theoretical extrapolation of the implication of policy changes.
The research agenda on oil palm advances toward sustainability and inclusion M Partially realized, clear portfolio contribution
Policy-makers recognize and reflect environmental sustainability and social inclusion in all oil palm-related policy M Partially realized, clear portfolio contribution
Private sector adopts more sustainable and inclusive business models L Partially realized, clear portfolio contribution
Accumulation of scholarship on oil palm influences organizational practice L Not realized, too early to assess. There is no evidence of realization. It is possible that portfolio research could indirectly contribute to practice change via portfolio influence on policy change.
Smallholders and women have improved oil palm market access and share of benefits L Not realized, too early to assess. Realization of this outcome relies on the assumption that policies are effectively implemented and enforced.
The oil palm sector (governments, private sector, NGOs, smallholders, CIFOR) develops more effective working arrangements L Partially realized, unclear portfolio contribution. Few respondents explicitly identified how the processes facilitated by the portfolio contributed to more effective working arrangements.
Intermediate Outcomes Government actors learn from oil palm research processes and findings H Realized, clear portfolio contribution

 
Government actors build their capacities and relationships within the oil palm sector M Realized, clear project contribution

 
Government actors engage CIFOR & partners to help make informed decisions on oil palm M Realized, clear project contribution

 
Project partnerships facilitate mutual learning on oil palm H Realized, clear portfolio contribution
Graduate students build their research capacities H Realized, clear portfolio contribution
CIFOR & partners are recognized for expertise in oil palm research H Realized, clear portfolio contribution
Private sector actors learn from oil palm research L Insufficient evidence, preliminary results indicate partial realization with clear portfolio contribution
End-of-Project Outcomes Policy-makers create new or adapt existing policy on oil palm (informed by research)

[EoP outcome]

 M Realized, clear portfolio contribution

 
Project partners & allies advocate for & pursue oil palm issues (using research)

[EoP outcome]

 H Realized, clear portfolio contribution
Graduate students continue careers in oil palm research to build on knowledge base

[EoP outcome]

 H Realized, clear portfolio contribution
Researchers use projects’ findings and methods

[EoP outcome]

 H Realized, clear portfolio contribution
Researchers pursue new questions on oil palm

[EoP outcome]

 M Realized, clear portfolio contribution
Private sector responds to (research-informed) policy change

[EoP outcome]

 L Partially realized, clear portfolio contribution

Evidence Rating: Low (L), Medium (M), High (H)

This article was written by Monica Evans.

This article was produced by the CGIAR Research Program on Forests, Trees and Agroforestry (FTA). FTA is the world’s largest research for development program to enhance the role of forests, trees and agroforestry in sustainable development and food security and to address climate change. CIFOR leads FTA in partnership with ICRAF, the Alliance of Bioversity International and CIAT, CATIE, CIRAD, INBAR and TBI. FTA’s work is supported by the CGIAR Trust Fund.

 

Latest post


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • Independent evaluation shows FTA's progress towards targets

Independent evaluation shows FTA’s progress towards targets


Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
Posted by

FTA communications

In 2020, the CGIAR Advisory Services Shared Secretariat (CAS) commissioned independent reviews of the CGIAR Research Programs (CRPs). The evaluation of the Forests, Trees and Agroforestry CRP was completed in December 2020 and is now available on the CAS website  https://cas.cgiar.org/evaluation/crp-2020-fta.

CAS Secretariat’s evaluation of FTA [PDF]
The purpose of the review was to assess the extent to which FTA is delivering quality of science and demonstrating effectiveness in relation to its theory of change. The review focuses on activities and results of the second phase of FTA, from 2017 to 2019.

According to the review, FTA showed high scientific productivity and strong implementation performance in phase II and is likely to make significant progress toward most of its planned end-of-program targets.

The review found that the close collaboration between FTA partners, its independent and efficient governance, and the effective prioritization and management of resources resulted in a high level of programmatic value-added. Strong partnerships with universities and research institutions, and collaboration between scientists were also found to have strengthened the CRP.

Going forward, the review recommended that the most important impact pathways for FTA should continue to be: its positive influence on government and on international policy processes and that the program should find ways to conserve and protect the significant value-added it has built beyond 2021.

The FTA Independent Steering Committee and Management Team have released a letter on their perspectives following this extensive independent review. You may read their letter here.

Download

 

This article was produced by the CGIAR Research Program on Forests, Trees and Agroforestry (FTA). FTA is the world’s largest research for development program to enhance the role of forests, trees and agroforestry in sustainable development and food security and to address climate change. CIFOR leads FTA in partnership with ICRAF, the Alliance of Bioversity International and CIAT, CATIE, CIRAD, INBAR and TBI. FTA’s work is supported by the CGIAR Trust Fund.

Latest post


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • New website showcases Forests, Trees and Agroforestry topics from the FTA 2020 Science Conference

New website showcases Forests, Trees and Agroforestry topics from the FTA 2020 Science Conference


Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
Posted by

FTA communications

Are you looking for some of the key latest research findings, keynotes, videos regarding trees, forests and agroforestry contributions to sustainable development? Go to the newly launched web-portal of the groundbreaking FTA Science Conference 2020 by the CGIAR Research Program on Forests, Trees and Agroforestry (FTA).

The fully digital conference, titled Forest, trees and agroforestry science for transformational change, ran from 14 to 25 September 2020 and drew more than 520 participants from 69 countries around the world. It featured close to 200 interventions from scientists involved in the FTA program spread over 10 days and 26 different sessions. It included keynote speeches, controversial panel debates on “hot topics”, and technical presentations and posters.

FTA 2020 Science Conference Book of Abstracts now available in PDF

The conference put an emphasis on collaborative research between FTA and the broader community, as 60% of the presentations were between FTA’s seven managing partners (CIFOR, ICRAF, The Alliance of Bioversity and CIAT, CATIE, CIRAD, INBAR and TBI) as well as the many national partners. The 179 abstracts accepted for the event are now made available in a book on the new web-portal, with more coming next, such as selected videos.

The conference was organized around six key technical themes that are pathways for transformational change:

  1. Inclusive value chains, finance and investments
  2. Towards resilient and diverse landscapes and food systems
  3. Transforming livelihoods through agroecological approaches with trees
  4. Nature-based solutions to address the climate crisis
  5. Inclusive governance for sustainable landscapes
  6. Designing, implementing and evaluating research for development impact

Three plenary sessions allowed for overall framing, linking-up across themes, stock-taking of discussions. The conference featured two sessions addressing “Hot & Controversial” issues, be it in science, in development, or in the media:

  • Competing understandings of the restoration problem and solutions
  • Systemic approaches in a ‘silver bullets’ world.

Restoration has emerged in the last decade as a key global political objective and debates on the topic are intense. The “Hot & Controversial” session used a variety of techniques, including role-playing, quick polls and devil’s advocacy, to highlight and debate some of the most disputed points, allowing to discuss strengths and shortcomings of the argumentations behind, and to debunk myths.

An innovative “Green” Dragons’ Den event was organized for the second “Hot & Controversial” session, to trial five innovations coming from the program. These were defended by their authors in quick elevator pitches, trying to convince the Green philanthropist dragons to invest a “virtual” sum of three million USD. The audience was also called to a virtual crowdfunding exercise. The session was a “live learning” event, for scientists to get better at telling convincing stories on often very complex issues and tools, to best sell their results, as well as understand needs, objectives, and ways of thinking of investors.

***

It was the second time FTA organized a global virtual conference, after the first one held in March 2017 on “cool insights for a hot world”, that gathered 200 participants over two days.

For the 2020 conference, technical developments, including live (“synchronous”) online collaborative tools such as Mural, virtual poster rooms, live polling, role-playing sessions, and the experience of FTA’s events team, allowed for a lively and smoothness event, marking probably a new era for large scale scientific conferencing.

Participation from within the program was double the size of what it would have been if held in-person, and several high-level stakeholders could join for engagement sessions, for which otherwise they may not have been able to travel for a full week. Also, with 3 hours of “air time” per day, it left participants still with time for their other activities, while allowing participation from scientists in time zones situated 15 hours apart, from Vancouver to Hobart.

As a follow-up, the FTA is now organizing a series of “Science to Action” webinars, which are open to all, and which will focus on the way forward for actors on the ground. The first webinar was held on 26 November 2020 on the topic: Innovations to overcome barriers to access to finance for smallholders, SMEs, and women, and was developed in coordination with FTA partner Tropenbos International. You can replay the whole event here.

***

Looking forward to engaging even more in 2021, as we wrap up a full decade of research since 2011.

By Sandra Cordon.

This article was produced by the CGIAR Research Program on Forests, Trees and Agroforestry (FTA). FTA is the world’s largest research for development program to enhance the role of forests, trees and agroforestry in sustainable development and food security and to address climate change. CIFOR leads FTA in partnership with ICRAF, the Alliance of Bioversity International and CIAT, CATIE, CIRAD, INBAR and TBI. FTA’s work is supported by the CGIAR Trust Fund.

Latest post


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • A joint stocktaking of CGIAR work on forest and landscape restoration by FTA, PIM and WLE

A joint stocktaking of CGIAR work on forest and landscape restoration by FTA, PIM and WLE


Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
Posted by

FTA communications

Despite the high level of political engagement and the wide range of organizations involved in restoration projects from local to global levels, beyond some success stories, restoration is not happening at scale. Research is urgently needed to design, develop and upscale successful restoration approaches. As part of this effort, FTA, PIM and WLE publish a synthesis of a survey of CGIAR’s projects on restoration.

Forest and landscape restoration (FLR) have gained traction on the political agenda over the past decades with the multiplication of pledges and commitments on restoration such as  the Bonn Challenge (2011), the New York Declaration on Forests (2014) and other global or regional initiatives. On March 1st, 2019, the United Nations General Assembly declared 2021-2030 the UN Decade on Ecosystem Restoration (Resolution A/RES/73/284).

This UN Decade could offer unprecedented opportunities to address food security, job creation and climate change simultaneously. The UN Environment Programme (UNEP) considers that restoring 350 million hectares (ha) of degraded land by 2030, as committed in the New York Declaration on Forests, could generate USD 9 trillion in various ecosystem services and remove about 13–26 gigatons of greenhouse gases from the atmosphere.

However, despite the high level of political engagement and the wide range of institutions (public, private or civil society; local to global) involved in restoration projects, and beyond some success stories, restoration is not happening at scale.

In 2018, starting with a joint workshop, three CGIAR Research Programs (CRPs) – Forests, Trees and Agroforestry (FTA); Policies, Institutions and Markets (PIM) and Water, Land and Ecosystems (WLE) – decided to strengthen their collaboration to address this issue by bringing together different research streams working on soil, water and forest restoration.

“There are huge opportunities in bringing the three CRPs together to work on land restoration. Each of these CRPs works on different aspects of land restoration. Pooling this evidence in a user-friendly and accessible manner holds great potential for scaling, and for delivering enhanced impact from our CGIAR research” said Vincent Gitz, Izabella Koziell and Frank Place, the three CRP directors.

The three CRPs agreed on a broad scope of restoration, focused on the restoration of “ecological functions”, with the following definitions:

Degradation: Loss of functionality of e.g. land or forests, usually from a specific human perspective, based on change in land cover with consequences for ecosystem services

Restoration: Efforts to halt ongoing and reverse past degradation, by aiming for increased functionality (not necessarily recovering past system states).

They also discussed theories of [induced] changes underlying landscape dynamics of degradation and restoration. The following questions helped structure the discussions:

  • Why? What are the final goals of restoration efforts, which sustainable development goals can they contribute to?
  • What? What are the drivers of degradation that need to be addressed? What are the ecological functions to be restored?
  • Who? Who cares? Who are the stakeholders responsible for or impacted by land degradation? How stakeholders are encouraged, empowered and organized to act for forest and landscape restoration?
  • How? How to design effective restoration interventions? What are the land use and land management options for change in different contexts, across countries and biomes?
  • Where and when? How to operationalize action recognizing the connectivity across different spatial and temporal scales in the restoration process, considering the landscape’s spatial configuration and temporal dynamics?

A joint stocktaking of CGIAR work on forest and landscape restoration
[Download the report in PDF]

As a first step of their collaboration, the 3 CRPs (FTA, PIM, WLE) conducted a broad survey of the CGIAR’s work on restoration, inviting contributions from other CRPs. The document published today is a synthesis of the survey results. The full database with full details on each initiative is available as an annex.

The survey reflects the implication of different CGIAR Centers (ICRAF, Bioversity, CIFOR, CIAT, IWMI, ILRI, ICRISAT, CIMMYT and IFPRI) in restoration projects across the tropics and sub-tropics, in Africa, Asia, and Latin America. Some countries, such as Ethiopia, Kenya, Peru, or Indonesia concentrate many projects and provide strong opportunities for further collaboration among the three CRPs.

The survey shows the wide range of restoration activities undertaken by CGIAR CRPs and Centers, with their partners, from knowledge generation, methods, planning, modelling, assessment and evaluation, monitoring and mapping, to action on the ground. CGIAR restoration work can be divided into three broad categories: (1) case studies and projects; (2) tools for development; (3) approaches and conceptual frameworks.

The first category gathers case studies and projects comprising an element of field research. It comprises experimental plots, trials, local capacity building and implementation, on-the-ground assessments and surveys at different scales. It distinguishes: (i) “restoration-focused projects” where forest and land degradation is the main entry point and restoration is the main objective; from, (ii) “restoration-related projects” that can contribute to forest and landscape restoration while following other objectives (such as sustainable intensification or climate-smart agriculture). Half of the “restoration-focused” projects aim at assessing restoration practices with the view to upscale successful restoration experiences, such as the Ngitili fodder management system which contributed to the restoration of up to 270,000 ha over about 25 years in Shinyanga region, Tanzania. The others focus on climate change and climate-smart restoration, or on desertification and sand fixation. Six projects in this category focus on genetic diversity and on the performance and organization of the seed supply system, identified in this survey as a critical factor of success for restoration interventions. “Restoration-related projects” focus on various topics closely linked to restoration, including: sustainable land and water management; climate-smart agriculture; land tenure security and land governance reform; participatory governance and planning and collective farming.

The second category regroups: (i) tools, methods and guidelines, directed at decision makers or restoration practitioners at different levels, to support decision making; as well as, (ii) maps and models, measuring at different scales the intensity of degradation (i.e. efforts needed for restoration) or modeling the impacts of different land-use changes or land management practices. Models and maps often serve as the first layer for decision-making supporting tools. This category includes for instance two entries on the Land Degradation Surveillance Framework (LDSF), developed by ICRAF and applied, since 2005, in over 250 landscapes (100 km2 sites) across more than 30 countries. Using indicators such as vegetation cover, structure and floristic compositions, tree and shrub biodiversity, historic land use, visible signs of land degradation, and physical and chemical characteristics of soil (including soil organic carbon content and infiltration capacity), the LDSF, applicable to any landscape, provides a field protocol for assessing soil and ecosystem health to help decision makers to prioritize, monitor and track restoration interventions.

The third category, covering more theoretical work, includes: (i) evaluations, conceptual or theoretical frameworks around restoration and related issues; and (ii) systematic literature and/or project reviews, as well as meta-analyses on different topics linked to restoration. For instance a global survey on seed sourcing practices for restoration, was realized between 2015 and 2017 by Bioversity International, reviewing 136 restoration projects across 57 countries, and suggesting a typology of tree seed sourcing practices and their impact on restoration outcomes (Jalonen et al., 2018).

The survey describes projects operating at the landscape level or across multiple scales. This shows the importance of the landscape level to effectively combine integrated perspectives that allow synergies among different ecosystem components and functions with a deep knowledge of, and a fine adaptation to, local conditions. While many projects focus on the technical performance of restoration projects, relatively few investigate the economics, cost and benefits, of restoration and few examine their underlying power structures and power dynamics/games. This relative paucity of costs and benefit data has been noted by other organizations, an aspect that led to the launch of the FAO-led TEER initiative, to which FTA and several CGIAR centers contribute.

All the answers taken together provide useful insights for future restoration activities. In particular, they identify five critical factors of success for restoration interventions:

  1. secure tenure and use rights;
  2. access to markets (for inputs and outputs) and services;
  3. access to information, knowledge and know-how associated with sustainable and locally adapted land use and land management practices;
  4. awareness of the status of local ecosystem services, often used as a baseline to assess the level of degradation; and
  5. (v) high potential for restoration to contribute to global ecosystem services and attract international donors.

This synthesis will inform future work of FTA, PIM and WLE. It can also be used to support the design of restoration activities, programs and projects. Finally, it also illustrates with concrete examples the powerful contribution of forest and landscape restoration to the achievement of many, if not all of the 17 sustainable development goals. In particular, forest and landscape restoration, through the recovery of a range of ecological functions, can contribute to:

  • enhance food security through the improvement of the ecosystem services sustaining agriculture at landscape scale
  • improve natural resource use efficiency, thus reducing the pressure on the remaining natural habitats and addressing water scarcity;
  • favour social justice by securing a more equitable access to natural resources (e.g. land, water and genetic resources), and a wider participation in decision-making processes, in particular for women and marginalized people; and,
  • strengthen ecosystem, landscape and livelihoods resilience to economic shocks and natural disasters in a context of climate change.

The COVID 19 crisis has shown the importance of healthy ecosystems for healthy and resilient economies and societies. We hope that this document will contribute to integrate restoration as part of the efforts to “build back better” after the crisis.

This article was produced by the CGIAR Research Program on Forests, Trees and Agroforestry (FTA). FTA is the world’s largest research for development program to enhance the role of forests, trees and agroforestry in sustainable development and food security and to address climate change. CIFOR leads FTA in partnership with Bioversity International, CATIE, CIRAD, INBAR, ICRAF and TBI. FTA’s work is supported by the CGIAR Trust Fund.

Latest post


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • Outcome Evaluation Approach – 5 Case Studies from FTA

Outcome Evaluation Approach – 5 Case Studies from FTA


Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
Yordana Yawate, carries a sack of sago pith to be filtered on the banks of the Tuba river in Honitetu village, Maluku province, Indonesia. Photo by Ulet Ifansasti/CIFOR
Posted by

FTA communications

Two recent publications discuss how to effectively assess the impact of transdisciplinary (TDR) research and apply these methods to 5 case studies.

The creation of the CGIAR Research Programs (CRP) was aimed to increase the social, economic, and environmental impacts of research. These programs have intentionally developed broader and deeper partnerships with a wide range of policy and development actors (i.e., international conservation and development organizations, NGOs, policy actors, other stakeholders), as well as with other researchers and research organizations. These efforts mirrored a shift in the broader research environment toward more engaged, problem-centred research. Such research, known variously as Transdisciplinary Research (TDR), Mode 2 Research, and Sustainability Science, among other terms, actively involves stakeholders to help ensure the relevance of the research, incorporate a broader range of expertise in the research process, and promote the co-generation of knowledge with research users.

In theory, engaged TDR approaches should help address complex sustainability problems and contribute to more and better outcomes. However, the increased complexity of these approaches makes impact assessment even more challenging than for traditional research approaches. Research impact assessment is chronically challenged by the fact that the uptake and use of research-based knowledge is incremental, with multiple steps and other intervening factors, often with long time-lags. Measuring and attributing impact are difficult. CGIAR research impact assessment has typically attempted to measure the benefits of improved technologies generated by CGIAR research; this assumes that the main contributions of the research are bundled within an improved plant variety or other technology package. TDR deliberately aims to contribute to several impact pathways simultaneously, by supporting capacity-building and empowerment among partners, facilitating dialogue and political processes, co-generating knowledge that will be implemented directly by partners, as well as through more conventional research products. However, empirical evidence of whether and how transdisciplinary approaches contribute to (more) effective scientific and social outcomes remains limited.

CIFOR Senior Associate Scientist Brian Belcher and his team in the Sustainability Research Effectiveness Program (SRE) at Royal Roads University have developed methods to assess TDR. The SRE Program has also conducted a series of case studies of completed FTA research projects to investigate the link between transdisciplinary research and societal effects. They recently published two papers to share lessons from their work.

A refined method for theory-based evaluation of the societal impacts of research [pdf]
“A refined method for theory-based evaluation of the societal impacts of research” (Belcher et al., 2020) provides a detailed description of concepts and a method for assessing the relationship between research processes, outputs, and outcomes. The Outcome Evaluation Approach uses an actor-centred Theory of Change as the analytical framework, and accounts for complexity by recognizing the role of other actors, context, and external processes in change. The article provides stepwise guidance on how to:

  • document a theory of change;
  • determine data needs and sources;
  • collect data;
  • manage and analyze data; and
  • present findings.

 

The paper responds to the need for appropriate methods to demonstrate (for accountability) and analyze (for learning) whether and how research projects contribute to change processes, in an effort to make research more effective in addressing complex sustainability challenges.

Linking transdisciplinary research characteristics and quality to effectiveness [pdf]
“Linking Transdisciplinary Research Characteristics and Quality to Effectiveness: A Comparative Analysis of Five Research-for-Development Projects” (Belcher et al., 2019) reports lessons from outcome evaluations [1] of five FTA projects. The five projects:

  1. Brazil Nut Project (BNP)
  2. Sustainable Wetlands Adaptation and Mitigation Program (SWAMP)
  3. Fire and Haze Indonesia (F&H)
  4. Global Comparative Study on Forest Tenure Reform-Peru (GCS-FTR), and
  5. Support to the Development of Agroforestry Concessions in Peru (SUCCESS)

 

 

represent a wide range of research approaches, social and policy contexts, and outcomes. Each case study used the Outcome Evaluation Approach described in Belcher et al. (2020) to document the project’s Theory of Change and assess whether and how outcomes were realized. The analysis also used Belcher et al.’s (2016) Transdisciplinary Research Quality Assessment Framework (QAF) to characterize each project by the degree to which its design and implementation conformed with transdisciplinary criteria.

Each project had a deliberate focus on moving beyond knowledge production to influence policy and practice. To do that, the projects employed a variety of strategies that crossed disciplinary bounds and engaged a range of partners and stakeholders at different levels. The results demonstrate that projects employing more transdisciplinary characteristics make more diverse contributions as they tend to leverage more diverse mechanisms of change. The participation of various system actors contributed to projects’ relevance and strongly contributed to the uptake and use of the research. Projects that invested most in developing and facilitating participation (e.g., the Global Comparative Study on Forest Tenure Reform-Peru and the Support to the Development of Agroforestry Concessions in Peru projects) were the most successful in generating social learning and building coalitions. Projects that employed the most traditional scientific models (e.g., the Brazil Nut Project and the Sustainable Wetlands Adaptation and Mitigation Program) but still invested in outreach and engagement, were able to realize significant outcomes. Research project efforts to support social processes helped translate and broker knowledge outputs and made substantial additional contributions through capacity-building, initiating and supporting discourse, and relationship-building.

Given the results, it is clear that research aiming to influence policy and practice change should consider integrating and reflecting on TDR characteristics more intentionally from the early planning stages and throughout the whole research process. This new Outcome Evaluation Approach will help linking outcomes, outputs and TDR more effectively, justifying the need for more transdisciplinary science, with an increase in overall results and global benefits.

[1] Two individual  project outcome evaluation reports have been published (Brazilian Nut, SWAMP), while the others are forthcoming (F&H, GCS-FTR, SUCCESS).

FTA is the world’s largest research for development program to enhance the role of forests, trees and agroforestry in sustainable development and food security and to address climate change. CIFOR leads FTA in partnership with Bioversity International, CATIE, CIRAD, INBAR, ICRAF and TBI. FTA’s work is supported by the CGIAR Trust Fund.

Latest post


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • Ensuring quality of research for development: The MELIA system

Ensuring quality of research for development: The MELIA system

05 October, 2017

Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
Posted by

FTA COMMUNICATIONS TEAM

The CGIAR Research Program on Forests, Trees and Agroforestry (FTA) is an integrated global research initiative that aims to enhance the use, management and governance of forests, agroforestry and tree genetic resources as a way to improve livelihoods and the integrity of the environment. To test methods, approaches, partnerships and engagement strategies, and to seek the most effective means of achieving positive change, the program uses an innovative system to ensure the quality of its research, to monitor, evaluate and assess the outcomes (defined as changes in technical, social and economic behavior) and impact (defined as changes in actual environmental quality and human wellbeing) of its work.

Latest post


Notice: Undefined index: id in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 3
  • Home
  • Strengthening the impact culture of research on forests, trees and agroforestry

Strengthening the impact culture of research on forests, trees and agroforestry


Notice: Undefined variable: id_overview in /home/ft4user/foreststreesagroforestry.org/wp-content/themes/FTA/template-parts/content.php on line 64
CIFOR researchers assess the nature of economic losses likely to be incurred through displacement of chitemene fields from large-scale jatropha plantations. ©Center For International Forestry Research/Jeff Walker
Posted by

FTA

 

Photo: Jeff Walker/CIFOR

By Brian Belcher, Senior Associate, Center for International Forestry Research (CIFOR)

In the new phase of the CGIAR Research Program on Forests, Trees and Agroforestry, the previous work under “Monitoring, Evaluation and Impact Assessment” has been re-labelled as “Monitoring, Evaluation, Learning and Impact Assessment” (MELIA).

FTA is developing its MELIA unit as a core research and support unit with team members from each of the participating centers. In the new phase, FTA has a great opportunity to make significant theoretical, methodological and empirical contributions to help improve research quality and research effectiveness within FTA, the CG, and beyond.

The MELIA work will be organized in four main clusters of activities:

Foresight/ex ante impact assessment

This is a new area for FTA, in which we will develop tools and approaches for assessing strategic opportunities and estimating potential impact to help inform priority setting and planning. In 2017, MELIA will develop our approach, in collaboration with other CGIAR Research Programs such as Policies, Institutions and Markets and Roots, Tubers and Bananas and in conjunction with research priority setting work in the FTA Management Support Unit.

Ex post impact assessment

MELIA will use and adapt experimental and quasi-experimental methods to assess the impact especially of technical or policy interventions. In 2017, for example, FTA will undertake an ex post impact assessment of the Agroforestry for Food Security Program in Malawi with support from the CGIAR Standing Panel on Impact Assessment.

Read also: Caught in a good loop – how to assess the usefulness of research

Outcome evaluation

MELIA will continue to develop, test and refine theory-based research evaluation methodology to assess research contributions in complex systems and to build a series of outcome evaluation case studies and a longer-term comparative analysis of cases. In 2017, at least four evaluations will be completed and four new studies will be launched. For example, an outcome evaluation of a gender-specific project focusing on women’s participation in forest management in Uganda will help improve the sensitivity of MELIA tools and strengthen FTA’s theory of gender transformative change in FTA landscapes. MELIA will also explore opportunities for collaboration with CGIAR Research Program on Climate Change, Agriculture and Food Security(CCAFS) and other partners.

Indicators, monitoring & reporting

This work will support the ongoing development of indicators, indicator frameworks and data collection to support the five Flagships in their reporting of outputs, outcomes and indicators for monitoring and reporting. Overall, this work will foster an impact culture within FTA, help ensure that FTA’s work remains relevant and useful in rapidly changing and complex circumstances, learn lessons from the rich FTA experience and guide ongoing research, engagement and capacity development to maximize effectiveness. In 2017, MELIA will support a FTA-wide work to revise the set of FTA indicators and milestones.

Key outcomes of MELIA’s work in 2017 will be

  • improved capacity and improved design, monitoring, evaluation and learning (DMEL);
  • strengthened impact culture within FTA;
  • stronger empirical evidence of contributions of FTA research; and
  • better, more transparent estimates of potential impact leading to well-informed strategies and improved donor confidence.

Latest post


Back to top

Sign up to our monthly newsletter

Connect with us