printUse ctrl + p to print the page

Impact evaluation in value chain partnerships: focusing on contribution analysis and mixing methods to reduce validity threats

Title: Impact evaluation in value chain partnerships: focusing on contribution analysis and mixing methods to reduce validity threats

Author(s): Giel Ton

Function of lead author: Senior researcher

Institution of lead author: Wageningen UR, LEI

Relevance of the paper for the conference theme ‘Partnering for Success: how M&E Can Strengthen Partnerships for Sustainable Development’.

The research shows the importance to theoretically disentangle the effects of partnership interventions and contextual processes, while emphasizing that these effects are a result of a configuration of factors in which the partnership (at most) is a contributory factor.

The research highlights the importance of Theory-based evaluation and a creative mix of methods to feed counterfactual thinking, a grounded reflection about the likely situation of partnership dynamics without the support intervention.

Abstract:

Partnerships between companies and non-governmental organizations that aim to incorporate smallholder farmers into value chains are increasingly being promoted as a way of pursuing development goals. Development support must be able to prove its effectiveness. Impact evaluation is a way to generate this information. The thesis is about the design of these  impact evaluations and how research methods can be combined to obtain credible evidence on effectiveness. It contrasts two approaches to impact evaluation design, ‘randomistas’ (Does it work?) and ‘realistas’ (For whom does it work, and under what conditions?), and distils seven principles for research design that create synergy between these two approaches. 

This presentation will illustrate these with experiences in impact evaluation design in several case studies. In two case studies in Uganda and Rwanda, we documented the sequences of events within such partnership interventions, their context, and the intermediate outcomes, identified as the new rules and practices, proto-institutions, that generate institutional change. Reflecting on the sequence of events within a partnership intervention, these case studies reveal how the interventions combined with other dynamics, such as changes in markets, government policy or business strategy, and became part of a wider configuration of factor that cause change.

A second example refers to research on a Bolivian grant fund that supports investments in processing by farmer groups. To assess the effectiveness we used Contribution Analysis, and used process tracing to define success or failure of the grant. The study combined two different logics of comparative analysis and exploration for causal relations, Qualitative Comparative Analysis and logistic regression, the study detected predictors of effectiveness in a moderate sample of 26 cases.

References

Ton, G. (2012). "The Mixing of Methods: a three-step process for improving rigour in impact evaluations." Evaluation 18(1): 5-25.

Ton, G., S. Vellema and L. Ge (2014). "The Triviality of Measuring Ultimate Outcomes: Acknowledging the Span of Direct Influence." IDS Bulletin 45(6): 37-48.

Ton, G. (2015). Measuring Tensions and Intentions: Mixing methods in the impact evaluation of development support to farmer organisations. PhD, Wageningen University.

Vellema, S; Ton, G; de Roo, N; van Wijk, J., 2013. Value chains, partnerships and development: Using case studies to refine programme theories. Evaluation 19(3) 304–320. Sage. http://edepot.wur.nl/265164  and  https://www.researchgate.net/publication/258136950