printUse ctrl + p to print the page

How do we know if a program made a difference? A guide to statistical methods for program impact evaluation

Authors: Peter M. Lance,  David K. Guilkey, Aiko Hattori & Gustavo Angeles (for MEASURE Evaluation)
Publication date: 2014

There is a long-running, vigorous and evolving methodological debate about the appropriate or optimal way to evaluate the impact of a program. This manual strives not to convince readers of the merits of one particular alternative or another, but instead simply to present the various options in as approachable a fashion as possible and then let them decide for themselves where they stand. In other words, it strives to be impartial. However, that is probably an impossible standard. We cannot help but have our own views, which probably subtly intrude on the manual. To those who disagree with us we ask for their pardon but remind them that this shortcoming was probably inevitable.

Writing this manual has required review of a large number of contributions to the literature on impact evaluation methods, as well as many actual impact evaluations. Many of these are cited in this manual, which has now been through hundreds of revisions. In the course of those revisions, with the attendant re-arrangement, deletion and insertion, it is possible that some citations were accidentally dropped and others have become un-moored from the train of thought for which they were originally introduced. We o er unquali ed apology for all such instances.

That said, updated versions of this manual will be released from time to time. We invite those with comments or criticism of the substance of the manual to submit their thoughts so that they can inform future versions. Errors and omissions in citations that are communicated to us will be corrected immediately.

------------------------------------------------------------------------------------------------------------------------------------

This manual provides an overview of core statistical and econometric methods for program impact evaluation (and, more generally, causal modelling). More detailed and advanced than typical brief reviews of the subject, it also strives to be more approachable to a wider range of readers than the advanced theoretical literature on program impact evaluation estimators. It thus forms a bridge between more basic treatments of the essentials of impact evaluation methods and the more advanced discussions. It seeks to discuss impact evaluation estimators in a thorough manner that does justice to their complexity, but in a fashion that is approachable.

The manual is targeted to: public health professionals at programs, government agencies, and NGOs who are the consumers of the information generated by program impact evaluations; professionals serving the aforementioned role in any area of programming that influences human welfare; graduate students in public health, public policy and the social sciences; technical staff at evaluation projects; journalists looking for a more nuanced understanding of the steady stream of impact (and, more broadly, causal) studies on which they are asked to report; analysts at health analytics organizations; and so on.

Download STATA do files for the programs behind the numerical examples in the manual. The first number of each do file indicates its associated chapter, and the second number indicates order within the chapter.

The current version of this manual is a revision posted on September 22, 2014. This manual was heavily edited before initial release, but inevitably for such a detailed, equation-intense piece, some typographical errors have been found since that time. We expect that some errors remain, and therefore anticipate periodic future revisions. We apologize for any inconvenience this causes.