36 Gordon Square
London WC1H 0PD

T +44 (0) 20 7958 8251

3ie-LIDC Seminar: 'Attribution of cause and effect in small n impact evaluations'

Thursday, November 17, 2011 - 17:15 to 18:45

The drive to demonstrate results is leading to an increased focus on impact evaluation by researchers and donors. Most studies involve ‘large n’ experimental and quasi-experimental impact evaluations (where n is the sample size).

The need for ‘small n’ approaches arises when data is only available for one or a few units of assignment with the result that tests of statistical differences in outcomes between treatment and comparison groups are not possible. This might be the case when evaluating a case of capacity building in a single organisation, an intervention has significant heterogeneity with the result that sub-groups will be too small for statistical analysis or when an evaluation budget does not allow for a sufficiently large sample.

The latest seminar in the 3ie-LIDC series ‘What works in international development?’ draws on work by Howard White and Daniel Phillips with the intention to bridge this gap by exploring how small n impact evaluations can address the attribution of cause and effect.

For 3ie, impact evaluations are about tackling attribution; they should answer the question ‘To what extent has an intervention altered the state of the world?’ and cover outputs, both intended and unintended outcomes, and consider all Project-Affected Persons. Small n evaluations set out to demonstrate ‘beyond reasonable doubt’ that the link between an intervention and observed changes goes much further than ‘simple association’.  While large n evaluations establish causation through statistical means, small n evaluations build a case by ‘opening the black box’ lying between cause and effect and assembling an in-depth account of exactly how outcomes came about.

Research for the seminar examined various methodologies suitable for small n analysis, assessing how they address attribution and dividing them into two categories: (1) ‘explanatory’ approaches such as Realist Evaluation and Contribution Analysis, which emphasise the need to draw on the implicit theory lying behind an intervention and map out steps by which an evaluator can assess whether the theory of change occurred as expected or whether observed outcomes were a result, in part or whole, of other external factors. (2) ‘participatory’ approaches which place stakeholder participation at the heart of data collection and analysis in order to help establish the changes that have occurred since a programme’s inception and the various factors that have combined to bring these changes about.

The research found there are a number of key methodological steps that can be drawn out from the methodologies examined, including the need to clearly set out the attribution question(s) to be answered, to set out an intervention’s theory of change, to identify other potential causal mechanisms and to critically appraise the evidence in order to document each link in the actual causal chain. However, the question of what constitutes valid evidence for each link in the causal chain remains; just as large n studies face threats to internal validity from sampling errors and selection bias, so bias may also arise in small n studies if there is a systematic tendency to over or under-estimate the strength of the causal relationship, caused by bias in collection, interpretation and analysis of qualitative data.  

The conclusions reached build upon much that is already well known in the evaluation field, but does demonstrate that small n analysis can address the crucial question of attribution by following a number of evaluation steps: define the intervention to be evaluated, identify the evaluation questions being asked and set out a theory of change along with alternative potential causal explanations, identify the mix of methods to answer evaluation questions and create a clear data collection and analysis plan designed to tackle potential biases.

The next seminar in the series will take place on 7 December 2011 and will look at the impact of micro enterprise.

Download the PowerPoint presentation
Listen to the vodcast (audio with slides)
Read more about the 3ie-LIDC seminar series