David Roodman from the Global Development Centre said: "Probably you agree that actions meant to help poor people should be guided by the best science about what works... And probably you’d concede that part of what makes science science is replicability... In this way, replicability is at the heart of the grand project to give everyone a shot at a decent life." ?
By replicating studies, we aim to assess how credible results are, or whether they can be achieved in different places or among different beneficiaries. Attempts to implement, and evaluate, the same programmes in different contexts (so-called 'external replication') are already at the heart of development practice. However, attempts to reproduce the same results, using the same or similar data ('internal replication'), to ensure credibility, is a relatively new but growing area of development research.
The 3ie-LIDC Replication Symposium featured three replication studies covering a variety of topics – the impact of colonialism, TV and modernisation, and microfinance.
Maren Duvendack from the Overseas Development Institute (ODI) talked about replication, reproduction and the credibility of micro-econometric studies of the impact of microfinance and informal sector borrowing in Bangladesh. She argued that the iconic microfinance impact evaluation of Pitt and Khandker (1998 - PnK) compares borrowing from microfinance institutions (MFIs) with not borrowing at all, rather than with borrowing from other sources. However, poor people often borrow from multiple sources, they continue to borrow from the formal and informal sectors, and often borrow from several MFIs. Duvendack and her team replicated the PnK study comparing the supposed advantages of borrowing from MFIs with borrowing from alternative sources, and not borrowing at all. Their findings suggest no greater impact of microfinance compared to other sources of finance. This means than 15 years of support for MFIs based the PnK study has been mistaken.??
Download the presentation
Kunal Sen, University of Manchester, presented his study of the impacts of British Colonialism in India. Banerjee and Iyer (BI) (American Economic Review, 2005) found that districts which the British assigned to landlord revenue systems systematically underperformed districts with non-landlord based revenue systems, especially in agricultural investment and productivity and mainly after the onset of the Green Revolution in the mid 1960s. On this basis BI claimed that there were long-lasting effects of the Land Revenue system on a variety of development outcomes after independence. Sen and his team corrected a miscoding of the land revenue system in Central Provinces, which BI characterise as landlord based, whereas reliable historical evidence suggests that this region should have been attributed to a mixed landlord/non-landlord based revenue system. Using a more appropriate classification constructed from documented archival research, the researchers found no evidence that agricultural performance of Indian districts in the post-independence period was adversely affected by the colonial landlord land revenue system. Their results demonstrate that the key BI argument that the more ‘oppressive’ landlord based colonial systems mattered for post-independent agricultural development in India rests on fragile historical and statistical foundations.??
Richard Palmer-Jones, University of East Anglia, delivered a critique of a paper about female empowerment and demographic transition in South Asia by Robert Jensen and Emily Oster (2009). Jensen and Oster (JO) identified causal positive impacts of cable TV on reducing women’s tolerance of spousal violence, son preference and fertility, and increasing female autonomy and educational enrolment in rural India. The replication study by Palmer-Jones corrected a programming error and adjusted index construction, so that JO’s main coefficients on tolerance of spousal beatings, female autonomy and enrolment in education shrink in size and are significantly weakened. Triangulation against alternative data sets casts doubt on some of the values reported. Motivated by theories derived from the literature on media and development supported by relevant descriptive statistics, the researchers introduced simple re-specifications of JO’s main model and uncovered heterogeneous effects along age and social identity dimensions, and suggestions of social externalities. The major finding is that cable TV introduction does not affect women without education, and son preference appears to be unrelated to cable TV. The team propose alternative plausible theories of change that weaken JO’s causal claims, but emphasise that the remaining associations between the introduction of cable TV and pro-social behaviour should not be considered robustly supported by either source of data used in the original analysis, due to the absence of covariates which could well confound the results.
Download the presentation
The Replication Studies Symposium is part of the 3ie-LIDC monthly Seminar Series ‘What works in international development’. The seminar series will be back in September after the summer break.
More about the 3ie replication programme