The World Bank’s Let’s Talk Development blog had a good post on this issue last month. Some examples of the challenge:
- A first example comes from efforts to reform education in Kenyan primary schools by changing the way teachers are hired.ref2 New teachers offered a fixed-term contract by an international NGO significantly raised test scores. However, when the Kenyan government offered identical contracts, this produced zero impact.
- A second example is seen in efforts to help low-income agricultural workers in rural Bangladesh to be less vulnerable to seasonal food insecurity during the agricultural lean season. Researchers tested the impact of offering small grants and interest-free loans to enable these workers to temporarily migrate to cities in search of work.ref3 This was incredibly successful in increasing internal migration flows, delivering higher income and consumption to program participants. The evidence was so striking that Givewell, a charity evaluator, recommended this program as one of the best values for money approaches of helping the poor, and an NGO scaled up the program dramatically to reach over 150,000 households in 2017. However, evaluations of this scaled-up program found it was unsuccessful in increasing migration rates, and the program was shut down.
What might be going on here?
- Small-scale pilots may concentrate efforts on those who benefit most It is natural for those planning a pilot to target the most promising households or locations first, where they think the program will have most effect. Any program expansion is then likely to involve relaxing eligibility criteria to include those who may not have such large benefits from the program. This is seen clearly in a study of electricity conservation programs carried out in the United States.ref4 The power company began this program by targeting locations it thought would benefit most from the program, and subsequent expansion to new areas had much smaller effects. This was also one of the reasons behind less impact in the Bangladesh migration example, since scaling up the program meant relaxing eligibility requirements and offering loans to many households for whom lack of credit may not have been the binding constraint.
- Implementation and political economy issues can arise as programs grow. Small-scale pilots are often closely monitored and implemented with a degree of oversight that becomes challenging to carry out at large scales. This is particularly the case once the government becomes involved in implementation, where a combination of low state capacity, poor bureaucratic management, and capture by vested interests can seriously degrade the quality of the program offered. This was a key feature for the lack of success of government implementation of the Kenyan school reform, as resistance from teacher unions helped prevent the new contracts from working as intended.
- General equilibrium effects can further reduce some impacts. Pilot programs to train a small number of workers may be successful in helping them find new jobs, but when large numbers are trained, they may end up competing with one another for a limited set of jobs and the impact on employment could be much lower.