Landscape review of global impact evaluation research

I was recently re-reading Cameron, Mishra & Brown’s 2015 landscape review of impact evaluation research, and realized that I hadn’t blogged about it previously. Obviously a few years out of date now, but it’s got some useful figures that I wanted to share. They reviewed 2200 studies from 1981 – 2012, finding that impact evaluations (both working papers and publications) grew rapidly after 2008.

Which countries are the subjects of impact evaluations? The Global South is clearly overrepresented here.

Where are these studies being published? The health sector accounts for as many impact evaluations as all other venues combined.

Which topics are most widely studied? Again, health remains the clear frontrunner here. I’m actually surprised that agriculture has so few evaluations, as I’ve always had the impression that it’s quite widely studied by social scientists.

Finally, where are the authors of these studies based? Unsurprisingly, most of them are from the Global North. This advantage actually appears to be increasing over time as well, which is worrying.

Snapshots of IPA’s RECOVR research hub

Screen Shot 2020-05-08 at 14.32.37

My partner Tom has done some interesting analysis of the projects listed on IPA’s RECOVR coronavirus research hub as of late April.  His previous research found that most social science research registries focused on coronavirus were focused on high income countries, but fortunately RECOVR is doing better on this front.

Some key points:

There are [56] projects all over the world, with 25 covering Africa, and 18 in Asia. They cover a huge variety of topics — health is naturally in the lead, but there are many studies of Social Protection, Peace and Recovery, and Financial Inclusion. It looks like there’s room for some more studies on education and agriculture…

One other welcome note: almost two thirds of the studies listed have at least one person from (one of) the countries being studied, on the team of named principal investigators.

Also, it’s not just RCTs!

1_ovEOg86d8k4nVoGrYARcSA

I should add here (with an inevitable bit of bias) that Tom’s a talented researcher and program manager, with a focus on behavioral science.  He’s currently open for consulting or full-time work based in London.

Why is it hard to scale up successful pilot projects?

The World Bank’s Let’s Talk Development blog had a good post on this issue last month.  Some examples of the challenge:

  • A first example comes from efforts to reform education in Kenyan primary schools by changing the way teachers are hired.ref2 New teachers offered a fixed-term contract by an international NGO significantly raised test scores. However, when the Kenyan government offered identical contracts, this produced zero impact.
  • A second example is seen in efforts to help low-income agricultural workers in rural Bangladesh to be less vulnerable to seasonal food insecurity during the agricultural lean season. Researchers tested the impact of offering small grants and interest-free loans to enable these workers to temporarily migrate to cities in search of work.ref3  This was incredibly successful in increasing internal migration flows, delivering higher income and consumption to program participants. The evidence was so striking that Givewell, a charity evaluator, recommended this program as one of the best values for money approaches of helping the poor, and an NGO scaled up the program dramatically to reach over 150,000 households in 2017. However, evaluations of this scaled-up program found it was unsuccessful in increasing migration rates, and the program was shut down.

What might be going on here?

  • Small-scale pilots may concentrate efforts on those who benefit most It is natural for those planning a pilot to target the most promising households or locations first, where they think the program will have most effect. Any program expansion is then likely to involve relaxing eligibility criteria to include those who may not have such large benefits from the program. This is seen clearly in a study of electricity conservation programs carried out in the United States.ref4 The power company began this program by targeting locations it thought would benefit most from the program, and subsequent expansion to new areas had much smaller effects. This was also one of the reasons behind less impact in the Bangladesh migration example, since scaling up the program meant relaxing eligibility requirements and offering loans to many households for whom lack of credit may not have been the binding constraint.
  • Implementation and political economy issues can arise as programs grow.  Small-scale pilots are often closely monitored and implemented with a degree of oversight that becomes challenging to carry out at large scales. This is particularly the case once the government becomes involved in implementation, where a combination of low state capacity, poor bureaucratic management, and capture by vested interests can seriously degrade the quality of the program offered. This was a key feature for the lack of success of government implementation of the Kenyan school reform, as resistance from teacher unions helped prevent the new contracts from working as intended.
  • General equilibrium effects can further reduce some impacts. Pilot programs to train a small number of workers may be successful in helping them find new jobs, but when large numbers are trained, they may end up competing with one another for a limited set of jobs and the impact on employment could be much lower.

 

Interesting academic articles for February 2020

Here’s what I’m looking forward to reading this month!

Kobina Aidoo and Ryan Briggs.  2019.  “Underpowered: Rolling blackouts in Africa disproportionately hurt the poor.”  African Studies Review.

Electricity demand exceeds supply in many parts of Africa, and this often results in rolling blackouts. This article argues that blackouts tend to concentrate on poorer places within countries, due to both economic and political factors. This argument is tested with an analysis of electricity availability across thirty-two neighborhoods in Accra and survey data from thirty-six African countries. Across these analyses, poorer people with a grid connection experience lower electricity supply than richer people. This article concludes by discussing implications for research on electricity availability, policymakers working on energy, and the distributive politics literature.

J. Andrew Harris, Catherine Kamindo, and Peter Van der Windt.  2020.  “Electoral Administration in Fledgling Democracies: Experimental Evidence from Kenya.”  Journal of Politics.

We examine the effects of national voter registration policies on voting patterns with a large-scale experimental study. Together with Kenya’s electoral commission, we designed an experiment in which 1,674 communities were randomized to a status quo or treatment group, receiving civic education on voter registration, SMS reminders about registration opportunities, and/or local registration visits by election commission staff. We find little evidence that civic education improves registration. Local registration visits improve voter registration, a relationship that increases in poorer communities. Moreover, local registration increased electoral competition and vote preference diversity in down-ballot contests in the 2017 Kenyan elections. Our results suggest that status quo voter registration policies constrain political participation and competition, and that inexpensive policy changes may attenuate the effects of such constraints.

Jeremie Gross, Catherine Guirkinger, and Jean-Philippe Platteau.  2020.  “Buy As You Need: Nutrition and Food Storage Imperfections.” Journal of Development Economics.

In this paper, we investigate whether and how a more steady supply of foodgrain in local markets impacts the nutritional status (measured with body-mass-indexes) of both children and adults, in a context characterized by large seasonal fluctuations in the price and availability of foodgrain. Taking advantage of the random scaling-up of a program of Food Security Granaries in Burkina Faso, we reach three conclusions. First, especially in remote areas where local markets are thin, the program considerably dampens nutritional stress. The effect is strongest among children, and young children in particular, for whom deficient nutrition has devastating long-term consequences. Second we argue that it is a change in the timing of food purchases, translated into a change in the timing of consumption, that drives the nutritional improvement. A simple two-period model shows that, once we account for various forms of storage costs, an increase in nutrition does not necessarily require larger quantity of food purchases or even consumption. Our last and unexpected conclusion is that the losses associated with foodgrain storage do not stem from physical losses in household granaries but rather from inefficient seasonal bodymass fluctuations. One plausible mechanism behind this particular storage imperfection rests on the households’ urge to consume readily available foodgrain.

Moses Khisa.  2020.  “Politicisation and Professionalisation: The Progress and Perils of Civil-Military Transformation in Museveni’s Uganda.”  Civil Wars.  

Problems of civil-military relations have been at the centre of recurring political crises in contemporary Africa. Routine military intrusion in politics characterised the first four decades of independent Africa. Citizens suffered at the hands of the armed forces, infamous for widespread human rights violations. One key response to this dual civil-military problem was to pursue a strategy of politicising the armed forces in order to make them a) subordinate to civilian authority and b) organically close to the public and protective than predatory. This also entailed the militarisation of politics ostensibly to bring the political class into closer conversation and collaboration with the military. To what extent did this strategy contribute to transforming civil-military relations? Taking the Ugandan case, this article argues that transformation was attained in making the military more respectful of citizens’ rights while simultaneously creating a fusion with the ruling class thereby subverting the very goal of professionalism.

Josephine Ahikire and Amon A. Mwiine.  2020.  “Gender equitable change and the place of informal networks in Uganda’s legislative policy reforms.”  Effective States in International Development working paper #134.

Uganda has had an uneven history and experience around gender equity policy reforms, particularly, from the late 1980s and early 1990s to-date. These range from the countrywide constitutional review processes of the early 1990s, legislative activism and reforms around domestic relations, land/property rights, and women’s access to public position, to mention but a few. While some of these gender reforms (commonly promoted through women’s collective mobilisation) were successful, other legislative initiatives faced intense resistance. This paper compares three policy cases – the 1997 Universal Primary Education policy, the 1998 legislative reform around spousal co-ownership of land and the 2010 Domestic Violence Act. Drawing on feminist institutionalism, the paper explores how gender norms operate within institutions (both formal and informal) and how institutional processes construct, reproduce or challenge gender power dynamics in policy reforms. The paper examines the place of informal networks and raises critical questions regarding ways in which women emerge as critical actors in securing and consolidating gender change, the strategies they draw upon to negotiate resistance, and whether the nature of policy reform influences the kind of resistance and (in effect) counterstrategies used to negotiate resistance to gender change. We also assess the implications these legislative processes have for activism around gender equity reforms. Findings indicate creative ways through which women draw on informal networks and networking practices to influence gender equitable change, often revealing the micro, subtly gendered dynamics that animate success or failure of a particular policy reform. We argue that the nature of policy reform, e.g. gender status policies or doctrinal policies, determines the nature and process of policy adoption.

Eric Mvukiyehe and Peter van der Windt.  2020.  “Assessing the Longer Term Impact of Community-Driven Development Programs: Evidence from a Field Experiment in the Democratic Republic of Congo.”  World Bank Policy Research working paper #9140.

Community-driven development programs are a popular model for service delivery and socioeconomic development, especially in countries reeling from civil strife. Despite their popularity, the evidence on their impact is mixed at best. Most studies thus far are based on data collected during, or shortly after, program implementation. Community-driven development’s theory of change, however, allows for a longer time frame for program exposure to produce impact. This study examines the longer term impact of a randomized community-driven development program implemented in 1,250 villages in Eastern Democratic Republic of Congo between 2007 and 2012. The study team returned to these villages in 2015, eight years after the onset of the program. The study finds evidence of the physical endurance of infrastructure built by the program. However, it finds no evidence that the program had an impact on other dimensions of service provision, health, education, economic welfare, women’s empowerment, governance, and social cohesion. These findings suggest that, although community-driven development programs may effectively deliver public infrastructure, longer term impacts on economic development and social transformation appear to be limited.

Cyril Brandt and Tom De Herdt.  2020.  “Reshaping the Reach of the State: The Politics of a Teacher Payment Reform in the DR Congo.”  Journal of Modern African Studies.

We analyse the politics of the reform of teacher payment modalities in the Democratic Republic of Congo (DRC) in light of the uneven territorial reach of the DRC state. The reform focused on extending this reach by paying all teachers via a bank account, replacing longstanding shared governance arrangements between state and faith-based organizations with a public-private partnership. By using qualitative and quantitative data, we map the political practices accompanying the implementation of the reform. While the reform itself was officially deemed a success, its intended effects were almost completely offset in rural areas. Moreover, governance of teacher payments was not rationalized but instead became even more complex and spatially differentiated. In sum, the reform has rendered governance processes more opaque and it deepened the existing unevenness in the geography of statehood.

Michel Thill and Abel Cimanuka.  2019.  “Governing local security in the eastern Congo: decentralization, police reform and interventions in the chieftaincy of Buhavu.”  Rift Valley Institute.  

In the Democratic Republic of the Congo (DRC or Congo), security governance is competitive, fragmented and marked by violence. Multiple actors—state and non-state—vie for influence and many areas of the country lack effective structures to ensure that their residents live in safety and security. In this context, the threat and use of violence has become central to the state’s efforts to maintain social control and public order. This tendency has come to shape the troubled relationship between Congolese citizens and the army and police, reflected in numerous fraught day-to-day interactions. Two ongoing processes— administrative decentralization and police reform—have been designed to turn a page on past practices, bring government and security closer to the population and, consequently, improve this relationship. While they have had some successes, they also risk the re-creation of existing governance dynamics within newly empowered local administrative and security-related entities.

Mary Hallward-Driemeier, Anna Kochanova, and Bob Rijkers.  2020.  “Does Democratization Promote Competition? Indonesian Manufacturing Pre and Post Suharto.”  World Bank Policy Research Group working paper #9112.  

Does democratization promote economic competition? This paper documents that the disruption of political connections associated with Suharto’s fall had a modest pro-competitive effect on Indonesian manufacturing industries in which his family had extensive business interests. Firms with connections to Suharto lost substantial market share following his resignation. Industries in which Suharto family firms had larger market share during his tenure exhibited weak improvements in broader measures of competition in the post-Suharto era relative to industries in which Suharto firms had not been important players.

Anne Buffardi, Samuel Sharp, Sierd Hadley and Rachel A. Archer.  2020.  “Measuring evidence-informed decision-making processes in low- and middle-income countries.”  Overseas Development Institute.

The evidence base on the practice of evidence-informed decision-making (EIDM) in international development is limited. Previous work has identified multiple roles that evidence could play; principles and desirable decision-making practices; and individual, interpersonal, organisational and contextual factors thought to influence the interpretation of evidence and decisions. Despite a proliferation of frameworks and guidance, there is a relative dearth of research on the extent to which and how they are applied in practice, at what cost and with what effects. EIDM faces measurement challenges, including investigation into largely undocumented and sometimes unobservable processes, multi-finality and equifinality (multiple pathways to multiple outcomes), often along extended time horizons, in addition to difficulties establishing counterfactuals. In the health sector, current indicators tend to cluster around two ends of a long change pathway: tracking upstream activities and immediate outputs, and downstream changes in health coverage and outcomes. Building on existing systems, future efforts could be directed at the ‘missing middle’ in measurement, filling notable gaps in defining what constitute quality EIDM processes, minimising biases in measuring these processes and investigating how evidence-informed recommendations make their way through the policy process.

Matteo Alpino and Eivind Moe Hammersmark.  2020.  “The Role of Historical Christian Missions in the Location of World Bank Aid in Africa.”  World Bank Policy Research Group working paper #9146.

This article documents a positive and sizable correlation between the location of historical Christian missions and the allocation of present-day World Bank aid at the grid-cell level in Africa. The correlation is robust to an extensive set of geographical and historical control variables that predict settlement of missions. The study finds no correlation with aid effectiveness, as measured by project ratings and survey-based development indicators. Mission areas display a different political aid cycle than other areas, whereby new projects are less likely to arrive in years with new presidents. Hence, political connections between mission areas and central governments could be one likely explanation for the correlation between missions and aid.

Abhishek Nagaraj and Scott Stern.  2020.  “The Economics of Maps.”  Journal of Economic Perspectives.  

For centuries, maps have codified the extent of human geographic knowledge and shaped discovery and economic decision-making. Economists across many fields, including urban economics, public finance, political economy, and economic geography, have long employed maps, yet have largely abstracted away from exploring the economic determinants and consequences of maps as a subject of independent study. In this essay, we first review and unify recent literature in a variety of different fields that highlights the economic and social consequences of maps, along with an overview of the modern geospatial industry. We then outline our economic framework in which a given map is the result of economic choices around map data and designs, resulting in variations in private and social returns to mapmaking. We highlight five important economic and institutional factors shaping mapmakers’ data and design choices. Our essay ends by proposing that economists pay more attention to the endogeneity of mapmaking and the resulting consequences for economic and social welfare.

Interesting academic articles for January 2020

Here’s what I’m looking forward to reading this month!

Shelby Grossman.  2019.  “The Politics of Order in Informal Markets: Evidence from Lagos.”  World Politics.

Property rights are important for economic exchange, but in many parts of the world, they are not publicly guaranteed. Private market associations can fill this gap by providing an institutional structure to enforce agreements, but with this power comes the ability to extort from group members. Under what circumstances do private associations provide a stable environment for economic activity? The author uses survey data collected from 1,179 randomly sampled traders across 199 markets in Lagos, Nigeria, and finds that markets maintain institutions to support trade not in the absence of government, but rather in response to active government interference. The author argues that associations develop protrade policies when threatened by politicians they perceive to be predatory and when the organizations can respond with threats of their own. The latter is easier when traders are not competing with one another. To maintain this balance of power, an association will not extort; it needs trader support to maintain the credibility of its threats to mobilize against predatory politicians.

Sabrina Karim.  2020.  “Relational State Building in Areas of Limited Statehood: Experimental Evidence on the Attitudes of the Police.”  American Political Science Review.  

Under what conditions does state expansion into limited statehood areas improve perceptions of state authority? Although previous work emphasizes identity or institutional sources of state legitimacy, I argue that relationships between state agents and citizens drive positive attitude formation, because these relationships provide information and facilitate social bonds. Moreover, when state agents and citizens share demographic characteristics, perceptional effects may improve. Finally, citizens finding procedural interactions between state agents and citizens unfair may adopt negative views about the state. I test these three propositions by randomizing household visits by male or female police officers in rural Liberia. These visits facilitated relationship building, leading to improved perceptions of police; shared demographic characteristics between police and citizens did not strengthen this effect. Perceptions of unfairness in the randomization led to negative opinions about police. The results imply that relationship building between state agents and citizens is an important part of state building.

Sarah Brierley.  2019.  “Unprincipled Principals: Co-opted Bureaucrats and Corruption in Ghana.”  American Journal of Political Science.

In theory, granting politicians tools to oversee bureaucrats can reduce administrative malfeasance. In contrast, I argue that the political control of bureaucrats can increase corruption when politicians need money to fund election campaigns and face limited institutional constraints. In such contexts, politicians can leverage their discretionary powers to incentivize bureaucrats to extract rents from the state on politicians’ behalf. Using data from an original survey of bureaucrats (N = 864) across 80 randomly sampled local governments in Ghana, I show that bureaucrats are more likely to facilitate politicians’ corrupt behavior when politicians are perceived to be empowered with higher levels of discretionary control. Using qualitative data and a list experiment to demonstrate the mechanism, I show that politicians enact corruption by threatening to transfer noncompliant officers. My findings provide new evidence on the sources of public administrative deficiencies in developing countries and qualify the presumption that greater political oversight improves governance.

Raúl Sánchez de la Sierra.  2019.  “On the Origins of the State: Stationary Bandits and Taxation in Eastern Congo.”  Journal of Political Economy.

A positive demand shock for coltan, a mineral whose bulky output cannot be concealed, leads armed actors to create illicit customs and provide protection at coltan mines, where they settle as “stationary bandits.” A similar shock for gold, easy to conceal, leads to stationary bandits in the villages where income from gold is spent, where they introduce illicit mining visas, taxes, and administrations. Having a stationary bandit from a militia or the Congolese army increases welfare. These findings suggest that armed actors may create “essential functions of a state” to better expropriate, which, depending on their goals, can increase welfare.

Pedro Carneiro, Lucy Kraftman, Giacomo Mason, Lucie Moore, Imran Rasul, and Molly Scott.  2019.  The Impacts of a Multifaceted Pre-natal Intervention on Human Capital Accumulation in Early Life.”  Working paper.

We present results from a large-scale and long-term randomized control trial to evaluate an intervention targeting early life nutrition and well-being for households residing in extreme poverty in Northern Nigeria. The multifaceted intervention provides: (i) information to mothers and fathers on practices related to pregnancy and infant feeding; (ii) high-valued unconditional cash transfers to mothers, each month from pregnancy until the child turns two. We document two- and four-year impacts among 3600 pregnant women and their children. The intervention leads to large and sustained improvements in anthropometric and health outcomes for children, including an 8% reduction in stunting by endline. These impacts are partly driven by information-related channels (such as improved knowledge, practices and health behaviors of mothers towards new borns). However, the value and certain flow of cash transfers is also key: these induce labor supply responses among women, and allow them to undertake investments in livestock. These are both a source of protein rich diets for children, and generate higher earnings streams for households long after the cash transfers expire. The results show the sustainability and cost-effectiveness of scalable multifaceted pre-natal interventions in even the most challenging and food insecure economic environments.

Kanika Jha Kingra, Francis Rathinam, Tony Tyrrell, and Marie Gaarder.  2019. Social protection: a synthesis of evidence and lessons from 3ie evidence-supported impact evaluations.”  3ie working paper #34.

The paper synthesises evidence from evaluation of transfer programmes in Ecuador, Malawi, Sierra Leone, Tanzania, Uganda and Zimbabwe and from public works programmes in Ethiopia and India.  [Key findings include the following.] Cash versus in-kind or food transfers and conditional versus unconditional transfers are issues of extensive debate amongst implementers of social protection programmes. Transfers can positively affect non-beneficiaries and the wider economy. Information on cost-benefit remains a gnawing gap. Analysis of gendered outcomes remains limited.

Who counts as a household head?

If you’ve done much survey research, you’re probably familiar with controversies over how to map out households, and in particular whether to assume that a married man is by default the household head.  The Center for Global Development and Data2X recently shared the discussion from an interesting event on this topic.  As the authors Mayra Buvinic and Dominique van de Walle note,

Traditionally, the uses of household headship have been both practical and conceptual. First, the delineation of a head is universally used in household surveys as a practical organizing principle to map out the household roster and relationships between household members. Second, comparing households according to the sex of the designated head has been used as a way to assess gender inequalities. Indeed, female headship has been interpreted as a proxy for women’s poverty.

They lay out three reasons why assigning headship according to gender can be a bad idea.

First, headship as a concept is value-laden and reinforces patriarchal gender stereotypes that are important to resist. Gender-biased concepts and measures can perpetuate stereotypical notions that only men should be heads of household. Second, assigning a head also depends on subjective assessments by household members. Finally, headship may also perpetuate gender bias if interviewers are themselves predisposed towards attributing headship to adult males. The conclusion to this line of reasoning is that getting rid of these data and organizing the household around a “primary household respondent” would solve the problem of using a construct that is not reducible.

However, they don’t feel that the concept should be dropped immediately.  One salient point is that many cultures still do in practice designate one adult to make the major household decisions, and ignoring this designation would throw away useful data.  In addition,

We would argue that analyzing households by their head’s gender can be a reliable source of information for:

  1.  Monitoring changes in society and family dynamics. The growing number of women heading households in prime adult age groups can signal social change towards gender equality and away from patriarchal family structures.
  2.  Female headship, when it captures marital dissolution and widowhood (Africa) or unpartnered motherhood (Latin America), often signals vulnerability and disadvantage for women and children.
  3.  The growing numbers of female heads resulting from wars and violent conflict signal both vulnerabilities and economic and political opportunities for women.