What counts as “policy relevant evaluation”?

Heather Lanthorn recently wrote a great post about defining “policy relevant evaluation” that really pushed me on my priors on this concept.  As she points out:

just because research is conducted on policy does not automatically make it ‘policy relevant’ — or, more specifically, decision-relevant. it is, indeed, ‘policy adjacent,’ by walking and working alongside a real, live policy to do empirical work and answer interesting questions about whether and why that policy brought about the intended results. but this does not necessarily make it relevant to policymakers and stakeholders trying to make prioritization, programmatic, or policy decisions. in fact, by this point, it may be politically and operationally hard to make major changes to the program or policy, regardless of the evaluation outcome.  …

jeff hammer has pointed out that even though researchers in some form of applied work on development are increasingly doing work on ‘real’ policies and programs, they are not necessarily in a better position to help high-level policymakers choose the best way forward. this needs to be taken seriously, though it is not surprising that a chief minister is asking over-arching allocative questions (invest in transport or infrastructure?) whereas researchers may work with lower-level bureaucrats and NGO managers or even street-level/front-line workers, who have more modest goals of improving workings and (cost-)effectiveness of an existing program or trying something new.

I think this is a great step towards an acknowledgement that different types of research will be useful to policymakers at different levels of government and with different policy goals.  Most of the RCTs I’ve seen operate within a fairly narrow set of parameters that correlate to the types of programming decisions made by senior managers at social welfare ministries, like health or education.  There’s a specific policy goal that someone wants to achieve (improving primary school children’s reading performance), a known segment of the population targeted by the policy (children ages 5 – 16 currently enrolled in school), and a strong sense of the limits of the type of solution that can be proposed, particularly financially (we can afford one hour of tutoring per day by a literate adult, but can’t build fully equipped libraries in every town).  Within these parameters, RCTs can be a great way to evaluate the effectiveness of different types of programs that might meet this policy goal.

That said, if you change any of the parameters, RCTs are often no longer efficient way to make programming decisions.  Outside of social welfare ministries, many important policy choices either can’t be randomized (providing military support to an ally, deciding whether to invest in nuclear power) or don’t need to be (it’s already quite well-documented that expansionary monetary policy leads to inflation).  As Heather noted, RCTs frequently can’t offer much guidance to policymakers making the inherently political choice between different policy goals.  And they often don’t generate new insights effectively when the underlying process that produces a social problem, and the particular segments of the population affected by this process, aren’t known.

This is especially visible in recent RCTs examining the effects of institution-building after civil war.  While people frequently speculate that the combination of poverty, inequality, and unemployed young people increases the risk of civil war, the majority of countries fitting this description don’t ever experience war.  And even among those which do, the question of why some people choose to rebel and what can be done to prevent these people or similar ones from fighting again in the future is basically unanswered.  Virtually every country caught up in civil war has a large population of poor, politically excluded young people, but only a tiny minority of those people will ever join a rebellion, making it very difficult to figure out how to target programs aimed at reducing the likelihood of future conflict.

The point here isn’t that RCTs are useless, but that “policy relevant research” might take very different forms depending on the type of question being answered and the underlying base of knowledge about the issue.

The women of the FARDC

The New York Times is running a remarkable photo essay on Sgt. Madot Dagbinza, who was one of only about 3000 women in the 150,000-strong FARDC before she died in an ambush in 2014.  Absolutely worth a read, along with Maria Eriksson Baaz & Maria Stern’s article on the experiences of female FARDC soldiers.


The NYT piece is also notable for moving past simplistic depictions of the FARDC as an incompetent force that does nothing but rape and steal.  There are still many units who abuse civilians, but it isn’t uniform – others are better trained and better disciplined.  Christoph Vogel consulted with the NYT journalists on this aspect of the story, and has written a very good summary of the FARDC as a “chameleon army.”  Maria Eriksson Baaz’s other work on this subject is also critical reading, such as this paper with Maria Stern on soldiers’ perceptions of violence, and this paper with Judith Verweijen on how the military sometimes solves disputes between civilians in places where civilian courts are inaccessible.

What’s the best way to pay people not to rebel?

Saumitra Jha recently gave a fascinating lecture at Berkeley’s comparative colloquium in which he discussed some of his current work on designing financial instruments that can promote political stability. He drew extensively on the case of Japan in the late 1800s, where the government granted bonds to ex-samurai who were opposed to its modernizing reforms, then encouraged the bondholders to invest in the national banking system.  This gave a potentially militant group a significant stake in political stability and financial modernization.  Jha’s description of this process is worth quoting at length:

The government created an innovative, ethnically-delimited asset – the bonds given to samurai — even while eliminating the privileges and obligations that had made this ethnic group distinct. It then took the ex-samurai, one [group] that was the most likely to engage in violence and enhance political risk, and gave them incentives to become local bankowners – a group with arguably the greatest incentives to avoid engaging in violent actions that would raise the political risk of their investments in local ventures (often rice and silk). By aligning the samurai’s interests against political risk, these financial innovations aligned their interests with not only the merchants who were their fellow merchant share-holders but society at large. Since all could benefit, and in fact the samurai had explicit stakes, as bankers, in the nation’s future, they also meant that the samurai could give up their arms and credibly share the gains that modernisation and reduced political risk provided.

This process also produced a truly phenomenal photo of samurai-turned-banker Eiichi Shibusawa, who’s known as the father of Japanese capitalism for his role in founding the Tokyo Stock Exchange and a number of other publicly held companies.

Eiichi_Shibusawa_transformationFrom Wikipedia

What I find fascinating here is the ways in which this process is both similar to and different from current debates about post-conflict power-sharing in Africa.  The idea behind consociationalism is that placing representatives of all contesting groups in power ought to give them a common interest in maintaining the stability of the state.  This appears to have worked out relatively well in Burundi for the last ten years.  Lemarchand is explicit about Bujumbura’s focus on maintaining interethnic stability even at the cost of good governance: “[the administration is] a top-heavy political machinery whose sole purpose is to provide as many jobs as are needed to meet the requirements of political stability.  The government is not meant to govern; its purpose is to offer an attractive alternative to rebellion” (2009, p. 149).  In a sense, then, this is simply a less efficient means of accomplishing what the samurai bonds did in Japan.

However, institutionalized power-sharing has often failed in Africa as well.  The prime case here is obviously the DRC, where the 2002 power-sharing accords got most – but not all – of the major rebel groups durably off the battlefield.  The Nkunda- and Ntaganda-centric set of groups which continually rebelled in the east were largely spurred on by Rwanda, but also presumably believed that they might get a better deal out of some future peace agreement.  Would a different benefit structure for ex-rebels – shares in banks as opposed to positions in the government – have led to a different outcome?  The Congolese central bank has been issuing bonds for several years now, and the banking sector is badly underdeveloped, so promoting investment (and of course concomitant regulatory mechanisms) there might indeed benefit everyone.  If readers have other examples of the strategic use of financial instruments to promote political stability, I’d love to hear about them.

Free resources for studying conflict & governance

The question of whether it’s ethical to charge US$40 for access to a single article in an academic journal is a heated one in Northern academia these days.  I don’t particularly think it is, and it seems that the academic community is slowly taking some steps towards making more journal articles freely available online, but in the meantime there’s a lot of interesting content locked behind paywalls.  In that spirit, here are some of the free resources I’ve found over the years related to the study of conflict and governance in low-income countries.

  • For two days only, beginning February 17, Africa Intelligence is offering free access to more than 100,000 articles on its site.  (Thanks to Ben Radley for the tip about this.)
  • Stability is the gold standard for open-access journals in this field.  All the content is free, peer-reviewed, and generally high quality.  Scott Ross has also pointed me to Cultural Anthropology, another great open-access source.
  • If you’re not already using Google Scholar for your research, you should be.  One particularly nice feature of this service is that it includes links to free versions of articles when they’re available.  Look at the link to the right of the title where it says [PDF].
  • In the US, many scholars upload either draft or finished versions of their papers to SSRN, where they’re often available to download for free.  Working papers on development economics can also be found at BREAD.
  • A lot of great content on conflict and governance is published by think tanks, which often post their discussion papers online for free.  Check out the list of think tanks in the sidebar of this blog to get started.
  • If you’re willing to pay for specialized content, but can’t afford much of it, you might start with the Annual Review of Political Science, which had a good recent edition on civil war.   The Annual Review journals cover every major academic discipline, and provide concise summaries of current research on some of the biggest questions in each field.  It’s a great way to start exploring a new field, and catch up on the latest questions and trends in research.
  • Finally, if you really need a specific article and can’t find a free version anywhere, it never hurts to email the authors directly to ask.  Alternatively, ask a friend at a university if they can download a copy for you.  If you live close to a public university, you may also be able to request temporary access to their library.  (Public institutions seem more likely to offer this since they presumably have some obligations to residents of the state more broadly, even if they’re not enrolled.  It’s been my experience that private universities often won’t let members of the public use their libraries.)

What other open-source resources do you use?

Land disputes and the limits of off-the-shelf data in the DRC

Recently I’ve been tossing around the idea of writing a paper on subnational variation in land disputes in the DRC (on the basis of a suggestion from the excellent Leo Arriola).  There’s obviously been a great deal written about how tensions over land in eastern DRC have contributed to the civil war there, but it’s surprisingly hard to find much recent work on land disputes or land tenure regimes in the rest of the country.  As Cathy Boone points out, disagreements over land are ubiquitous in rural Africa, but most of them stay at the level of the community and don’t scale up into organized rebellion against the government.  What might patterns of land disputes look like outside of eastern DRC?

As a first cut at this question, I searched the SCAD and ACLED datasets for any observations of protests, riots or battles that were explicitly connected to land.  (For instance, ACLED has an observation from 6 August 2011 which notes, “Lendu Communal Militia attacked the nearby village of Kpachu following a land dispute.”  This is the red dot in Orientale Province.)  This produced 41 unique observations from 1993 – 2013.  A good number of them are in the east, as one might have expected, but there’s also an interesting cluster around Mbuji-Mayi and a handful in Equateur Province as well.


I’d hoped to study the conflicts outside of the east in greater depth, but when I looked at the underlying data again, I had to pause.  It turns out that the data is really uneven in its geographic and temporal coverage.  SCAD covers the period from 1990 – 2013, sources its data from LexisNexis (which in turn sources largely from the AP and AFP), and finds 16 land disputes turned violent between 1993 and 2010.  ACLED covers the period from 1997 – 2013, relies on a wider range of data sources (including local newspapers and NGO reports), and finds 25 violent land disputes – but only between 2011 and 2013.  There’s no overlap in their observations, which makes me wonder if either of them individually, or both together, are really reliable sources for data on this particular question.

There might be two things at play here.  First, the two datasets might actually have observed similar events during the 1997 – 2013 period that they both cover, but included different levels of detail about the motivation for a particular dispute, meaning that one of them shows up in a search for “land” and the other doesn’t.  (This could be evaluated by identifying observations with approximately the same dates and locations and comparing the notes on their motivations, which I haven’t done yet.)  Second, the underlying data sources might matter a great deal, and the two datasets might not be observing the same things at all.  In SCAD’s case, I wonder why the large cluster of conflicts that ACLED finds around Mbuji-Mayi from 2011 – 2013 didn’t show up in any of their news reports; in ACLED’s, I wonder why they only began identifying any conflicts as land-related after 2010.  I don’t question the validity of the observations they did include, but the degree to which each dataset appears to be either missing data or coding it in conflicting ways seems to be significant.

I wanted to share this here not to snipe at SCAD and ACLED – which are providing a huge service in collecting all of this data and making it freely available to anyone who’d like to use it – but to emphasize the importance of ground-truthing your data, even when it’s from a well-known dataset.  As Jay Ufelder has written about at length, translating complex political events such as battles and protests into an easily comparable quantitative format is a huge challenge, and the process by which events are chosen and coded for inclusion isn’t always uniform.  As for me, I’m glad I got to explore this data, but out of concern for its representativeness I think I’m going back to the drawing board for the paper.

Post-conflict recovery isn’t uniform

The Overseas Development Institute  recently released a new report on post-conflict recovery in Liberia.  They suggest that the country is making slow but steady steps towards improved security.  But check out the spatial variation in people’s perceptions of safety (from this graphic):


Your daily reminder that countries are probably the wrong unit of analysis for thinking about the impact of civil war; where one lives within a country also matters a great deal.

Does precolonial political centralization matter in Africa?

7-KumasiKumasi in the late 19th century, from Encyclopaedia Britannica

For a long time, Northern scholars of Africa used to write about the continent as though the colonial period was the beginning of history.  Jean-François Bayart famously argued against this, but even after his book appeared well-known authors like Mahmood Mamdani and Crawford Young made the case that colonization changed everything in Africa.

More recently, however, Northern researchers have started to take precolonial politics seriously again.  I was thinking about this recently when Tanu Kumar sent me a link to this working paper by Mark Dincecco, James Fenske, and Massimiliano Gaetano Onorato.  They argue that precolonial warfare in Africa led to greater levels of political centralization, but is also associated with higher rates of civil war today.  Since civil war is generally bad for state capacity and development outcomes, this suggests that more centralized states in the precolonial era should be less developed today.

How does this argument hold up?  Jacob Hariri suggests that stronger precolonial states outside Europe tended to resist the spread of European institutions which could promote democracy and economic growth, leading to lower income levels and higher rates of autocracy today.  However, a number of other authors find that precolonial centralization in Africa is actually good for development.  Nicola Gennaioli & Ilia Rainer and Stelios Michalopoulous & Elias Papaioannou all find higher rates of local public goods provision in places that had strong precolonial states.  The mechanism here is presumably that strong states are able to solve coordination problems and engage in more economic activity.  Philip Osafo-Kwaako & James Robinson also find that stronger precolonial states lead to better development outcomes today, although they argue that centralization wasn’t driven by warfare like Dincecco, Fenske and Onorato suggest.

It’s a really interesting literature, and I think it would be even stronger with more of a focus on mechanisms, and more explanatory case studies.  If you look at subnational examples within Ghana and Uganda, you do tend to see stronger economic growth in the southern parts of those countries where precolonial polities were strongest (the Asante and Buganda kingdoms, respectively).  But does this mean that the kingdoms were solving coordination problems somehow, or that centralized states simply arose where the economic prospects were better in the first place?  Similarly, the link between precolonial centralization and contemporary civil war isn’t very intuitive to me.  Civil war is badly overdetermined in Africa, in that most countries fit the criteria (poverty and weak institutions) that are thought to increase civil war risk.  Academics still don’t seem to have a good model of why war happens when and where it does, rather than looking at aggregate risk factors, and I think until we understand more about the specificity of civil war it’s hard to know how to add precolonial centralization into the equation.