I think this is a really important step towards acknowledging that policymakers face lots of constraints in using research results, and we need to move beyond ideas like “hold more dissemination conferences” to overcome them. Check out the whole list of recommendations below.
Apolitical recently published a profile of Burkina Faso’s national cash transfer program, which grew out of a pilot funded by the World Bank. It’s an interesting contribution to the recent discussion about scaling up successful interventions which has been going on at places like Vox and Evidence Action.
One of the main points is that expanding a pilot already run by the government may be more feasible than having the government adopt a program previously run by NGOs.
But the World Bank evaluation did make an important difference to the design of the national policy. One valuable factor was the way the trial involved the government from the beginning, creating expertise among local officials before the national program was launched.
That’s quite unusual, de Walque said. “What you find often is it’s done by some local or international NGO,” he explained, which means the government is less familiar with the program it’s trying to implement.
In Burkina Faso, the cash transfer trial was organised by a senior government official. “The scaling up is more likely to be successful if people from the government use the pilot as a training ground,” de Walque suggested.
As well as involving senior figures from an early stage, the trial created a pool of qualified employees for the early stages of the national program. Local workers who were hired and trained to implement the pilot were top candidates to help launch the policy at scale.
Another takeaway is that it’s likely a pilot program will need to be simplified to be implemented at scale — but understanding how to simplify it is crucial.
Creating this kind of [government] ownership and involvement is valuable because of the way governments inevitably leave out some details from a pilot. “Obviously when you go to a larger scale governments, and probably rightly so, at least in the first attempt, choose more simple programs,” de Walque said.
If the officials in charge have direct experience from the trial stage, they’re more likely to know which simplifications are feasible and which could seriously undermine the program.
Given my background in development economics and political science, it’s no surprise that I’m excited by the work that Evidence Action does to translate rigorous economic research into policy implementation. Karen Levy and Varna Sri Raman recently published a remarkably frank blog post discussing the challenges they faced when scaling up an anti-poverty program in Bangladesh after a successful pilot. The post stood out to me not only for its honesty about the difficulties of implementing at scale, but also for the amount of thought that EA and its implementing partner put into diagnosing and correcting the problems at hand.
The intervention at hand was the “No Lean Season” program. In a pilot project, Gharad Bryan, Shyamal Chowdhury, and Mushfiq Mobarak gave rural residents small subsidies to temporarily migrate to cities to look for work during the hungry season before annual harvests. They found that this substantially increased consumption in the sending households. It’s a clever response to the shortage of non-farm employment opportunities in rural areas, and also demonstrates how even small costs can prevent people from accessing better-paid opportunities elsewhere.
EA’s Beta Incubator subsequently worked with a Bangladeshi NGO to expand the subsidy program from about 5000 households per year up to 40,000. It was switched from a pure subsidy to a loan in the process However, they found that the NGO employees who were supposed to deliver the loans handed out fewer than expected. In addition, the loans didn’t seem to have the same effect, as recipients didn’t seem much more likely to migrate than a comparison group which didn’t receive any money.
The section of the EA post that’s really worth reading is the analysis of why the scaling didn’t go according to plan. It stood out to me for its use of both qualitative and quantitative methods to better understand the newly scaled-up context in which the program operated, and the internal operations decisions of its partner NGO. Among the salient points, they found that the program had been expanded into new districts which had much higher baseline rates of migration than the district in which it was piloted. A miscommunication with the NGO also meant that employees had performance targets set for the number of loans to disburse which were lower than the program actually required.
This is arguably the best example I’ve ever seen of why questions about the external validity of social policy RCTs are beside the point. Any program has to be adapted to its local context — and that context can vary significantly even at different scales of implementation, or between different districts in the same country.
Jesse Driscoll recently wrote a fantastic post for Political Violence at a Glance reflecting on survey work as statebuilding in Somalia. This was drawn from his experience doing one of the country’s first representative surveys in decades for this paper. One important point is that survey work is never positionally neutral — and this lack of neutrality is amplified in a conflict zone:
Our discouraging conclusion, after a 5-year study, was that practically any kind of intervention that touched the lives of Somali’s most vulnerable would invite skepticism about researcher motives—and perhaps rightly so. To the extent we were neutral observers we could be accused of engaging in virtual poverty tourism. To the extent we were something other than neutral observers, however, we were aspirational partisans. One of our Somali enumerators once asked, point blank, if we were being funded by the US military to put together a predator drone list. We weren’t, of course, but his concern was valid. Some of the most productive research programs in political science over the last decade produce knowledge that is explicitly (and unapologetically) seek-and-destroy.
Census knowledge in particular is not a public good, in the economic sense of the term:
An inaugural survey of a landed population after a civil war is not a pure public good, but more akin to club goods for politically powerful social groups (who stand to benefit most from counting and will, predictably, design survey/census categories to benefit them). Residents inclined towards distrust of political centralization may wish to remain invisible.
(The title here, if you didn’t catch the reference, is a play on statebuilding scholar James Scott’s most famous work.)
Ajoy Datta had a good post at Research to Action recently about how Indonesian policymakers interact with research evidence. Here are some of his key points. First, policymakers are interested in evidence, but they tend to look for data rather than papers initially:
Our results show that when mid-level Indonesian policymakers in both large ‘spending ministries’ and smaller ‘influencing ministries’ are tasked with, say, developing or revising a regulation or law, their first priority is to acquire not research, but statistical data. Seen as objective, policymakers feel data will, for instance, identify current trends, recognise issues that need to be addressed, assign targets, and/or demonstrate impact.
However, the reality is that some policymakers find it difficult to access high-quality data, while others struggle to make sense of the huge volume of data that exists. Data on its own fails to show the causes of trends and does not point to potential solutions. This is where research can help.
Second, if policymakers want more context for the data they find, they’re fond of inviting experts in for discussions:
Most importantly, however, when policymakers did seek out research, rather than commission or read comprehensive research papers, they are more likely to invite experts they already knew to provide advice through social processes (which some policymakers consider as research). These processes usually feature formal and informal meetings or phone conversations, focus group discussions (FGDs), or seminars.
Part of this is because of constraints on the ability to either rapidly access existing research, or commission new papers on specific topics:
Procedures to procure research from internal research and development units, where they exist, is lengthy and cumbersome. This usually discourages them from making a request at all. In any case, these internal units often lack the capacity to produce high-quality research. Meanwhile, other procedures constrained policymakers from hiring top-end researchers from outside government to undertake research.
The main takeaway is that the social process of building trust between researchers and policymakers matters a great deal. This certainly poses a challenge for academics, as creating these relationships takes time, and unfortunately doesn’t count towards one’s tenure packet.
Here’s the latest edition of my Africa Update newsletter. We’ve got Mali’s 35-year old foreign minister, the dodgeball association of South Sudan, accountability for Mozambican mayors over gay rights, the future of nuclear power on the continent, and more.
Southern Africa: At some South African universities, nearly 80% of black students report that they sometimes don’t have enough to eat. A South African court has ruled that marriages between Muslim couples in the country must be legally registered and not simply recorded with religious authorities, giving women legal protection in the event of divorce. Zimbabwe’s harsh laws criminalizing the transmission of HIV are discouraging people from coming for testing and treatment.