The World Bank’s Development Impact blog had two great posts recently which touched on the idea of contextualizing RCTs. David McKenzie, writing about clinical equipose in RCTs, says that it would be useful to do more experiments on targeting interventions, to understand how targeting needs might differ across countries. And both he and Eva Vivalt are concerned that impact evaluations rarely include the costs of the intervention in published materials, or, better yet, compare the intervention to a cash transfer of equivalent value. Development agencies who hope to implement a “proven” intervention will have more difficulty doing so if they can’t learn about the costs of implementation. (I think this is why Ted Miguel & Michael Kremer’s paper on school-based deworming has gained so many eager implementers: they included a persuasive cost-benefit analysis. Of course, it helps that deworming is very cheap in the first place.)
I would love to see more information on two additional topics: how an intervention was administered, and how the respondents themselves understand the effects that it had on their lives. Most impact evaluations provide a solid overview of the program or policy being implemented, but don’t offer much other local context. For example, what is the relationship of the implementing NGO/bank/government branch to the respondents? Is it seen as favoring some community members over others, or as even-handed towards all? Has it been active in the community for a long time, or is it a newcomer? A new organization may not have earned much trust among the community. At the same time, people may participate more often than they otherwise would if the intervention is offered by a well-established organization, in order not to offend them and possibly lose access to future services.* It’s reasonable to think that respondents’ beliefs about the implementing organization may have an effect on their participation in an intervention, and it would be useful to understand more of this context, perhaps through key informant interviews. (In fact, you could write a much longer list of contextual effects – whether there was a banking scandal recently, whether the last round of animal donations included sick goats which infected the rest of the town’s herd and then died, whether there was an especially good harvest the previous year which left everyone flush with cash, etc.)
I also think that there’s a great need to hear more from respondents themselves about how an intervention affected them. As one of my former colleagues at IPA (I think it was Liz) pointed out to me recently, there’s no reason for RCTs to be constrained to quantitative data collection. Doing more qualitative process tracing would be a useful way for researchers to examine whether they’re correctly identified the mechanisms underlying an observed social change, or whether respondents themselves perceive things differently.** For instance, in a 2007 paper, Xavier Gine and Dean Yang found that farmers in Malawi were half as likely to take out a loan when it came bundled with a completely free crop insurance product. Based on some suggestive correlations in the data, they believe this occurred because it was harder for less-educated farmers to evaluate the value of the crop insurance, and that this dissuaded them from accepting the loan. It’s a reasonable explanation, but one can easily imagine how it might have been more informative to ask the farmers themselves to discuss their decisions.
* While every implementing organization should make clear that the decision to participate or decline participation will not affect eligibility for future benefits, I’m skeptical about whether people actually believe this. If I had a rich aunt who was funding my studies, and she said, “I’d love it if you could reschedule those other plans for next week and come to my birthday party, but it’s quite all right if you can’t make it,” I would definitely think twice before skipping the party, even if I were completely convinced that she wasn’t about to defund me if I missed it. In many areas of our lives, access to resources depends on maintaining good relationships with others. I suspect that it’s hard to put this idea aside, even if an NGO clearly promises that future benefits don’t hinge on current behavior. It would be interesting to test this somehow, although I can’t think of any ethical way to do so at the moment.
** There is, of course, a much larger set of questions about voice and agency in the practice of international development, which I think is beyond the scope of this post but should still be acknowledged.