I received an email from my friend Jon Stever sometime ago recounting an exchange he had with Paul Collier at a conference:

I went to see Paul Collier speak at the LSE about his penultimate book…[and] asked him to explain the usefulness of his methodology to his critics (like me) who think his results fall into one of two categories: 1. obvious tautologies or near-tautologies. or 2. those that lend themselves to potentially dangerous extrapolation (or both).

I gave two examples from his lecture that night. Here is one: He said last night that he had found that the statistical relationship between individual leadership characteristics and economic growth didn’t hold when you controlled for the fairness of elections. Moreover, he discovered that individual leadership characteristics effected growth rates only when elections were ‘dirty’ and that individual leadership characteristics did not effect growth rates when elections were ‘clean’. This is a ‘sexy’ finding and sounds interesting. But, you can rephrase this same statement into a simple tautology: leaders are not stronger than institutions (ie. leaders do not have a greater impact than institutions on growth) when institutions are stronger than leaders (ie. when institutions prevent leaders from cheating elections). Vice versa: leaders are stronger than institutions when institutions are not as strong as leaders.

Unfortunately, he completely skirted my broader invitation to explain the usefulness of his methodology. Instead he focused on the one point about leadership and muddled through an answer; he said that he thought it was an interesting finding and that he didn’t know what the data would tell him in advance etc.  I wasn’t convinced by [this response]…

[It raises] several interesting questions, in my mind, about Collier’s style and methodology:

1. Would some of Collier’s more unpalatable findings–such as that democracy doesn’t work in poorer countries–be more (or less) widely accepted if numbers were not telling the story?
2. How should the results of randomized testing be used to develop policy interventions? More specifically, would an indication of success through randomized testing in country X imply that such a policy would be useful in country Z?
3. As a public intellectual is it necessary to take extreme, contentious, or overly simplistic views? Should we, therefore, apply a massive public intellectual discount to people like Collier’s statements?

I found the first two questions especially interesting in light of the recent Microfinance Impact & Innovation conference, where a number of the same questions of narrative context and cross-country generalizability were raised.  (Tim Ogden has a thorough round-up of blog reactions to the conference here.)  Part of the ceteris paribus condition between control & treatment groups in an RCT is inevitably the broader country environment in which the experiment is taking place – and once you start making cross-country observations, well, the ceteris is no longer paribus.  Of course, Collier-style observational studies of governance at the national level can only be cross-country, and you can’t ever statistically control for all sources of variation between two countries.

It makes me wonder if there’s a certain level of complexity up to which either randomized or observational studies can in fact be generalized outside of their original contexts.  IPA folks talk a lot about the Kenya school deworming study, which showed that giving inexpensive deworming medication to schoolchildren improved educational outcomes, and I think it’s become a popular example in part because it’s so obviously generalizable to non-Kenyan contexts.  It’s rooted in biological fact.  At a slightly higher level of complexity, Erica Field had a good paper at the MII conference showing that modifying the design of Indian microfinance contracts to allow longer grace periods before repayment increased both profits & defaults among participating clients.  Assuming equivalent oversight, there seems little reason to assume that the psychological & financial aspects of a grace period might not produce similar results if implemented in Latin America or Africa.  But a country is a unit of observation that’s orders of magnitude larger and more complex than a single borrower, and it’s perhaps unsurprising that the observed nature of governance across countries is too variable, too path-dependent, to allow such cleanly identifiable relationships to exist.