On Paul Collier, and whether statistical research should be taken seriously

I received an email from my friend Jon Stever sometime ago recounting an exchange he had with Paul Collier at a conference:

I went to see Paul Collier speak at the LSE about his penultimate book…[and] asked him to explain the usefulness of his methodology to his critics (like me) who think his results fall into one of two categories: 1. obvious tautologies or near-tautologies. or 2. those that lend themselves to potentially dangerous extrapolation (or both).

I gave two examples from his lecture that night. Here is one: He said last night that he had found that the statistical relationship between individual leadership characteristics and economic growth didn’t hold when you controlled for the fairness of elections. Moreover, he discovered that individual leadership characteristics effected growth rates only when elections were ‘dirty’ and that individual leadership characteristics did not effect growth rates when elections were ‘clean’. This is a ‘sexy’ finding and sounds interesting. But, you can rephrase this same statement into a simple tautology: leaders are not stronger than institutions (ie. leaders do not have a greater impact than institutions on growth) when institutions are stronger than leaders (ie. when institutions prevent leaders from cheating elections). Vice versa: leaders are stronger than institutions when institutions are not as strong as leaders.

Unfortunately, he completely skirted my broader invitation to explain the usefulness of his methodology. Instead he focused on the one point about leadership and muddled through an answer; he said that he thought it was an interesting finding and that he didn’t know what the data would tell him in advance etc.  I wasn’t convinced by [this response]…

[It raises] several interesting questions, in my mind, about Collier’s style and methodology:

1. Would some of Collier’s more unpalatable findings–such as that democracy doesn’t work in poorer countries–be more (or less) widely accepted if numbers were not telling the story?
2. How should the results of randomized testing be used to develop policy interventions? More specifically, would an indication of success through randomized testing in country X imply that such a policy would be useful in country Z?
3. As a public intellectual is it necessary to take extreme, contentious, or overly simplistic views? Should we, therefore, apply a massive public intellectual discount to people like Collier’s statements?

I found the first two questions especially interesting in light of the recent Microfinance Impact & Innovation conference, where a number of the same questions of narrative context and cross-country generalizability were raised.  (Tim Ogden has a thorough round-up of blog reactions to the conference here.)  Part of the ceteris paribus condition between control & treatment groups in an RCT is inevitably the broader country environment in which the experiment is taking place – and once you start making cross-country observations, well, the ceteris is no longer paribus.  Of course, Collier-style observational studies of governance at the national level can only be cross-country, and you can’t ever statistically control for all sources of variation between two countries.

It makes me wonder if there’s a certain level of complexity up to which either randomized or observational studies can in fact be generalized outside of their original contexts.  IPA folks talk a lot about the Kenya school deworming study, which showed that giving inexpensive deworming medication to schoolchildren improved educational outcomes, and I think it’s become a popular example in part because it’s so obviously generalizable to non-Kenyan contexts.  It’s rooted in biological fact.  At a slightly higher level of complexity, Erica Field had a good paper at the MII conference showing that modifying the design of Indian microfinance contracts to allow longer grace periods before repayment increased both profits & defaults among participating clients.  Assuming equivalent oversight, there seems little reason to assume that the psychological & financial aspects of a grace period might not produce similar results if implemented in Latin America or Africa.  But a country is a unit of observation that’s orders of magnitude larger and more complex than a single borrower, and it’s perhaps unsurprising that the observed nature of governance across countries is too variable, too path-dependent, to allow such cleanly identifiable relationships to exist.

4 thoughts on “On Paul Collier, and whether statistical research should be taken seriously

  1. I enjoy reading Colier’s work, but I think it’s important to remember that when running cross-national studies the number of data available for analysis is really not very large… ESPECIALLY in the context of such complex systems. We often include data that have large issues with externalities; in psychology they wouldn’t even make it past the screening process. Yet in order to have something resembling a legitimate sample size we try to just correct for all of these complexities (which is next to impossible) and then generalize the results despite having an analysis which would be ripped apart in any other field. If we had a world 100x the size, with 100x the number of nations to choose from, and the ability to more properly construct an analysis I think the generalizations would be more useful. Currently, it’s like we’re trying to analyze the effect of a medication and our test subjects consist of a bat, dog, human, whale, etc… unless it’s something as dramatic as cyanide the findings are going to be borderline worthless. We need 100 subjects of the same species, at least.

    Like

  2. I’m not that convinced the results are so tautological. It’s still perfectly plausible that the quality of leaders matters even in a context where institutions are good (clean elections). You can probably think of stories/models where the quality of a leader and the quality of institutions are complements, in which case you may well expect Collier’s results to go completely the other way (the post instead seems to assume perfect substitutability).

    Like

    1. @Nicholas That’s an interesting point – can you think of examples?

      @Lee Spoken like a true randomista. : ) Thanks for the link as well! That’s such a cool result.

      Like

Comments are closed.