I loved this post from the World Bank’s Let’s Talk Development blog on uncovering researchers’ own biases in survey design:
Last year, I was in Nairobi, Kenya … to set up the data collection efforts for a four-country study. One of the goals of this study was to replicate results from lab experiments that suggested poverty is a context that shapes economic decision-making amongst households.
One of our replication questions was a vignette proposed by Sendhil Mullainathan and Eldar Shafir in their book Scarcity. … Shafir and Mullainathan’s findings show that commuters in Princeton, NJ were more likely to say that they would travel to another store for a $50 discount when purchasing a $100 product than when purchasing a $1,000 product. Less affluent individuals at a soup kitchen in Trenton, NJ, however, did not display this kind of inconsistency. For them, the marginal utility of money remained constant, regardless of the cost of the hypothetical product.
With the aim of replicating this question in four developing countries, we took the vignette to … Nairobi. After spending two days field-testing this hypothetical scenario, I was surprised that most of the respondents, particularly those from low-income households, said that they would not travel for the discount. Why was everyone hesitant to travel, regardless of the discount amount? Why are they seemingly exhibiting the preferences of the more affluent in the US?
Somewhat surprised by this discrepancy, I asked multiple respondents’ the reasons behind their choice. One person replied quite plainly, “There is no guarantee that the product will still be there once I go across town. It’s very likely that the product is gone by the time I get there.” Of course! By assuming the availability of the product, we had let our own implicit biases, based on our mental models, influence the design of the question. Since the original question was conducted in the United States, a developed country, implicit in the question was the assumption that availability is generally not a problem. However, for the respondents from less affluent communities, this assumption was not explicit.