Making x-centric less eccentric

Lant Pritchett’s latest post about the limits of randomized controlled trials in development economics has been making the rounds of the small universe of people who care deeply about randomized controlled trials for a few weeks.  His critique, of course, is that there’s a fad for examining whether “intervention X affects outcome Y” (or “x-centric” research), but researchers often give too little attention to whether the proposed intervention is feasible and cost-effective outside the context of an academic study.

This line of criticism isn’t new, and most people I know who do development RCTs would probably agree with it.  There’s a lot of work already underway to remedy some of these shortcomings.  To take several of Lant’s points in order:

“X-centric can become eccentric by being driven by statistical power.”  Lant’s point here is that many questions we might care about, such as why China grew so rapidly after the 1970s, or why some countries have better educational outcomes than others, aren’t amenable to randomization.  This is very obviously true, and I don’t know a single person who argues that RCTs are the only valid research method for every question in economics. As the graph below shows, RCTs are still a minority of all published research in the discipline. There’s also a lot of interesting case-based research that addresses these issues, although you sometimes have to go next door to political science to find it.  Two examples that come to mind are Douglass North, Jim Wallis, and Barry Weingast’s work on the institutional prerequisites for economic growth, or Stephen Kosack on the politics of education in Taiwan, Ghana and Brazil.

The image shows a graph demonstrating that RCTs are still a fraction of all published papers in most economics journalsGraph via David McKenzie

“X-centric can become eccentric by never asking how big.”  The idea here is that many published development RCTs have results which are statistically significant, but substantively small.  For example, a study might report the headline result that tutoring improves students’ test scores — but the substantive impact might only be a difference of one percentage point.  This is definitely a challenge, and I think it’s exacerbated by economists’ tendency to present their results to non-specialists using statistical terms of art (like standard deviations) rather than more straightforward measures (like percentage point changes in test scores).  One organization that is taking some good steps towards comparing impact size across interventions is AidGrade, which has built an online tool for anyone to carry out their own meta-analysis of aid effectiveness.

“X-centric can become eccentric by ignoring external validity.”  This is the issue addressed by Evidence in Governance and Politics’ Metaketa Initiative, which offers funding for clusters of studies which examine similar interventions in different countries.  Current projects focus on questions of information and accountability, taxation, natural resource governance, and community policing.  There are also one-off initiatives like IPA’s series of Ultra Poor Graduation pilots, which replicated the same social protection intervention in seven different countries.

“X-centric can become eccentric by ignoring implementation feasibility.”  I find this critique a bit curious because it assumes we know ex ante which types of interventions will or won’t work in a given context.  One could easily assume that it wouldn’t be possible to provide biometric identification for 99% of Indian citizens, or get 94% of children in Burundi into primary school — twenty percentage points higher than the regional average, in one of the poorest countries in Africa.  But there is a valid point here that simply knowing that an intervention is effective doesn’t automatically translate into the political will to implement it on a large scale.  Organizations like Evidence Action and Evidence Aid are tackling this challenge by working with governments and NGOs to share information about successful interventions and scale them up.  Rachel Glennerster and Mary Anne Bates of JPAL have also created a new framework for assessing when an intervention can be successfully scaled or used in different country contexts.

Links I liked

The photo shows a beachfront scene, framed by a window, in Durban, South AfricaThinking of this beautiful view in Durban on a rainy day here in Berkeley

The image shows a tweet from Tolu Ogunlesi, expressing admiration for the percentage of books on South Africa which are by South African authors

  • Enthusiasm for universal basic income is spreading, with new pilot projects recently announced in Scotland and Finland.  An interesting argument for the positive effects of UBI is that it already exists for the 1% in the form of capital income.

Fall conference highlights

Three weeks, three conferences, many great papers to share!

apsa_4c_logo-300

Ryan Briggs.  “Power to which people? Electoral politics and electrification targeting in Ghana.”  Presented at APSA.

This paper isn’t available yet, and there isn’t an abstract online for it.  However, it focuses on a very interesting puzzle: why do the two main political parties in Ghana differ in their willingness to provide electricity to their core supporters?  I won’t give away the ending here, but the general message is that history and ideology matter more for distributive politics in Africa than is often assumed.  There’s some background in a blog post here.

Abhit Bhandari.  “Political Determinants of Business Formalization.”  Presented at APSA.

This paper also isn’t available yet, which I should take as a sign that I found all the panels where people were presenting genuinely new research!  Another extremely interesting question, though: do the owners of informal businesses have political, rather than economics, reasons to enter the formal sector?  Abhit uses a new dataset of all formal business registrations in Senegal to explore this issue.

Shervin Malekzadeh. Education as Public Good or Private Resource in Postrevolutionary Iran.”  Presented at APSA.

A third great paper that’s not online yet!  It seems to draw from Shervin’s PhD thesis, so I’ll share the abstract for that.

Abstract: This project examines efforts by the Islamic Republic of Iran to produce loyal “Islamic Citizens” through its postrevolutionary school system as way of securing the hegemonic rule of the state. Drawing upon eighteen months of fieldwork in Iran, including archival research of textbooks published from 1979 to 2008, as well as interviews and participant observation in two private Islamic high schools in Tehran, I show that Iranian schools have both emancipatory and disciplinary effects on students. Ordinary Iranians do not blindly accept or internalize the ideology of the state, instead resisting, reinterpreting or even ignoring aspects of the postrevolutionary project taught to them in school. Yet they often do so using the language, practices, and formal procedures of dominant groups. The dissertation demonstrates the incoherent and contested nature of the New Islamic Citizen, a concept that has changed often and dramatically over the past 30 years. Competition between rival groups for the moral authority to insert their vision of the ideal Islamic society into the education system accounts for the variation in the political and religious content of formal education. These ongoing and unresolved conflicts have resulted in a postrevolutionary curriculum layered with contradictions and tensions that in turn provide students with the resources and opportunities to challenge the totalizing project of the state. The dissertation reveals the relationship between the politics of schooling and the politics of nationalism in Iran. Looking beyond the usual antinomies of domination/resistance, modern/traditional, or secular/religious attached to the study of political socialization in postrevolutionary Iran, this dissertation contends that interactions of state and society around the topic of schooling contributes to the production of a mutually produced and shared Islamic-Iranian framework for consent and opposition to state rule. This discursive framework is but the latest manifestation of a 200-year effort in Iran to produce an indigenous modernity rooted in an “authentic” and shared national culture.

76376a0fad3cf5f7-dsa

Ken Mitchell.  “Taxation after the Commodity Boom: Argentina, Chile and Brazil.”  [Paper forthcoming.]  Presented at DSA.

Abstract: Latin America’s commodity boom crested round 2010, and the regional terms-of-trade deteriorated thereafter. How commodity price declines might impact the region is a pivotal and potentially a troubling socioeconomic issue. Taxation is a significant area of interest because tax-to-GDP ratios rose fast during the commodity boom (2002-2010) and paid for new social programs (conditional cash transfers, popular sector pensions, etc.) and public employment schemes that expanded the middle class and lowered poverty. More consumer spending resulted, which fueled economic growth. Latin America has been the world’s lowest taxed region post-WWII, and the regional tax-to-GDP ratio flattened during the 1990s, so the end of the commodity boom rightly raises concern that public revenue mobilization might revert to its historical, low norm. Did tax-to-GDP ratios decline with commodity prices? Aggregate taxation combines varied taxes (i.e., income, corporate, consumption, trade, etc.), and here country case studies differ. The Value Added Tax (VAT) merits special attention due to its rising importance as a revenue tool across Latin America. Which national tax strategies managed to maintain commodity boom-era tax-to-GDP ratios? This paper tries to answer the above question by comparing Argentina, Chile and Brazil between 2002 and 2014, with special attention to the period 2010-2014 (i.e., post-commodity boom). Counterintuitive given the literature on taxation specific to Latin America, Argentina, historically an especially low tax country outperforms its neighbors after 2010, something the paper tries to explain. The paper will use OECD tax data to make cross-national comparisons.  [Note: the interesting conclusion is that rising use of credit cards in Argentina explains strong VAT collection even though economic growth slowed after 2010.]

Miguel Niño-Zarazúa.  “Natural resources, electoral behaviour and social assistance in Latin America.” [Paper forthcoming.]  Presented at DSA.

Abstract: The introduction of social assistance in Latin America in the late 1990s coincided with a democratization process in the region and a significant increase in the contribution of revenues from non-renewable resources to the public budgets. This paper provides an analysis of the distributional effects of revenues from the natural resources via social spending. A primary concern is to establish whether the redistribution of income via social spending would have not taken place in the absence of natural resources. Another aspect of this relation is that lessons from Latin America can also provide insights into the political incentives that natural resource rents generate to the incumbent. Experimental and quasi-experimental studies suggest that social assistance programmes can produce electoral gains to the incumbent. Our working hypotheses are the following: H1) revenues from non-renewables have facilitated social spending in Latin America, and H2) natural resources have generated electoral gains to the incumbents in increasingly more competitive political systems. In order to test our hypotheses, we first examine the economics of redistribution via revenues from natural resources, with a particular focus on the incentives that drive incumbent decisions on social spending. Second, we consider a model of income redistribution in which an incumbent can make allocation decisions of public funds in the presence of taxation. We expand the model by allowing revenues from natural resources facilitating social spending without affecting the disposable income of better-off households. We empirically test our hypotheses using fixed effects estimators with instrumental variables in three stages. The results indicate that the expansion of social spending in Latin America over the period 1990-2009 has indeed been facilitated by the natural resource rents; however, the electoral gains hypothesis is not supported by the empirical analysis.

aenlogo

Nasreen Jessani, Caitlin Kennedy and Sara Bennett.  “The human capital of knowledge brokers: An analysis of attributes, capacities and skills of academic teaching and research faculty at Kenyan schools of public health.”  [Ungated draft here.]  Presented at Evidence 2016.

Abstract: Academic faculty involved in public health teaching and research serve as the link and catalyst for knowledge synthesis and exchange, enabling the flow of information resources, and nurturing relations between ‘two distinct communities’ – researchers and policymakers – who would not otherwise have the opportunity to interact. Their role and their characteristics are of particular interest, therefore, in the health research, policy and practice arena, particularly in low- and middle-income countries. We investigated the individual attributes, capacities and skills of academic faculty identified as knowledge brokers (KBs) in schools of public health (SPH) in Kenya with a view to informing organisational policies around the recruitment, retention and development of faculty KBs. During April 2013, we interviewed 12 academics and faculty leadership (including those who had previously been identified as KBs) from six SPHs in Kenya, and 11 national health policymakers with whom they interact. Data were qualitatively analyzed using inductive thematic analysis to unveil key characteristics. Key characteristics of KBs fell into five categories: sociodemographics, professional competence, experiential knowledge, interactive skills and personal disposition. KBs’ reputations benefitted from their professional qualifications and content expertise. Practical knowledge in policy-relevant situations, and the related professional networks, allowed KB’s to navigate both the academic and policy arenas and also to leverage the necessary connections required for policy influence. Attributes, such as respect and a social conscience, were also important KB characteristics.  Several changes in Kenya are likely to compel academics to engage increasingly with policymakers at an enhanced level of debate, deliberation and discussion in the future. By recognising existing KBs, supporting the emergence of potential KBs, and systematically hiring faculty with KB-specific characteristics, SPHs can enhance their collective human capital and influence on public health policy and practice. Capacity strengthening of tangible skills and recognition of less tangible personality characteristics could contribute to enhanced academic–policymaker networks. These, in turn, could contribute to the relevance of SPH research and teaching programs as well as evidence- informed public health policies.

Taryn Young.  “Policy BUDDIES – BUilding Demand for evidence in Decision making through Interaction and Enhancing Skills.”  Presented at Evidence 2016.

Summary:  Policy BUDDIES is a collaborative project including the Centre for Evidence based Health Care (CEBHC and Health systems and services research unit at Stellenbosch University, the South African Cochrane Centre (SACC), the Centre for the Development of Best Practices in Health (Cameroon) and the Liverpool School of Tropical Medicine. The collaborations draw on partners experience in working in the area of knowledge translation. The collaborations draw on partners experience in working in the area of knowledge translation. The project is being conducted in South Africa and Cameroon. The project consists of 5 phases  and commenced with a situational analysis which informed subsequent phases. The situational analysis aimed to understand policymakers’ capacity, as well as enablers and constraints related to demanding evidence during policy formulation and implementation, and to map existing communication between policymakers, research intermediaries and researchers. Health programme managers and programme coordinators in programmes related to MDGs 4, 5 and 6 at provincial level in Cameroon and South Africa were purposively selected and interviewed. One of the aspects which emerged is the need for capacity development in evidence informed policy making and the use of systematic reviews.  We drew on experiences in offering similar workshops as part of the Effective Health Care Research Consortium and the SUPPORT Collaboration and offered workshops in both Cameroon and South Africa in January 2014 and November 2013 respectively. We are currently implementing the buddy model linking researchers and policymakers to work together towards promoting evidence-informed policy making.

policy-buddies-phases-648x401.gif

Yvonne Erasmus.  “What works to build capacity to use research evidence in South Africa and Malawi.”  Presented at Evidence 2016.

Summary: The use of research evidence in decision-making by policymakers in the health sector is critical for enabling the formulation and implementation of the most effective policies and programmes. Despite that reality, many scholars have documented that policymakers often do not sufficiently use research evidence to inform their decisions because of several common barriers (see Innvaer et al 2002; Oliver et al 2014). One of the well-documented barriers to research use is the lack of knowledge and skills in finding, appraising, interpreting and applying evidence as part of the policymaking process.  The DFID-funded Strengthening Capacity to Use Research Evidence in Health Policy (SECURE Health) programme being implemented in Kenya and Malawi has designed a training programme for policymakers in the health sector to strengthen their knowledge and skills in accessing, appraising, synthesising and applying research evidence in policymaking (i.e. the SECURE Health evidence-informed policymaking (EIPM) training programme).  Through these workshops, 76 policymakers comprising Ministry of Health (MoH) and parliament staff from the two countries were trained.  The training workshops effectively increased the knowledge and skills of policymakers in finding, assessing, synthesizing and applying research evidence in their work.

Links I liked

Cn41OAvXgAA2tJjThis and other interesting graphs to be found at the new African Visual Data site

Summer conference highlights

The summer’s off to a good start with a lot of travel and two excellent conferences so far.

ABCDE-2016-Data-and-Development.jpg

Belinda Archibong. “Where Local Kings Rule: Long-Term Impacts of Precolonial Institutions and Geography on Access to Public Infrastructure Services in Nigeria.”  Presented at the World Bank’s Annual Bank Conference on Development Economics.

Abstract: Though previous works have discussed the benefits of precolonial ethnic state centralization for development in Africa, the findings, of a positive relationship between centralization and development and the mechanisms provided, of local accountability of ethnic state leaders, do not explain the heterogeneity in outcomes, reflected in the unequal distribution of access to public services among formerly centralized states today. Here, I find that centralization has had a negative effect on access to federally administrated, high state control goods when cooperation failed between ethnic state and autocratic federal government leaders in the kind of cooperative federalist regimes that defined much of colonial and postcolonial Africa. I focus on the case of Nigeria, and specifically, I find a significant negative effect of centralization on access to high federal state control goods for centralized states whose leaders failed to cooperate with the autocratic military regime, and whose jurisdictions were subsequently subject to a punishment regime, typified by underinvestment in public services, with lasting impacts till today. I also posit that the long-term effects of this punishment can be seen in the relatively lower reported trust in institutions of federal authority over traditional institutions today from respondents from these previously punished, centralized precolonial states.

Toni Oki.  “Bandits on Patrol: An Analysis of Petty Corruption on West African Roads.”  Presented at ABCDE.

Abstract: This paper explores the spatial determinants of petty corruption on West African roads, employing a unique micro-dataset on bribes extorted from truck drivers by officials at various checkpoints. First, I use road traffic levels to predict the spatial distribution of corruption, finding a broadly inverted-U relationship.  [NB: the highlight of the theoretical model is that high traffic levels should lower the value of bribes demanded, rather than the converse.] Secondly, I investigate how regional favouritism might affect this distribution. When a new president comes into power in Mali, bribe values in his birth region change . This change is heterogeneous: there are both winners and losers with in his region. Finally, I critique my theoretical framework by finding an unusually large relationship between bribery and rainfall.

Kristen Himelein, Stephanie Eckman, Siobhan Murray and Johannes Bauer.  “Second Stage Sampling for Conflict Areas: Methods and Implications.”  Presented at ABCDE.

Abstract: The collection of survey data from war zones or other unstable security situations is vulnerable to error because conflict often limits the implementation options. Although there are elevated risks throughout the process, this paper focuses specifically on challenges to frame construction and sample selection. The paper uses simulations based on data from the Mogadishu High Frequency Survey Pilot to examine the implications of the choice of second-stage selection methodology on bias and variance. Among the other findings, the simulations show the bias introduced by a random walk design leads to the underestimation of the poverty headcount by more than 10 percent. The paper also discusses the experience of the authors in the time required and technical complexity of the associated back-office preparation work and weight calculations for each method. Finally, as the simulations assume perfect implementation of the design, the paper also discusses practicality, including the ease of implementation and options for remote verification, and outlines areas for future research and pilot testing.

Centre for Social Protection

Stephen Devereux, Edoardo Masset, Rachel Sabates-Wheeler, Michael Samson, Althea-Maria Rivas and Dolf te Lintelo.  “Evaluating the Targeting Effectiveness of Social Transfres: A Literature Review.”  Presented at the short course on social protection at the Institute for Development Studies’ Centre for Social Protection.

Abstract: This paper reviews empirical evidence from a range of social protection programmes on the accuracy of these mechanisms, in terms of minimising four targeting errors: inclusion and exclusion, by eligibility and by poverty. This paper also reviews available evidence on the various costs associated with targeting, not only administrative but also private, social, psycho-social, incentive-based and political costs. Comparisons are difficult, but all mechanisms generate targeting errors and costs. Given the inevitability of trade-offs, there is no ‘best’ mechanism for targeting social transfers. The key determinant of relative accuracy and cost-effectiveness in each case is how well the targeting mechanism is designed and implemented.

Ugo Gentilini.  “Entering the City: Emerging Evidence and Practices with Safety Nets in Urban Areas.”  Presented at CSP.

Abstract. Most safety net programs in low and middle-income countries have hitherto been conceived for rural areas. Yet as the global urban population increases and poverty urbanizes, it becomes of utmost importance to understand how to make safety nets work in urban settings. This paper discusses the process of urbanization, the peculiar features of urban poverty, and emerging experiences with urban safety net programs in dozens of countries. It does so by reviewing multidisciplinary literature, examining household survey data, and presenting a compilation of case studies from a ‘first generation’ of programs. The paper finds that urban areas pose fundamentally different sets of opportunities and challenges for social protection, and that safety net programs are at the very beginning of a process of urban adaptation. The mixed-performance and preliminary nature of the experiences suggest to put a premium on learning and evidence-generation. This might include revisiting some key design choices and better connecting safety nets to spatial, economic and social services agendas compelling to urban areas.

Francesca Bastagli.  “Bringing taxation into social protection analysis and planning.”  Presented at CSP.

Abstract: The expansion of social protection in low- and middle-income countries over the last two decades has been accompanied by a growing number of studies on the distributional impact of social protection spending. When such analyses consider social protection separately from tax policy, they provide a partial picture of the poverty and inequality impact of fiscal policy. In addition to determining the net distributional impact of fiscal policy, tax revenue levels and ‘mix’ matter to the resources available for social protection financing and its sustainability over time. Efforts to support and increase social protection spending in a sustainable fashion to meet poverty and inequality reduction goals are increasingly looking at options to increase revenue through taxation. This paper contributes to efforts to include tax considerations in social protection analysis and design by discussing the key methodological issues in carrying out joint distributional analysis, reviewing the evidence on the incidence and distributional impact of taxes and transfers and discussing alternative tax revenu e sources and their implications for social protection financing and sustainability.