The English cartoonist Ashleigh Brilliant once offered the following piece of advice to strategists of all sorts who are concerned with their reputation: “To be sure of hitting the target, shoot first, and call whatever you hit the target…” With little time and fewer resources than elsewhere to battle the burning issues of poverty, insecurity and sociopolitical instability, economists and policymakers in developing countries may not be in the position to benefit from such cynical wisdom. Rather than listening to Ashleigh Brilliant, they should always keep in mind the constraints they face and the urgency of the situation in poor countries, and reflect on the maxim that recommends to “always aim before shooting.
A policy and research domain where there is a serious deficit of strategic thinking and prioritization is that of evaluation, which is traditionally defined as the systematic assessment of the worth or merit of some project, program or policy. The importance of evaluation cannot be underestimated: first, in a world where ideas compete constantly for funding, it is essential to ensure that value for money is at the core of public policy. Second, only by assessing the pertinence and efficiency of development initiatives can we get a full picture of their outcomes, and ensure accountability. Third and perhaps even more importantly, evaluation helps define the criteria for decision-making on new initiatives, and chart the course of future action. It highlights what works and what does not. It is therefore not surprising that evaluation has become a hot area of research and policy.
For good reasons, randomized controlled trials (RCT) are among the most popular new approaches and methodologies for impact evaluation. They suggest applying the simplicity and robustness of RCT techniques--the same approach used by the medical industry to determine if a drug or treatment does what it was designed to do--to poverty interventions to determine whether or not a program is effective. A noble goal, indeed, and a clever idea. However, as my colleague Martin Ravallion has pointed out one can wonder whether RCTs are really delivering on their main goal and promise. (See Martin’s articles “Should the Randomistas Rule?” and “Evaluation in the Practice of Development.”) Assessing the impact of specific projects, no more than one at a time, without taking into consideration the many sources of heterogeneity relevant to behavior and the interaction effects--the fact that each such project is only one component of a development portfolio that often cuts across sectors--is a recipe for biases. The very paradox of RCTs is that they do not allow for a random selection of projects and programs. While they certainly provide some useful “micro” feedback to a variety of audiences including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies, they have few secrets to yield to policymakers facing big strategic “macro” decisions. As the Chinese would say, RCT researchers may be spending most of their time trying to pick up sesame seeds but neglecting or even throwing away watermelons. In other words, they may be concentrating on small matters to the expense of some important ones because the small matters happen to be amenable to RCTs.
To be sure, some questions asked by the “randomistas” are interesting and relevant. But one constraint with this approach is that it is not suitable for and may exclude other learning opportunities that are at least equally important for improving our understanding of how to reduce poverty. After all, not too long ago, most countries in the world were poor. Yet some have managed to break the poverty trap, start a sustained dynamic growth and become middle-income or even high-income economies, sometimes in a matter of just one or two generations. Outside the Western Europe and North America, Japan was the first economy in Asia to start this dynamic transition process after the Meiji Restoration in the late 19th century, followed by Korea, Taiwan-China, Hong Kong-China, Singapore, Malaysia, Indonesia, and Thailand in the 1960s, and China, Vietnam, Cambodia and Laos in the 1980s. This dynamic process has spread beyond the East Asia to Mauritius in the 1970s, India in the 1990s, and even African countries in the 2000s. In fact, among the 29 countries that grew annually at 7 and above percent in 2000-2008, 11 of them are in Sub-Saharan Africa. In terms of the impact of a country’s sustained, dynamic growth on poverty reduction, 600 million people in China alone have been lifted out of extreme poverty since 1980.
The governments in the above mentioned successful countries must have drawn valuable policy lessons or at least inspiration from the successful countries before them in the formulation of policies that unleashed their growth potentials. For economists engaged in research on poverty reduction, one of the “watermelon” questions should be why and how some countries succeeded and others failed to make it out of poverty, so that the countries that remain trapped into poverty can derive policy insights from their experiences to avoid mistakes and start successfully a sustained, dynamic growth in their countries.
RCTs and other evaluation techniques are not designed to analyze these big success and failure experiences as results of policy “experiments” of many governments. By focusing on individual development projects or programs—even in a “radiographic” fashion—they explore relevant but “sesame seed” type of questions. They may provide interesting information about particular trees but they do not really help uncover the secrets of the forest. I hope that the enthusiasm about RCTs will not cause the development research community to “throw away watermelons for picking up sesame seeds”.
Development economists should ignore Ashleigh Brilliant’s humoristic suggestion of considering as legitimate targets for their investigations and good subjects of research any topic can be studied with currently fashionable methodologies—regardless of whether they actually generate useful new knowledge or not. After all, the ultimate test of relevance is not the desirability of a certain research methodology, but the importance of the question that the research asks and the policy insights that such research generates. The greater the extent to which the policy insights generated by a research help a country start a sustained, dynamic, and inclusive growth, the more important is the research.
Join the Conversation