Tomorrow and on Tuesday (October 24-25), there is a workshop at the World Bank titled “CCTs: The Second Generation of Evaluations.” If you are at the World Bank or in the DC area, you may want to make your way to this event, as it promises to be a good one – focusing on research conducted on the topic in the past three years or so. The website is here, according to which, there are three main goals: (i) present key results from selected recent high-quality impact evaluations, (ii) discuss the implications of those new lessons for policy (as well as their external validity) and (iii) identify gaps in existing knowledge regarding CCTs.
I am making a presentation in the session titled “Which CCT design features really matter?” presumably because our cash transfer experiment in Malawi is among a small, but growing, bunch of recent studies that explicitly examine that variation in the impact of these programs along various key design parameters. Since we have multiple papers from this study, preparing for this talk gave me a good opportunity to summarize all the things we have learned from this experiment so far. I share these with you below, with the obvious caveat that these findings are from one experiment in one country, so they are not meant to be universal lessons (although I would argue that many of the findings have some claim to external validity because of the careful study design and the framework within which they're considered).
The Zomba Cash Transfer Program was designed to examine the importance of (a) the conditionality in CCTs programs, (b) the size of the transfer, (c) the identity of the transfer recipient within the household, and (d) spillover effects. In addition to these, it was designed to examine the roles of income and schooling in affecting the risk of HIV infection among school-age girls and young women.
Lessons so far:
1. Unconditional cash transfers (UCTs) did cause a small reduction in dropout rates, but CCTs were much more cost-effective: The big debate in cash transfers in the past few years has been about whether the conditions (to attend school, visit health clinics, etc.) are needed. Our forthcoming paper "Cash or Condition" (advance access online here, ungated here) provides the first experimental evidence on this question. Previous literature comparing CCTs and UCTs (cited in the paper linked above) had suggested that it was the condition that mattered and that UCTs would do very little to improve enrollment rates or reduce child labor in Latin America. We find that the answer, in the case of Malawi, lies somewhere in between: the UCTs were not completely ineffective in increasing school enrollment, reducing the 20% dropout rate in the control group by 4 percentage points (pp) to approximately 16% during the two-year program. However, CCTs reduced the dropout rate to about 11% and our calculations show that schooling CCTs are easily more cost-effective than UCTs in reducing dropouts.
2. CCTs improved student achievement: Previous studies of CCTs, while finding sizeable impacts on school enrollment and grade attainment, did not, by and large, show any impacts on learning. The thought was that CCTs may be bringing back to school (or keeping in school longer) students who were either not interested in school or less able to learn. In Malawi, the CCT treatment caused statistically significant improvements (between 0.12 and 0.17 standard deviations) in mathematics, English reading comprehension, and cognitive skills. We did not detect any similar effects in the UCT arm. There is also some evidence that, among girls enrolled in school, those in the CCT arm were in class more frequently. These learning gains are encouraging for CCTs programs in Africa and it will be interesting to see whether they translate to future gains in, say, labor market outcomes.
3. Measurement matters! Until recently, many CCT studies measured enrollment using measures reported by the student or his/her parents through household surveys. However, as has been discussed in this blog before, more recent studies are finding that using self-reported data may be causing impact findings to be biased. Our study joins these in finding that had we used self-reported data, the impact findings would have been very different – leading us to find no difference between the CCT and the UCTs arms in terms of enrollment. The lessons from the pattern of misreporting are interesting in and of themselves, and I encourage you to read them in "Cash or Condition" and, in more detail, in this paper forthcoming in a special issue of JDE on measurement.
4. It is fair to conclude that CCTs outperformed UCTs in terms of improvements in schooling outcomes, BUT…rates of marriage and pregnancy were substantially lower in the UCT than the CCT arm. This difference was due almost entirely to the effect of UCTs among girls who dropped out of school during the two-year program. This finding is interesting and puzzling at the same time, but not so once you’ve thought about it for a few minutes. I again invite you to read the relevant section in “Cash or Condition” linked above, where we provide a framework to assess the relative impacts of CCTs and UCTs on various outcomes. The fact is thoughtful researchers and policymakers have made this point before: households that do not comply with the condition(s) in a CCT are being left out of a program that is the main source of social protection support in many countries. If such household can be thought of as a vulnerable group in a given context, UCTs may deserve careful consideration given the possible trade-offs indicated in this study. The following excerpt from our paper summarizes this point much better than I can paraphrase:
By exploiting an experiment featuring a CCT and a UCT arm and by broadening the impact assessment beyond schooling, our study exposes a trade-off that is inherent in CCT programs. The existing literature is focused primarily on assessing the desired behavior change in CCT programs, and may have overlooked the effects of denying benefits to those who fail to satisfy the conditions. Our findings show that UCTs can improve important outcomes among such households even though they might be much less effective than CCTs in achieving the desired behavior change. The trade-off between improved schooling outcomes and delayed marriage and childbearing among school-age girls illuminates the importance of carefully considering what exactly transfer programs are trying to achieve in the target population. …This study makes clear that while CCT programs may be more effective than UCTs in obtaining the desired behavior change, they can also undermine the social protection dimension of cash transfer programs.
5. In CCTs, small transfers are as effective as larger ones: We found no gradient in schooling, marriage, or pregnancy with respect to the random variation in transfer size. This finding is consistent with previous evidence from other studies (such as this study by Filmer and Schady on Cambodia’s CCT program) and is good news for CCTs. As it seems that the condition is doing most of the work, if the main aim is to improve schooling outcomes in the target group, policymakers can use their limited budgets to treat more households with smaller transfer amounts rather than treating a smaller number of households with larger transfers. In the UCT arm, as one would expect, schooling and early marriage outcomes improved with increasing transfer sizes. But, as mentioned above, when it comes to schooling you’d have to pay more than $10 unconditionally to a household to get the same impact from a CCT of $5/month.
6. Transfers to children are NOT more effective than transfers to the parents: CCT programs can outperform UCTs, but if they are not addressing a market failure, such as intra-household bargaining problems, lack information about the value of schooling, or positive social externalities, the distortions they cause are inefficient. An example of such failures is "incomplete altruism" – i.e. a conflict of interest between the girls and their parents with respect to her education, which is sometimes mentioned as a justification of a preference for CCTs over UCTs. We find NO evidence that reallocating some of the transfers from the parents to the girls would improve program impacts on enrollment, achievement, early marriage, or pregnancy.
7. Cash transfers can substantially reduce risky sexual behavior and risk of HIV infection: Cash transfers not only caused a reduction in the number of lifetime sexual partners and unprotected sexual activity, but also led to a significant increase in partner’s safety. The following picture says it all by presenting the age of sexual partners among sexually active girls for the combined treatment arms vs. the control group: cash transfer practically eliminated the chance that girls have a male sexual partner 25 years or older, compared with approximately one out of every five girls in the control group. Biomarker data on HIV and HSV-2 support these findings. We are more or less able to rule out increased school enrollment as the reason behind the reduction in STIs and suggest that the positive income shock empowered young women to make safer choices.
8. Spillover effects are important to measure: If untreated girls in treated clusters are affected by treatment, we could be under or overestimating the program impacts on the population as a whole. The way to deal with this problem is to have treated and untreated clusters (across which there are no spillovers, i.e. the Stable Unit Treatment Value Assumption or SUTVA holds) and treated and untreated units within treated clusters (where SUTVA is expected to be violated). Then, comparing untreated individuals in treated areas to those in untreated clusters, one can get a sense of spillover effects. For example, in the context of our study, many people we talked to were worried that while the program may reduce the incidence of sexually transmitted diseases among girls receiving cash transfers, it could also increase it among untreated girls of the same cohort or among older women. In what is work in progress, our findings suggest that there are no such detrimental spillover effects with respect to early marriage, teen pregnancy, or sexual behavior. In fact, when we analyze spillovers among baseline friend networks rather than the entire village, we find protective effects of treatment for these outcomes: each treated friend increases the likelihood that an untreated girl in a treated cluster is more likely to be in school and less likely to be engaging risky sexual activity. These findings are encouraging. However, the real possibility that the existence of spillovers can change how we view program impacts is borne out when we examine mental health (working paper here) among the study population of adolescent girls. While the program led to large declines in psychological distress in the treatment group (larger in the UCT than the CCT arm), it also led to an equally large increase in psychological distress among girls living in untreated households in treatment clusters, leading to a finding of no effect at the community level in comparison to the control communities.
9. Experimental variation in key design parameters can also help shed light on channels of impact: The fact that we find small transfers are as effective larger ones in CCT programs is pertinent for policymakers trying to design similar programs. However, the study design that randomized the recipient of the transfer in the household, along with transfer size, can be useful to open the black box of program impacts. For example, I mentioned above that UCTs were more effective in improving girls’ mental health than CCTs, which, in turn, performed better than the control group. But why? One hypothesis supported by the data is that while the transfers are mainly given to the parents, it is the girl who has to comply with the condition to attend school. When the transfer amount is large enough to be an important (or the main) source of income for the family, this burden of being the de facto breadwinner may become too much for a school-age girl to shoulder. Our data show that the mental health of the girl deteriorates significantly with each additional dollar transferred to her parents conditional on her school attendance. We find no such gradient in the UCT arm or when the money is given to the girl in the CCT arm.
Your reactions to these points are welcome. In addition, preparing this presentation gave me the chance to think about the remaining question marks that I find interesting and/or puzzling. I will address these and future directions for cash transfer program evaluations in part II of this post -- with the benefit of listening to others speak and discuss at tomorrow's workshop -- hopefully later this week.
Join the Conversation