Random lotteries to allocate scarce slots for an oversubscribed program provide a useful tool for estimating impacts of such a program. However, an issue which can arise in practice is that there may be multiple lotteries that an individual can apply for. For example,
· Abdulkadiroglu and co-authors (QJE paper, ungated version) study the impact of charter school attendance on student attainment in Boston. Students decide which schools to apply for, with independent lotteries – so a student that applies to multiple schools has more chance of being selected for a charter school than a student who applies only to one school.
· In forthcoming work with John Gibson and Steve Stillman, I look at the impact of migrating from Samoa to New Zealand on household members left behind. There is one lottery each year, and so once we consider outcomes several years later, households could have potentially applied to the lottery more than one time.
· This issue is coming up in a number of Government projects that I have recently been discussing evaluations for. For example, in a matching grant program, there is a call for proposals each six months, with lotteries planned to allocate scarce funds among eligible applicants each round. However, firms which are not selected in one round are free to choose whether to try again in future rounds.
If individuals or firms self-select into how many lotteries to apply for, then the characteristics of lottery winners may differ from those of lottery losers, since those who apply more times are more likely to win. Luckily, Abdulkadiroglu note there is an easy fix for this concern when it comes to estimation. They define an applicant’s risk set as the list of all lotteries to which the individual applied. One then adds dummy variables for each lottery entered as controls in the estimation.
To make this concrete, suppose there are 3 oversubscribed schools a student can apply to, and let dummy variables d1, d2, and d3 indicate whether the student applied to school 1, 2 and 3 respectively. Then to look at the impact of being in a charter school on a learning outcome, one estimates:
Learning outcome = a + b*Charter School + c1*d1+c2*d2+c3*d3 + e
By 2SLS, instrumenting for Charter School with variable Z, indicating whether an individual won a lottery admitting them to a charter school, using a first stage like:
Charter School = f + g*Z + h1*d1 + h2*d2 + h3*d3 + v
The odds of winning, and hence value of Z, will depend on which lotteries an individual entered, but conditional on which lotteries are entered, it is random whether or not they win. The same basic approach can be used in a panel setting.
The key to implementing this fix is to know exactly which lotteries an individual entered. This wasn’t the case in our Samoa work, where we did not have much confidence in the ability of remaining family members to know exactly in which years the household’s migrant member had entered his or her name unsuccessfully before their name was eventually drawn. We therefore have to use a combination of administrative data and data from the lottery loser sample to show robustness to this issue. Luckily this worked out okay in our case, but this is something that people should pay attention to in designing their data recording systems going forward – without thinking clearly about this ex ante, I can easily imagine surveys or administrative data merely recording whether or not applicants won a lottery or not, and perhaps the number of lotteries they entered, rather than yes/no records of entry into each lottery on offer.
Join the Conversation