There are a number of things we would like to measure for which a direct question may be refused or met with an inaccurate answer. Three new papers demonstrate some of the methods that can be used to help overcome these problems.
Paper 1: List randomization for measuring illegal migration
The first is a paper I wrote with Melissa Siegel which looks at how to measure illegal migration . We explore the use of the list randomization approach. This works by dividing the sample randomly into two groups. The first group (Group A) gets asked “how many of the following three statements do you regard as true?”
1. At least one member of my household plans on opening a new business in the next five years
2. The economic situation of my household has improved considerably over the past five years
3. Corruption in my country is a less serious problem than ten years ago
The second, group B, gets asked “can you tell me how many of the following four statements you regard as true?”:
1. At least one member of my household plans on opening a new business in the next five years
2. The economic situation of my household has improved considerably over the past five years
3. Corruption in my country is a less serious problem than ten years ago
4. This household has at least a member currently residing abroad without a legal residence permit
The difference mean number of responses for Group B – mean number for Group A is then a measure of the proportion of the sample with an illegal migrant. In the paper we test this approach in four countries (Ethiopia, Mexico, Morocco, and the Philippines) and show that it seems to give useful information - rankings of illegal migration rates conditional on migration are consistent with other estimates of illegal migration; and that in two countries where it was not considered too sensitive to also ask directly about illegal migration we obtain rates which are reasonably close to our list randomized estimates. However, the paper also discusses some limitations of the approach, such as wide confidence intervals unless samples are very large, and potential concerns about the extent to which individuals understand how this method protects their privacy.
Paper 2: Endorsement experiments to understand support for the Taliban & Foreign Forces in Afghanistan
A second paper by Jason Lyall, Graeme Blair and Kosuke Imai looks at the question of how civilian attitudes toward combatants are affected by wartime victimization. Directly asking someone “do you support the Taliban?” or “do you support foreign forces?” may be unlikely to get a truthful response. So instead they use endorsement experiments. The basic idea is to again divide the population randomly into groups. Then one group gets asked about their opinion towards a policy endorsed by a specific actor whose support they want to measure, while the other group gets asked about their opinion towards this policy without mentioning the endorsement. For example:
Paper 3: Reticence-adjusted measures of Corruption
Finally, in a third recent paper, Aart Kraay and Peter Murrell look at measuring corruption in an enterprise survey in Peru, and through inclusion in Gallup World Poll questions fielded in 12 Asian countries. They use the random-response method: individuals are given a series of seven sensitive questions. Respondents privately toss a coin before answering each question and are instructed to answer "Yes" if the coin comes up heads. If the coin comes up tails, they are instructed to answer the sensitive question truthfully. However, they note that in Peru, 23.6% of the sample answer no to all of the 7 questions – whereas the probability of observing no "Yes" responses over seven coin tosses should only be 0.008 if respondents were correctly following the protocol of the question. They call these individuals “reticent”, and then develop a model which corrects for this tendency. Their reticence-adjusted estimates of corruption in Peru are twice as high as standard estimates (37% versus 18%), and similarly they also see much higher estimates of corruption in the Gallup World Poll after they adjust for reticent behavior. Moreover, corruption estimates are even higher in models that allow for a positive correlation between reticence and guilt.
Taken together these three papers illustrate three methods that can be used to try and measure concepts that people might be reluctant to directly report – although as the third paper shows, even methods which are set up to hide individual responses enough to hopefully elicit the truth may still not be enough to overcome people’s concerns (or maybe they might not understand that their individual responses are not identified from such data), so the statistical methods they develop to try to get at how much of a problem this is are welcome.
Paper 1: List randomization for measuring illegal migration
The first is a paper I wrote with Melissa Siegel which looks at how to measure illegal migration . We explore the use of the list randomization approach. This works by dividing the sample randomly into two groups. The first group (Group A) gets asked “how many of the following three statements do you regard as true?”
1. At least one member of my household plans on opening a new business in the next five years
2. The economic situation of my household has improved considerably over the past five years
3. Corruption in my country is a less serious problem than ten years ago
The second, group B, gets asked “can you tell me how many of the following four statements you regard as true?”:
1. At least one member of my household plans on opening a new business in the next five years
2. The economic situation of my household has improved considerably over the past five years
3. Corruption in my country is a less serious problem than ten years ago
4. This household has at least a member currently residing abroad without a legal residence permit
The difference mean number of responses for Group B – mean number for Group A is then a measure of the proportion of the sample with an illegal migrant. In the paper we test this approach in four countries (Ethiopia, Mexico, Morocco, and the Philippines) and show that it seems to give useful information - rankings of illegal migration rates conditional on migration are consistent with other estimates of illegal migration; and that in two countries where it was not considered too sensitive to also ask directly about illegal migration we obtain rates which are reasonably close to our list randomized estimates. However, the paper also discusses some limitations of the approach, such as wide confidence intervals unless samples are very large, and potential concerns about the extent to which individuals understand how this method protects their privacy.
Paper 2: Endorsement experiments to understand support for the Taliban & Foreign Forces in Afghanistan
A second paper by Jason Lyall, Graeme Blair and Kosuke Imai looks at the question of how civilian attitudes toward combatants are affected by wartime victimization. Directly asking someone “do you support the Taliban?” or “do you support foreign forces?” may be unlikely to get a truthful response. So instead they use endorsement experiments. The basic idea is to again divide the population randomly into groups. Then one group gets asked about their opinion towards a policy endorsed by a specific actor whose support they want to measure, while the other group gets asked about their opinion towards this policy without mentioning the endorsement. For example:
- CONTROL CONDITION: A recent proposal calls for the sweeping reform of the Afghan prison system, including the construction of new prisons in every district to help alleviate overcrowding in existing facilities. Though expensive, new programs for inmates would also be offered, and new judges and prosecutors would be trained. How do you feel about this proposal?
- TREATMENT CONDITIONS: A recent proposal by the Taliban [or foreign forces] calls for the sweeping reform of the Afghan prison system, including the construction of new prisons in every district to help alleviate overcrowding in existing facilities. Though expensive, new programs for inmates would also be offered, and new judges and prosecutors would be trained. How do you feel about this proposal?
Paper 3: Reticence-adjusted measures of Corruption
Finally, in a third recent paper, Aart Kraay and Peter Murrell look at measuring corruption in an enterprise survey in Peru, and through inclusion in Gallup World Poll questions fielded in 12 Asian countries. They use the random-response method: individuals are given a series of seven sensitive questions. Respondents privately toss a coin before answering each question and are instructed to answer "Yes" if the coin comes up heads. If the coin comes up tails, they are instructed to answer the sensitive question truthfully. However, they note that in Peru, 23.6% of the sample answer no to all of the 7 questions – whereas the probability of observing no "Yes" responses over seven coin tosses should only be 0.008 if respondents were correctly following the protocol of the question. They call these individuals “reticent”, and then develop a model which corrects for this tendency. Their reticence-adjusted estimates of corruption in Peru are twice as high as standard estimates (37% versus 18%), and similarly they also see much higher estimates of corruption in the Gallup World Poll after they adjust for reticent behavior. Moreover, corruption estimates are even higher in models that allow for a positive correlation between reticence and guilt.
Taken together these three papers illustrate three methods that can be used to try and measure concepts that people might be reluctant to directly report – although as the third paper shows, even methods which are set up to hide individual responses enough to hopefully elicit the truth may still not be enough to overcome people’s concerns (or maybe they might not understand that their individual responses are not identified from such data), so the statistical methods they develop to try to get at how much of a problem this is are welcome.
Join the Conversation