Les Roberts’s new study September 21, 2009

“Our data suggests that the (March 2003) shock-and-awe campaign was very careful, that a lot of the targets were genuine military targets.”
(Les Roberts, explaining Lancet 2006 findings)

Les Roberts’s new* study, ‘Media Coverage of Violent Deaths in Iraq‘, sort of attempts to evaluate the completeness of IBC’s database. The pre-publication references to the study, from Roberts himself (see note below) and his supporters, Medialens, were made in the context of bashing Iraq Body Count, and were incorrect – they stated that the study “found that the majority of violent deaths in a phone sample from Baghdad were not recorded by IBC”. In fact, the study finds that IBC may’ve captured 62% of the violent deaths (or rather that 38% are “absolutely not” in the IBC database). Given that Medialens regularly promoted Roberts’s old claim that IBC capture only 5-10% of deaths, it’s not surprising that they haven’t been shouting from the rooftops about Roberts’s new study.

The study interviewed mainly Baghdad residents (“Seventeen out of 18 primary interviewees resided in Baghdad, although some interviewees described deaths of neighbors that occurred while the neighbors were elsewhere”), and perhaps I was imagining things, but I thought it sounded pretty desperate when it continued with the following statement (after the 38%/62% coverage thing): “Evidence from past studies suggests that IBC reporting is far more complete, and perhaps five times more complete, in Baghdad than in the remainder of the country”. With this claim (which isn’t supported or referenced in the study, as far as I can see) it’s as if Roberts et al are saying: “Well, we’re evaluating what we regard as IBC at its best, covering Baghdad – just wait until you see IBC at its worst, even though we’re not showing you that here, and even though we’re not providing anything at all to substantiate this distinction we’re making”.

Another reason for Medialens (and co.) to stay quiet about the study is the type of “rigour” on display. Here’s an example:

The proportion of deaths [that we] matched in the IBC dataset seemed to increase for shorter recall periods: 22% of deaths were matched in the dataset in 2004, compared to 56% matched in 2007. It cannot be determined whether this is because of changing completeness on the part of IBC, because the interviewers gave less accurate information with longer recalls, or because of some other unseen factor.

Incorrect pre-publication descriptions from Roberts (et al) and Medialens

Roberts (and/or his JHU colleagues) wrote the following incorrect (as explained above) advance description of Roberts’s study (without identifying Roberts as a co-author) in ‘Answers to Questions About Iraq Mortality Surveys’, originally on the JHU website (no longer available, cached version here):

A review of Iraq deaths reported by 4 major U.S. newspapers found that IBC missed more than 1 of every 10 deaths reported by the news media. The separate and soon to be published study from Columbia University researchers also found that the majority of violent deaths in a phone sample from Baghdad were not recorded by IBC. (Siegler A. et al. has been accepted by the journal Prehospital and Disaster Medicine. It is slated for publication in the September 2008.)

Medialens reproduced the incorrect statement almost word for word in their “Alert” of January 22, 2008:

In fact there is no longer any excuse for this innocent reliance on Iraq Body Count (IBC). A review of Iraq deaths reported by four major US newspapers found that IBC missed more than one of every ten deaths reported by the news media. A separate study soon to be published by Columbia University found that the majority of violent deaths in a phone sample from Baghdad were not recorded by IBC.

Notice how Medialens dismiss “innocent reliance on Iraq Body Count” on the basis of an incorrect statement about the findings of a weak study which they hadn’t even seen. Incidentally, the first study mentioned in the above references (“separate” from the Columbia University study) appears to be part of the same study (co-authored by Roberts). There only appears to be one published study.

* It was actually published in 2008, but it’s probably “new” to most readers.