Numbers That Go Crunch

Daniel Davies, better known to bloggers as Daniel of Crooked Timber, explains the Johns Hopkins/Lancet study on deaths in Iraq.

First, don’t concentrate on the number 600,000 (or 655,000, depending on where you read). This is a point estimate of the number of excess Iraqi deaths – it’s basically equal to the change in the death rate since the invasion, multiplied by the population of Iraq, multiplied by three-and-a-quarter years. Point estimates are almost never the important results of statistical studies and I wish the statistics profession would stop printing them as headlines.

The question that this study was set up to answer was: as a result of the invasion, have things got better or worse in Iraq? And if they have got worse, have they got a little bit worse or a lot worse. Point estimates are only interesting in so far as they demonstrate or dramatise the answer to this question.

The results speak for themselves. There was a sample of 12,801 individuals in 1,849 households, in 47 geographical locations. That is a big sample, not a small one. The opinion polls from Mori and such which measure political support use a sample size of about 2,000 individuals, and they have a margin of error of +/- 3%. If Margaret Beckett looks at the Labour party’s rating in the polls, she presumably considers this to be reasonably reliable, so she should not contribute to public ignorance by allowing her department to disparage “small samples extrapolated to the whole country”. The Iraq Body Count website and the Iraqi government statistics are not better measures than the survey results, because one of the things we know about war zones is that casualties are under-reported, usually by a factor of more than five.

And the results were shocking. In the 18 months before the invasion, the sample reported 82 deaths, two of them from violence. In the 39 months since the invasion, the sample households had seen 547 deaths, 300 of them from violence. The death rate expressed as deaths per 1,000 per year had gone up from 5.5 to 13.3.

Just as a number on a chart, a mortality rate of 13.3 is not self-evidently absurd. A great many third-world nations have mortality rates at least that high, if not higher. (You can find a handy-dandy table of world demographics in this PDF file. I believe Mozambique wins the mortality prize at 20 deaths per 1,000.)

Talk of confidence intervals becomes frankly irrelevant at this point. If you want to pick a figure for the precise number of excess deaths, then (1.33% – 0.55%) x 26,000,000 x 3.25 = 659,000 is as good as any, multiplying out the difference between the death rates by the population of Iraq and the time since the invasion. But we’re interested in the qualitative conclusion here.

That qualitative conclusion is this: things have got worse, and they have got a lot worse, not a little bit worse. Whatever detailed criticisms one might make of the methodology of the study (and I have searched assiduously for the last two years, with the assistance of a lot of partisans of the Iraq war who have tried to pick holes in the study, and not found any), the numbers are too big. If you go out and ask 12,000 people whether a family member has died and get reports of 300 deaths from violence, then that is not consistent with there being only 60,000 deaths from violence in a country of 26 million. It is not even nearly consistent.

Most of the criticism coming from the Right Blogosphere amounts to “I don’t believe it,” albeit expressed in more colorful language and accompanied by ad hominem attacks on the researchers. And all of the criticisms of the study that I’ve seen pick apart the point estimate number but do not seriously address the methodology or the increase in mortality rate.

Some of the comparisons are downright weird. For example, some guy quoted on Instapundit said “It is a larger number than were killed in Germany during five years (and 955,044 tons) of WWII bombing.” I assume that’s true, but that’s not an honest comparison. An honest comparison would compare pre-war mortality rates to mortality rates of German citizens during World War II — deaths from everything, including mumps, traffic accidents, and the Holocaust — and extrapolate from that the number of people who would not have died had the mortality rate not risen.

The “WWII bombing” comparison just tells me that whoever came up with it does not grasp what the Johns Hopkins/Lancet study even is about, never mind understand the methodology. Yet he declares himself an expert and says it’s wrong.

Daniel Davies continues,

A particularly disgusting theme of some right-wing American critics of the study as been to impugn it by talking about it being “conveniently” released before the November congressional elections. As if a war that doubled the death rate in Iraq was not the sort of thing that ought to be a political issue. Nobody is doing anything about this disaster, and nobody will do until people start suffering some kind of consequences for their actions (for example, no British politician, soldier or spy has lost his job over the handling of the Iraq war and no senior member of the Bush administration either).

There has to be some accountability here. It is not good enough for the pro-intervention community to shrug their shoulders and say that the fatalities caused by the insurgents are not our fault and not part of the moral calculus. I would surely like to see the insurgents in the ICC on war crimes charges, but the Nuremberg convention was also correct to say that aggression was “the supreme international crime differing only from other war crimes in that it contains within itself the accumulated evil of the whole”. The people who started this war of aggression need to face up to the fact, and that is a political issue.

Richard Horton writes in The Guardian

Not only do we have a better understanding of the toll our invasion has had on the country; we also understand better just how those deaths have come about. Before the invasion only a tiny proportion of deaths were due to violence. But since the invasion over half of all deaths have been due to violent causes. It is our occupation and our continued presence in Iraq that is fuelling this violence. Claims that the terrorist threat was always there are simply disproved by these findings.

The nature of these causes has changed too. Early on in the post-invasion period deaths were made worse by aerial bombing. But now gunshot wounds and car bombs are having a far greater effect. Far from our presence in Iraq stabilising the chaos or alleviating the rate at which casualties are mounting, we seem to be making the situation worse. In each year since the invasion, the mortality rates due to violence have increased.

In each year since the invasion, the mortality rates due to violence have increased.
That’s the important point, and that’s what the Bush Administration and its rightie supporters need to explain. And answer for.

6 thoughts on “Numbers That Go Crunch

  1. This is an excellent clarification of the study. Not that the wingnuts will care.

    One point about the mortality rate: this is in a country with a median age of 19 or so. In other words, people aren’t dying of old age. Somebody at the Corner said, in effect, ‘that’s not so much higher than the US (8 or so/1000) or the UK (10 or so/1000)’–completely ignoring the fact that we have a median age of 36 and the UK has a median age of 39.

  2. I’m so friggin’ glad the Iraqis are willing to tolerate this level of violence and death to be free….
    because we sure wouldn’t
    The things Mr. Bush says are remarkable, I hope someone is jotting them down.. As evidence for his trial…..

  3. Questions for rightwingers:

    And when exactly should vital information for election decisions be released? Should it be held off until after the elections, as the New York Times, CNN, and CBS did with several different stories in 2004, stories they held off because they felt they might be damaging to Bush? Do the Republicans have a right to run for office without pre-election news stories about their actions and the consequences of those actions? Is ignorance really bliss, and if so, why are rightwingers so continually angry?

  4. And all of the criticisms of the study that I’ve seen pick apart the point estimate number but do not seriously address the methodology or the increase in mortality rate.

    Multiple credible studies, using multiple methodoligies, have been done concerning the body-count of Iraq. All of them lived up to acceptable standards of practice, and all of them got “different” results. However, there is NOT A SINGLE OTHER STUDY that arrived at a death toll of even half that forecasted by the L:ancet study. The Lancet result is an extreme outliermany standard deviations away from the mean.

    Could Lancet be right? Yes. However, it would mean that every other poll suffered from some as-yet-unrecognised experimental bias that acted unformly across different methodologies. The other choice that the Lancet study is the one that suffers from an experimental bias that will be understood only with the benefit of hindsight. If this were not political there would be no debate as to which hypothosis is more credible in the absence of further evidence.

  5. r4d20 — Most of the other civilian death estimates are not statistical samples, but instead are just simple tallies of reported deaths by violence. If you’ve been following the discussion it ought to be obvious why such tallies would undercount the deaths extrapolated by the sample.

    Strictly speaking, I don’t believe it’s accurate to call the Lancet results an “outlier” if you aren’t comparing the results to other statistical samples. It can’t be an outlier if it’s the only statistical sample available.

    I’ve heard that the UN did some statistical sampling of mortality in the first year after the invasion, but they were using different criteria from Lancet and it’s not clear to me if the two samplings were measuring exactly the same thing.

  6. Pingback: The Mahablog » More Crunching

Comments are closed.