Reactions to the study
This section is currently being updated. We apologise for any broken links.
Summary: Scientific experts in epidemiology, public health, and statistics have noted nearly uniformly that the MIT/Bloomberg study used the standard methodology and robust implementation for estimating mortality in conflict, and that its conclusions therefore are likely to be the best available estimate. Some argue that the difficulties of implementing a survey in a warzone may have added to the uncertainty of the results.
The MIT/Bloomberg study was subjected to significant scrutiny before publication, with peer review by four independent experts. In addition, much of the media coverage has sought the opinion on scientific experts who have reviewed the study and its methodology. The assessment by scientific experts is nearly uniformly that the study used an accepted methodology and robust procedures, and that its findings and conclusions therefore are likely to be reliable.
The below is a compilation of statements by a range of different scientific experts. (Please let us know if we are missing any assessments, either positive or negative.)
Following the 2004 Bloomberg study, Lila Guterman wrote in the Columbia Journalism Review:
"I called about ten biostatisticians and mortality experts. Not one of them took issue with the study’s methods or its conclusions. If anything, the scientists told me, the authors had been cautious in their estimates."
In Australia, 27 of the country's leading scientists in epidemiology and public health signed a letter supporting the study, noting that it
"was undertaken by respected researchers assisted by one of the world's foremost biostatisticians. Its methodology is sound and its conclusions should be taken seriously. ... The study by Burnham and his colleagues provides the best estimate of mortality to date in Iraq that we have, or indeed are ever likely to have."
Paul Bolton, a public-health researcher at Boston University, said the methodology was "excellent" and standard procedure in a wide range of studies he has worked on. He also said: "You can't be sure of the exact number, but you can be quite sure that you are in the right ballpark". (See this FAQ for an explanation of the the uncertainty of the study)
Richard Brennan, Director of the Health Unit of the International Rescue Committee and a medical doctor with publications in on complex humanitarian emergencies, disaster management and clinical emergency medicine, said
"This is the most practical and appropriate methodology for sampling that we have in humanitarian conflict zones ... While the results of this survey may startle people, it's hard to argue with the methodology at this point."
The choice of method is anything but controversial .. I too find the survey's estimates shockingly high ... However, dismissing Burnham et al.'s work simply on gut feeling grounds seems more than irrational. A very similar methodology is routinely applied in many other settings for the same purpose."
"I loved when President Bush said 'their methodology has been pretty well discredited'. That's exactly wrong. There is no discrediting of this methodology. I don't think there's anyone who's been involved in mortality research who thinks there's a better way to do it in unsecured areas. I have never heard of any argument in this field that says there's a better way to do it."
"The investigators used a solid study design and rigorous, well-justified analysis of the data. They used several analytic techniques having different levels of assumptions to ensure the robustness of mortality estimates and the estimated margin of error. The researchers are also world-class."
"Given the conditions (in Iraq), it's actually quite a remarkable effort ... I can't imagine them doing much more in a much more rigorous fashion."
It is worth emphasising the quality of this latest report, as judged by four expert peers who provided detailed comments to editors."
"I think this is an extremely credible study"
"Over the last 25 years, this sort of methodology has been used more and more often, especially by relief agencies in times of emergency"
Sir Richard Peto. Professor of Medical Statistics at the University of Oxford, described the study as "statistically reliable". (BBC Newsnight, October 11, 2006) and Newsnight journalist Paul Mason said:
By far the most interesting part of the day for me was interviewing Sir Richard Peto - a statistician who had been critical of the first [John Hopkins University] report but said this one was statistically valid. He kept saying it in every answer he gave me: to me this took the story further than simply "one side's truth versus another's" - it made it credible. Note there is a difference between being credible and true.
Professor Sheila Bird, Principal Statistician at the Medical Research Council Biostatistics Unit, said on Channel 4 News, October 11, 2006:
"They have enhanced the precision this time around and it is the only scientifically based estimate that we have got where proper sampling has been done and where we get a proper measure of certainty about these results."
Donald Waldman, an epidemiologist at the Heilbrunn Department of Population and Family Health of Columbia University, said to the Washington Post that the MIT/Bloomberg survey method is "tried and true," and added that:
"this is the best estimate of mortality we have."
"The sampling is solid. The methodology is as good as it gets ... The value of the Johns Hopkins survey is that it was nationwide, in places out of the view where the media and most observers are."
Some other experts have sounded a more cautionary note:
Stephen Apfelroth disagrees that cluster sampling is an appropriate method to use to estimate crude mortality rates in emergencies:
"this technique would be adequate for rough estimates of variables expected to be fairly homogeneous within a geographic region, such as political opinion or even natural mortality, but it is wholly inadequate for variables (such as violent death) that can be expected to show extreme local variation within each geographic region."
Donald Berry, chairman of the statistics department at the University of Texas' M.D. Anderson Cancer Center in Houston, said he believes the study was done "in a reasonable way." But he said the range of uncertainty given for the estimates was much too narrow, because of potential statistical biases in the survey.
Debarati Guha-Sapir, Olivier Degomme, and Jon Pedersen noted in a letter to the Lancet that the study "fills an important information gap in a country where reliable mortality statistics are rare. It transforms anecdotes of violence into systematic evidence", but also had concerns about the implications of some of the findings:
"the study suggests that, over a 3-year period, around 90% of the deaths were directly related to violence. However, experience from other conflicts indicates that indirect causes (disease, malnutrition) typically outnumber the deaths due to violence (bombs, gunshots, etc)"
They also argued that "better accounting for differences in violence by governorate separately and the effect of excluding the Sunni triangle would have strengthened the study".
Prabhat Jha, Vendhan Gajalakshmi, Neeraj Dhingra, and Binu Jacob in a letter to the Lancet call the study a "a commendable study of mortality in Iraq in difficult circumstances" but raise the concern whether households over-reported deaths and whether the number of clusters is sufficient. They suggest that resampling by independent teams could have produced more reliable estimates. We discuss potential over-reporting in this FAQ and this FAQ, sample size and the number of clusters in this FAQ, and "resampling" in this FAQ.
Madelyn Hsiao-Rei Hicks, a psychiatrist at the Institute of Psychiatry, University of London, has raised the concern that the survey did not allow enough time for each interview, to the detriment of the reliability of the results:
"it is infeasible that “One team could typically complete a cluster of 40 households in 1 day”. Assuming continuous interviewing for 10 h despite 55°C heat, this allows 15 min per interview including walking between households and obtaining informed consent and death certificates. The improbability of so many interviews being done so quickly and reliance on “word of mouth among households” during selection and recruitment suggest potential sources of bias, ethical compromise, and risk to interviewees during interview-gathering.
See this FAQ for a more extensive discussion of this issue.
"high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much."
Dr Pedersen did not state on what basis this figure was derived, but further revised his assessment in comments to a correspondent, putting the figure at "perhaps ... 150,000" excess deaths. From the discussion, it appears his main concern is that he found the baseline crude mortality rate from which excess deaths are calculated low. (See this FAQ on the calculation of excess deaths.)
- the authors "do not report who issued these [death] certificates. Neither do they discuss why the availability of death certificates increased from 81% in 2004; and
- "the existence of a substantial reporting error is supported by the finding of low child mortality [compared to other estimates of preinvasion under-5 mortality] ... the results suggest that fewer than half of child deaths were reported".
They suggest that "Without an explanation for the high availability of death certificates, one could assume that the reporting error is of the same size as the sampling error (±30%). This assumption still yields at least a five-fold higher number of violent deaths than the passive surveillance mortality numbers". Please see more discussion of sampling error in this FAQ, death certificates in this FAQ, and child mortality in this FAQ.
Comments by security experts
Much of media reporting has relied on commentary by experts in fields not directly relevant to the study. For example, many media accounts quoted Anthony Cordesman, at the Center for Strategic and International Studies; who said said the estimates "[a]re almost certainly way too high,”. Similarly, the BBC quoted Michael O'Hanlon saying "I do not believe the new numbers. I think they're way off."
"the research method used ... is a respected tool for calculating fatalities in situations where complete records are unlikely to be available. The survey team in Iraq seems to have obtained considerable documentation to support their conclusions."
Cordesman, O'Hanlon, or Thompson are prominent experts on both the Iraq war and international relations generally, and as such have often been called upon to make comments on developments. However, they do not have expertise in in epidemiology, statistics, or public health. In the case of Cordesman and O'Hanlon, they have not made clear what their scientific criticism of the study may be, and their assessment is in stark contrast to the assessment of most scientific experts in biostatistics and epidemiology.
Reactions by politicians to the MIT/Bloomberg findings have been closely aligned with their stake in having advocated the invasion that sparked the increase in mortality.
U.S. President George Bush dismissed the methodology as "pretty well discredited".
UK Prime Minister Tony Blair's spokesman said
"The problem with this [study] is that they are using an extrapolation technique from a relatively small sample, from an area of Iraq which isn't representative of the country as a whole.
We have questioned that technique right from the beginning and we continue to do so.
The Lancet figure is an order of magnitude higher than any other figure; it is not one we believe to be anywhere near accurate."
UK Foreign Secretary Margaret Becket said:
"No-one disputes that there have been many deaths in Iraq and that all of those deaths are regretted and tragically many of them are deaths of civilians ... That doesn't mean that one has to accept every figure someone comes up with. "
Australian Prime Minister John Howard said:
"It's not plausible, it's not based on anything other than a house to house survey ... it is an unbelievably large number and it is out of whack with most of the other assessments that have been made"
These criticisms may be summarised as follows:
The methodology has been "pretty well discredited". This is, as one expert put it "exactly wrong": in fact, it is the standard accepted methodology for measuring mortality in conflict situations, as discussed in this FAQ. See this FAQ for the assessments of scientific experts of the methodology.
The survey uses an "extrapolation technique" and "is not based on "anything other than a house to house survey". This is the standard method to find out out about population characteristics and nearly all statistical data, whether government statistics, market research, or opinion polls are derived in this way. This is discussed in this FAQ. Moreover, such surveys are the standard methodology for measuring mortality in conflict situations and is known to give more complete and accurate figures than other methods, as discussed in this FAQ and this FAQ.
The sample size is not big enough to draw conclusions ("relatively small sample"). This is incorrect, the sample is large enough to conclude that the number of deaths is between 390 and 940 thousand, with 655 thousand the most likely number. This is discussed in this FAQ
The sample was drawn from a particularly violent area of Iraq. This is incorrect, the sample was drawn from all of Iraq, weighted by population, as explained in this FAQ
"The Lancet figure is an order of magnitude higher than any other figure". This is factually incorrect, as one UNDP study found a similar order of magnitude of deaths in 2004 (see this FAQ). It also is misleading, as other figures are derived from what is known to be incomplete reporting mechanisms that result in an under-count of mortality. The discrepancy is consistent with previous experience of measuring mortality in other conflict situations, as discussed in this FAQ.
The study was carried out by a team of researchers from the Bloomberg School of Public Health at John Hopkins University in collaboration with researchers from Al Mustansiriya University in Baghdad, and was funded by the Massachusetts Institute of Technology (we refer to it as the "2006 MIT/Bloomberg study").
The authors were are Gilbert Burnham, Riyadh Lafta, Shannon Doocy, and Les Roberts.
"Dr. Gilbert M. Burnham is the co-director of the Center for Refugee and Disaster Response at Johns Hopkins. He has extensive experience in emergency preparedness and response, particularly in humanitarian needs assessment, program planning, and evaluation that address the needs of vulnerable populations, and the development and implementation of training programs. He also has extensive experience in the development and evaluation of community-based health program planning and implementation, health information system development, management and analysis, and health system analysis. He has worked with numerous humanitarian and health development programs for multilateral and non-governmental organizations, regional health departments, ministries of health (national and district level), and communities in sub-Saharan Africa, Asia, and Eastern Europe. A major current activity is the reconstruction of health services in Afghanistan."
Department of Community Medicine, College of Medicine, Al-Mustansiriya University, Baghdad, Iraq
"Shannon Doocy is currently a Research Associate with the Johns Hopkins Center for Refugee and Disaster Response. Her research focuses on populations affected by disasters and conflict, including both refugees and internally displaced populations. Within the emergency context, her areas of interest include population based assessments, mortality, nutrition and food security, livelihoods and cash interventions, and monitoring and evaluation of health programs.
Recent research includes coping capacity and vulnerability Ethiopia; an effectiveness trial of a point-of-use water treatment product in camps for displaced populations in Liberia; the demographic and nutritional assessment of conflict-affected populations in Northeastern Sudan; tsunami-related research in Aceh Indonesia; assessment of food security in North Korea; assistance to projects in Sierra Leone, the West Bank, Nepal, Northern China, and Afghanistan."
"Les Roberts, PhD, holds has a Masters degree in public health from Tulane University and a Ph.D. in environmental engineering from Johns Hopkins. He did a post-doctorate fellowship in epidemiology at the Centers for Disease Control and Prevention where he worked for 4 years. In 1994, he worked as an epidemiologist for the World Health Organization in Rwanda during their civil war. He previously served as the Director of Health Policy at the International Rescue Committee. He is a lecturer at the Johns Hopkins University Department of Geography and Environmental Engineering where he teaches each fall. He teaches a class entitled Water and Sanitation in Complex Emergencies and the 7-week quantitative component of the Program’s Investigative Methods in Complex Emergencies course."
The Lancet article notes:
"The Massachusetts Instituve of Technology, which was the major funder, had no role in the collection or the analysis of the data. The John Hopkins Center for Refugee and Disaster Response used some general funds to cover research expense."