Thursday, July 19, 2012

2 August 2011 – Peer Review: The Bear Facts


Two of my interests - scientific methodological rigour in general, and climate science in particular - bumped uglies this week, in more than just the figurative sense.

For those of you who shelled out the requisite bucks a half-decade or so ago, or those who remember the 2007 Oscars, you might recall Al Gore’s venture into the art of documentary film-making, entitled An Inconvenient Truth.  Known throughout the climate blogosphere by the shorthand “AIT”, Gore’s flik purported to display conclusive evidence that human-produced greenhouse gases are imperilling life on this planet.

One of the pieces of evidence he trotted out in support of his argument - a superb example of photogenically-apt saccharine bathos if there ever was one - was a study suggesting that polar bears are being endangered by climate change because retreating Arctic ice means that they are having to swim further to find ice to hunt from.  The film was full of footage of the big, pretty, fluffy predators frolicking on the floes, gambolling merrily with their adorable offspring, and gamely swimming hither and yon…at least, until they drowned and were found floating and dead, allegedly (according to Gore) because even superb swimmers like polar bears eventually get tired.

The film footage touched hearts worldwide and sparked all manner of imitative imagery from AGW advocates and green groups, from articles accompanied by photoshopped images of a lonely polar bear in the flagship journal Science...

(Which the magazine itself captioned as follows: “This image is a photoshop design. Polar bear, ice floe, ocean and sky are real, they were just not together in the way they are now”*) a papier-mache polar bear riding a plastic iceberg in the Thames… polar bears hanging themselves from bridges (in what seems to be a bizarre bi-polar suicide pact with penguins, a startling display of inter-species solidarity given that penguins live roughly 20,000 km away in the antipodean Antarctic, where sea ice extent has been steadily increasing)... the amazingly ham-fisted advertisement by, who - in order to convince air travellers to air-travel less - executed a feat of propaganda not to be surpassed until 10:10 UK made a commercial showing school children being blown into bloody flinders by their teacher for asking skeptical questions about consensus climate science. produced a video showing polar bears plummeting from the skies and smashing themselves into ursine tartare on pavement and the odd car.

 “Oh, the bearmanity!”

And all because Al Gore swore up and down that polar bears were threatened by climate change. 

Incidentally, one wonders how these Planestupid folks think to get off their island, if not via some piece of fossil-fuel-burning man-made equipment.  These people put me in mind of what Lord St. Vincent had to say about Napoleon's invasion prospects in 1803: "I do not say that they cannot come. I only say that the cannot come by sea." Or by air, if the Planestupiders get their way. Or via the Chunnel.  Perhaps they intend to swim.  Hopefully the plastic polar bears won't jump off their plastic icebergs in the Thames and eat them.

Let’s ignore a few obvious facts - for example, the fact that polar bears as a species have survived at least three warmer interglaciations without going extinct; the fact that they seem somehow to be able to cope with an annual phenomenon known as “summer”, when the amount of Arctic sea ice shrinks by two-thirds; the fact that polar bear numbers are vastly higher than they were forty years ago, during which period the Earth has allegedly undergone “unprecedented warming”; and for that matter, the fact that if polar bears were really drowning in record numbers, we would expect to find a corpse now and again (references available on request) - and turn instead to the source of Al’s assertions.

The source was a paper by Dr. Charles Monnett, an Alaska-based researcher with the U.S. Bureau of Ocean Energy Management, Regulation and Enforcement.  In 2006, Monnett lead-authored an article in Polar Biology entitled “Observations of mortality associated with extended open-water swimming by polar bears in the Alaskan Beaufort Sea”.  The article was based on multi-decadal observations during aerial transect flights aimed principally at observing whale populations. From 1987-2003, hundreds of swimming polar bears were observed during these flights, and no dead ones were reported.  In 2004, however, four polar bear carcasses were seen floating offshore and were “presumed drowned” by the researchers.  The researchers first extrapolated this into a broader trend, and then interpreted it as proof that climate change was threatening polar bear populations (especially, according to the authors, lone females and cubs) because the bears have to swim longer distances to find ice to rest on.  Gore cited Monnett’s research in his film, launching the whole furry fandango.  The following year he collected his Oscar (and his Nobel Prize); and the year after that, the US Government classified polar bears as a “threatened species”.

If you’re interested, Monnett’s original article is here:

Fast-forward to 2011.  Monnett was recently suspended from his position and has been grilled extensively about his research.  The transcript of the inquiry makes interesting reading for a number of reasons, ranging from his methodology (using models to infer a species-wide threat from only four carcasses) to his startling inability to articulate what he was doing and how he was doing it. (the transcript of the interview with Monnett can be found here:

Apart from the difference between modelled (extrapolating from four corpses) and genuinely empirical (counting all the corpses) methodologies demonstrated in the paper, what I found especially interesting was the fact that Monnett’s paper - the paper that inspired Al, and that sparked the whole panic about the poor, doomed polar bears - seems to have been peer-reviewed by his wife (see line 26, page 35 of the transcript).  It was apparently reviewed elsewhere too, of course (including, presumably, by whomever the journal sent it to for review), but the fact that the internal review for the first version of the report seems to have been by Monnett’s spouse, a co-worker who shares his views and research funding, certainly sets off some alarm bells.  If nothing else, it gives the appearance of a conflict of interest - something that presumably any research organization would want to avoid.  One of the key complaints about the climate science community in recent years has been its insularity and propensity for gentle “pal reviews” rather than robust “peer reviews”.

It’s also worth noting that it doesn’t seem clear whether the questions posed by secondary reviewers were ever addressed.  At one point in the interview transcript (see page 41), for example, the investigator cites a reviewer’s question about Monnett’s numbers and statistics and asks whether Monnett ever addressed the reviewer’s concern, and Monett replies as follows:

“I would assume since they signed off on it, that they were satisfied with whatever answers they got”.

It’s an interesting argument, because there is no indication that the reviewer in question was ever asked whether he was satisfied with Monnett’s answers.  Monnett then cites “management review” in his defence, arguing that the paper must have been okay because his manager signed off on it.  It’s an amusing fallback position, since Monnett argues, near the end of the interview, that his managers have been trying to kill his study “ever since the polar bear thing came out” due to the sensitivity and policy implications of the issue; “management”, according to Monnett, doesn’t want anything to interfere with oil drilling in Alaska.  "God forbid" he publish, he says, “something that has anything to do with the climate change debate.”
The transcript of Monnett’s interview is as painful to read as any transcript where the interviewers don’t understand what the interviewee is talking about, and the interviewee isn’t particularly good at explaining it.  But a couple of interesting points emerge.  One is that peer review as a mechanism for ensuring scientific rigour depends for its utility on a combination of expertise and objectivity.  While it is possible for any adequately experienced scientist to offer a critique of the methodology in a paper, one can’t provide an adequate review of the substance of a paper unless one is at least as expert in the subject as the author (this is what the word “peer” means).  For this reason, most peer reviews of complex and detailed arguments tend necessarily to be about methodology more than substance.  Objectivity is equally important, and consists of two parts: perceived disinterest, i.e., that the reviewer has no stake in the results of the study he or she is reviewing (and thus has no incentive either to support the author’s conclusions, or to challenge them); and actual disinterest, i.e., that the reviewer is able to step back from personal or professional involvement with the author and/or his arguments, and provide a qualitatively impartial analysis of the strengths and weaknesses of the arguments presented in the paper.  These are of course in addition to the basic elements of scientific professionalism, e.g., refraining from ad hominem observations and other fallacies of logic while reviewing another scientist’s work, providing evidence when challenging data or conclusions, etc.
For all of these reasons, even in cases where the reviewee and the reviewer possess roughly comparable levels of expertise, it is inadvisable to have papers reviewed by members of the same organization as the author.  It is especially inadvisable - as the Monnett case demonstrates - to assign as a reviewer for a new and potentially controversial project someone who (a) has a demonstrable personal, pecuniary or bureaucratic interest in either supporting or undermining the paper’s conclusions; or (b) someone who, for whatever reason, is unlikely to be able to take an objective, unbiased stance vis-à-vis the subject of the paper under review.  We are none of us saints, and in the interest of preserving the integrity of the process, it is always best to avoid even the possibility of the appearance of bias.  Scientific rigour is a little like Caesar’s wife in that regard; it must not only be pure, but also be seen to be pure.
There are a couple of interesting codas to this story.  One is that Al Gore’s polar bear claim was cited in 2007 by a UK High Court as one of eleven demonstrably disprovable inaccuracies in An Inconvenient Truth; the Court noted that Monnett’s paper stated that only four carcasses had been observed, and that the deaths of these bears was attributable to “a particularly violent storm” rather than “climate change” (or for that matter, to falling out of the sky onto the M25).  Another is that Arctic sea ice extent was the same in 2006 as it was in 2004, when the dead bears were observed, and was actually lower in 2005, 2007, 2008, 2009 and 2010 than it was in 2004 - and yet, as with the 20 years preceding Monnett’s 2004 observations, no floating, ”presumably drowned” bears were observed in any subsequent years.  
Finally, according to the CBC, two polar bears did in fact drown in 2007.  However, they drowned because they slid off the ice after being shot with tranquilizer darts by researchers from Nunavut’s Environment Department.  While expressing regret over the deaths, Steve Pinksen, the department’s Director of Policy, defended the value of polar bear research, and added that gaining important scientific data always has costs - “including the odd dead bear.”  He added that these were only the third and fourth deaths in 25 years of research. 
So in other words, the number of dead bears that Al Gore used to spark worldwide panic over the future of the species (and which led directly to the whole string of bizarre imagery and mind-bogglingly insane public relations campaigns noted above) - four in a twenty-five year period, in other words - is precisely the same as the number of bears that the Nunavut Environment Department accidentally killed in a twenty-five year period as the regrettable but necessary cost of doing scientific business.
Anyway, it seems that while much of the interview transcript concerned Monnett’s (in)famous polar bear paper, his suspension seems to be related to an entirely different issue.  According to last Friday’s Sacramento Bee, Monnett was suspended over “integrity issues”, possibly in conjunction with his acting as a contracting officer for some $50M in research funds that he’s been responsible for administering for the last few years.

Even if we can’t support Monnett on his research methods, his results, or how he deals with adverse comments from peer reviews, I’m sure we can all sympathize with a fellow scientist finding himself driven to distraction by the ineffable joys of contracting.



P.S. Here's a picture of Elvis Presley playing the guitar next to the Venus de Milo atop a steam locomotive on the Moon.
*In the spirit of Science magazine, I should probably warn you that this image is a photoshoppe design. Steam locomotives, the Venus de Milo, Elvis Presley, and the Moon are all real; they were just “not together in the way they are now”.