IPCC Audit part 2: the importance of peer review

We are told that the science has been settled on the basis of “an immense edifice of painstaking studies published in the world’s leading peer-reviewed journals…vetted and documented in excruciating detail…” And a climate modeler claims that the input to the reports “has been scrutinized to the highest level possible”  (p 33). [Tell that to Steve McIntyre, in an episode to be described in another post]. And “A core principle of the IPCC is that only peer-reviewed literature is cited” (p 41).

The Chairman himself made that claim in a speech to the legislators of North Carolina.

Leaving aside what we have learned about the corruption of the peer-review process, claims about the exclusive use of peer-reviewed literature are bogus. In a chapter on “the peer review fairy tale” Laframboise described a collaborative project involving a worldwide team of helpers who checked all the cited references in the 44 chapters of the 2007 report, counting how many were peer-reviewed and how many came from the “gray” literature.

Her suspicions were aroused by reports from IPCC expert reviewers (not insiders to the writing) that some items were being submitted which did not have scientific status. These even included some press releases, however their concerns were dismissed and the reports were listed as input to the final report (p46).

The final score for 18,531 references in the 2007 report was 5,587 (one third) not peer reviewed. In 21 of the 44 chapters the score for peer reviewed references did not reach 60%. This would not be so bad if it was admitted up front and in public, also if there were clearly defined and properly policed rules for vetting the grey matter (not peer-reviewed) for use by the inner circle of authors.

Among the sources used to support IPCC recommendations were newspapers and magazine articles, unpublished theses, Greenpeace and World Wildlife Fund documents, and yes, press releases.

While Chairman Pachauri had declared an Indian government discussion paper fit only for the rubbish bin, we found that the Climate Bible cites dozens of discussion papers. In one case the decument relied on by the IPCC was clearly labelled as ‘version one’ of a draft. (p 48)

After the release the Chairman wrote a piece referring to “approximately 18,000 peer-reviewed publications”, with the concession that “a limited amount of gray literature” was used “in cases were peer-reviewed literature was unavailable”. One third is a limited amount? And what happened to planning and consultation with the profession? How did all the thousands of climate scientists in the world leave the IPCC short of peer-reviewed material for vital parts of the report? 

This discovery suggests that the great mass of journalists and environmental commentators have been irresponsible and unprofessional in failing to report on this situation so the myth of universal peer-review remains alive.

The rules for using “gray” literature

Apart from the “rule” that limited quantities of non peer-reviewed material are used in special circumstances, there was a rule that such items have to be clearly identified in the list of references. One of the helpers found that a total of 6 (six) of almost 6,000 gray items were flagged.

Some people close to the action suggested that more efforts should be made, perhaps using a different colour to flag gray material. After some internal discussion there was a policy statement that peer-reviewed material should have priority and all mention of flagging gray matter has vanished (p. 53).

Relevant stuff on Quadrant on Line.

This entry was posted in Uncategorized. Bookmark the permalink.

25 Responses to IPCC Audit part 2: the importance of peer review

  1. JamesK

    And even the ‘peer-review’ process in rarefied subspecialities of science is questionable.

    Wegman in in his 2006 Review of the hockeystick controversy for Congress and his subsequent congressional questioning was deeply suspicious of the the ‘peer-review’ process itself in paleoclimatology

    Climategate proved those suspicions.

  2. Rafe

    That shows the way the credibility of science and its institutions, protocols and practices have taken a huge hit as a result of the fuss over climate science.

    Terence Kealey’s work is important in relation to this, the implication of his work and the current politicization (and corruption) of science is that the best hope for the future of science is for the government to draw back and permit significant privatization of science.

  3. jupes

    And even the ‘peer-review’ process in rarefied subspecialities of science is questionable.

    Yes it is. Indeed, the UEA scientists themselves admit it:

    I can’t see either of these papers being in the next IPCC report. Kevin [Trenberth] and I will keep them out somehow – even if we have to redefine what the peer-review literature is !

  4. TerjeP

    The IPCC website explains the purpose of the organisation.

    http://www.ipcc.ch/organization/organization.shtml

    It was established by the United Nations Environment Programme (UNEP) and the World Meteorological Organization (WMO) to provide the world with a clear scientific view on the current state of knowledge in climate change and its potential environmental and socio-economic impacts. The UN General Assembly endorsed the action by WMO and UNEP in jointly establishing the IPCC.

    I suppose that for many people, especially those affiliated with the UN, it seemed natural that a single, taxpayer funded, centralised organisation was the logical way to achieve these ends. Certainly it would be less messy than a free for all competition of ideas.

  5. Rafe

    What do you mean “less messy”? No unseemly diversity of opinion?

  6. Lazlo

    Certainly it would be less messy than a free for all competition of ideas.

    Umm that is called science. You reflect the true totalitarian agenda of the left and the UN.

  7. TerjeP

    Rafe – it is called damning with faint praise.

  8. So what did the IPCC get right then? Hint: the mean global temperature is increasing as is the density of greenhouse gases. The IPCC reports are supposed to be peer reviewed and reviewed by the relevant government agencies. The CSIRO, it would seem, generally follows IPCC methodology, yet considers other approaches, at least in this instance without concluding that one is superior to the other.

  9. Boris

    As someone deeply involved in peer review as author, reviewer and editor, I think peer review is like democracy. It is a very bad system but everything else is much worse.

  10. Rafe

    Yes. I have had trouble with reviewers when it became apparent that I knew far more about the issue than they did, so some of their objections were inane. Still it is a matter of improvement rather than throwing it out.

    I am currently reviewing a paper where the author has done a huge amount of work that misses the point, he or she should have been protected from this by colleagues.

    The interesting/scandalous thing about peer review in the IPCC which I will have to say more about is the way it violates the norms of peer review as generally understood – for example it is NOT done blind!

  11. jupes

    Hint: the mean global temperature is increasing as is the density of greenhouse gases

    Crap.

    The mean global temperature hasn’t increased for over a decade despite the increase in man made CO2.

    The AGW hypothesis has been debunked.

  12. Viva

    That shows the way the credibility of science and its institutions, protocols and practices have taken a huge hit as a result of the fuss over climate science.

    Absolutely. This morning I watched a group of Nobel laureates being interviewed on BBC World being very intersted to hear their views – until one physicist starting solemnly describing the how a consensus view was reached in science – at which point I chortled and switched off.

  13. Winston Smith

    Viva, the best technological advance this decade will be a TV that, when you turn it off, can be heard back in the studio.

  14. Jupes:

    I think this is the really important issue, and so the evidence one way or another is very important. The AGW hypothesis may have been debunked but that is different from proving it incorrect.

    Now if you look at the mean temperature records there is a period roughly from the late 1940’s to the late 1960’s (equivalent to the period of nuclear testing) during which time temperatures are not rising, rather falling. Presumably during this period CO2 and other gases are being produced.This is a longer period than the last years, where just by eye-balling the data the trend is rising. The climate system is too diverse to operate in a strictly monotonic or linear way. For example consider the CSIRO example relating to aglime.

    So what is the explanation for the failure of peer review in the IPCC reports? Is there another explanation other than some conspiracy or lack of due diligence?

  15. jupes

    So what is the explanation for the failure of peer review in the IPCC reports?

    At the risk of sounding like a broken record:

    I can’t see either of these papers being in the next IPCC report. Kevin [Trenberth] and I will keep them out somehow – even if we have to redefine what the peer-review literature is !

    Is there another explanation other than some conspiracy or lack of due diligence?

    No

  16. Boris

    I would also add that even the best peer-review system is neither a necessary nor a sufficient condition for quality. Most of peer-reviewed publicaitons are crap. Sounds strange? well, the proportion of crap is still much lower than in non-refereed publications.

    Why is that. As an editor in chief of an international journal once told me: ‘if I applied my gold standard (the one I used as a reviewer), the journal would be 10% of the present size’. The implication was that he would be removed from his (unpaid) position.

    Basically, the current ‘publish or perish’ situation creates perverse insentives to publish every little thing you have.

    peer-review works as a very loose and subjective filter. A useful filter nonetheless.

  17. Rafe

    The best you can hope from the system is that a few totally useless papers get rejected and some others are improved. But you can’t apply the gold standard because the publish or perish game demands huge numbers of papers and in the nature of things most to not add value. Most are read by about 1.5 people.

  18. Boris

    Rafe, I agree. I’d say that I do not want that editor’s gold standard either because this is still subjective and I do not want a baby to be thrown away with the bathwater. Better to accept 10 crap papers than get a good one rejected. Not sure how to find the right balance.

  19. Rafe

    In a situation where good papers like mine get held up and even rejected you would rather allow a heap of bad papers for the sake of some good ones that would be rejected under the gold standard because they do not fit the paradigm of the gatekeepers at the moment.

  20. AndrewL

    Jupes, those papers were in the IPCC AR4 even though they have proved to not add much to the field. Do you feel they were particularly groundbreaking?

  21. dover_beach

    From the Soon and Baliunas et al paper that was a cause of much derision back in 2003:

    Across the world, many records reveal that the 20th century is probably not the warmest nor a uniquely extreme climatic period of the last millennium.

    \

    However, in an email, Wigley wrote:

    June 5, 2003 (email 0682). By chance SB03 may have got some of these precip [sic] things right, but we don’t want to give them any way to claim credit.

    Eight years later we find the IPCC AR5 Zero Draft summarize the present evidence on precipitation extremes over the past millennium as follows:

    Overall, multiple studies suggest that current drought and flood regimes are not unusual within the context of last 1000 years [(e.g., Cook et al., 2010; Seager et al., 2008; Graham et al., 2010)].

    I doubt any of these papers “added much to the field” or were “particularly groundbreaking”.

    McIntyre has an more detailed narrative of the above here.

  22. Lazlo

    Well good analyis d_b. but you fail to realise that these bastards are not into rationalism, only fascism. Try taking Jones out to lunch – he thinks you are a oil-funded wanker.

  23. Pingback: Laframboise on the IPCC No. 6 at Catallaxy Files

  24. POR

    I used information from this blog post as part of a response to John Quiggin.

    He now assures me that Donna Laframboise is a liar. I’d like to know whether you are aware she is a liar. Or if anyone else is aware of that.

    I’ve tried to contact Donna Laframboise over Facebook but don’t know whether she’ll get my message. If you know anyone who may know Donna, could you please ask them to let her know that she’s now known publicly as a liar. If she is, that shouldn’t bother her one bit. If she’s not, it is only fair that she know.

    The relevant blog post: http://sabhlokcity.com/2012/07/john-quiggin-alleges-that/

Comments are closed.