My Times column is on the dangers of omitting inconvenient results:
Perhaps it should be called Tamiflugate. Yet the doubts reported by the House of Commons Public Accounts Committee last week go well beyond the possible waste of nearly half a billion pounds on a flu drug that might not be much better than paracetamol. All sorts of science are contaminated with the problem of cherry-picked data.
The Tamiflu tale is that some years ago the pharmaceutical company Roche produced evidence that persuaded the World Health Organisation that Tamiflu was effective against flu, and governments such as ours began stockpiling the drug in readiness for a pandemic. But then a Japanese scientist pointed out that most of the clinical trials on the drug had not been published. It appears that the unpublished ones generally showed less impressive results than the published ones.
Roche has now ensured that all 77 trials are in the public domain, so a true assessment of whether Tamiflu works will be made by the Cochrane Collaboration, a non-profit research group. The person who did most to draw the world’s attention to this problem was Ben Goldacre, a doctor and writer, whose book Bad Pharma accused the industry of often omitting publication of clinical trials with negative results. Others took up the issue, notably the charity Sense About Science, the editor of the British Medical Journal , Fiona Godlee, and the Conservative MP Sarah Wollaston. The industry’s reaction, says Goldacre, began with “outright denials and reassurance, before a slow erosion to more serious engagement”.
The pressure these people exerted led to the hard-hitting PAC report last week, which found that discussions “have been hampered because important information about clinical trials is routinely and legally withheld from doctors and researchers by manufacturers”.
The problem seems to be widespread. A paper in the BMJ in 2012 reported that only one fifth of clinical trials financed by the US National Institutes of Health released summaries of their results within the required one year of completion and one third were still unpublished after 51 months.
The industry protests that it would never hide evidence that a drug is dangerous or completely useless, and this is probably so: that would risk commercial suicide. Goldacre’s riposte is that it is also vital to know if one drug is better than another, say, saving eight lives per hundred patients rather than six. He puts it this way: “If there are eight people tied to a railway track, with a very slow shunter crushing them one by one, and I only untie the first six before stopping and awarding myself a point, you would rightly think that I had harmed two people. Medicine is no different.”
Imbued as we are with an instinctive tendency to read meaning into nature, we find it counter-intuitive that many experiments get significant results by chance and that the way to check if this has happened is to repeat the experiment and publish the result. When the drug company Amgen tried to replicate 53 key studies of cancer, they got the same result in just six cases. All too often scientists publish chance results, or “false positives”, like gamblers or fund managers who tell you about winners they backed.
Outside medicine, we popular science authors are probably guilty of too often finding startling results in the scientific literature and drawing lessons from them without waiting for them to be replicated. Or as Christopher Chabris, of Union College in Schenectady, New York, harshly put it about the pop-psychology author Malcolm Gladwell: cherry-picking studies to back his just-so stories. Dr Chabris points out that a key 2007 experiment cited by Gladwell in his latest book, which found that people did better on a problem if it was written in hard-to-read script, had been later repeated in a much larger sample of students with negative results.
To illustrate how far this problem reaches, a few years ago there was a scientific scandal with remarkable similarities, in respect of the non-publishing of negative data, to the Tamiflu scandal. A relentless, independent scientific auditor in Canada named Stephen McIntyre grew suspicious of a graph being promoted by governments to portray today’s global temperatures as warming far faster than any in the past 1,400 years — the famous “hockey stick” graph. When he dug into the data behind the graph, to the fury of its authors, especially Michael Mann, he found not only problems with the data and the analysis of it but a whole directory of results labelled “CENSORED”.
This proved to contain five calculations of what the graph would have looked like without any tree-ring samples from bristlecone pine trees. None of the five graphs showed a hockey stick upturn in the late 20th century: “This shows about as vividly as one could imagine that the hockey stick is made out of bristlecone pine,” wrote Mr McIntyre drily. (The bristlecone pine was well known to have grown larger tree rings in recent years for non-climate reasons: goats tearing the bark, which regrew rapidly, and extra carbon dioxide making trees grow faster.)
Mr McIntyre later unearthed the same problem when the hockey stick graph was relaunched to overcome his critique, with Siberian larch trees instead of bristlecones. This time the lead author, Keith Briffa, of the University of East Anglia, had used only a small sample of 12 larch trees for recent years, ignoring a much larger data set of the same age from the same region. If the analysis was repeated with all the larch trees there was no hockey-stick shape to the graph. Explanations for the omission were unconvincing.
Given that these were the most prominent and recognisable graphs used to show evidence of unprecedented climate change in recent decades, and to justify unusual energy policies that hit poor people especially hard, this case of cherry-picked publication was just as potentially shocking and costly as Tamiflugate. Omission of inconvenient data is a sin in government science as well as in the private sector.
Post-script:
This column is not mainly about climate change, but about the ubiquitous problem of selective citation of data. As I said, we all do it to some extent, but it is still a sin against statistics. However, as usual when publishing anything that touches on climate change, there has been an immediate and highly misleading attempt to rubbish the work I reported and to imply that I am evil for even reporting others’ opinions that climate scientists, unlike all other scientists, might not all be infallible. This touchiness is quite striking and not reassuring. For those who wish to know the full story of bristlecones and Siberian larch, please follow the links in the piece, one of which is to a Guardian article, and please note that this article reports what one of Professor Briffa’s colleagues had to say in an email not intended for publication:
“In October last year, Briffa’s old boss at CRU, Tom Wigley, said in an email to Briffa’s current boss Phil Jones: “Keith does seem to have got himself into a mess.” Wigley felt Briffa had not answered McIntyre’s charges fully. “How does Keith explain the McIntyre plot that compares Yamal-12 with Yamal-all? And how does he explain the apparent ‘selection’ of the less well-replicated chronology rather than the later (better replicated) chronology?… The trouble is that withholding data looks like hiding something, and hiding something means (in some eyes) that it is bogus science that is being hidden.”
There has been a subsequent attempt to justify the selectivity of the larch data, but it is not very convincing, and I recommend McIntyre’s ripostes to it here, here and here. The rich irony is that one of Briffa’s justifications for ignoring one of the larger samples is that it contains “root collar” tree samples. This is exactly equivalent to the strip-bark problem that leads McIntyre and the National Academy of Sciences to reject the inclusion of “strip-bark” bristlecones in temperature reconstructions – that they give a false signal. They cannot have it both ways.
Leave the last word on this business to McIntyre, discussing Briffa’s explanation for why we should all ignore the fact that tree rings inconveniently show a decline in temperatures in recent years, or “hide the decline”:
“You’ll probably roll your eyes at the following Briffa-ism used to rationalize Hiding the Decline:
‘In the absence of a substantiated explanation for the decline, we make the assumption that it is likely to be a response to some kind of recent anthropogenic forcing. On the basis of this assumption, the pre-twentieth century part of the reconstructions can be considered to be free from similar events and thus accurately represent past temperature variability.’
When I first encountered this, I could not believe that credentialed scientists could either write such bilge. That the authors of such bilge should be among the most respected members of the field was even more unbelievable.”
There is an even more shocking story of data omission in the manufactured attempt to claim a 97% consensus among scientists on dangerous man-made climate change. As Jo Nova details here, the conclusion was based on just 0.3% of the data:
“Of nearly 12,000 abstracts analyzed, there were only 64 papers in category 1 (which explicitly endorsed man-made global warming). Of those only 41 (0.3%) actually endorsed the quantitative hypothesis as defined by Cook in the introduction.”
Post-script 2:
The Times published a letter from the UK chief scientist and the head of the UK Met Office, which rather misrepresented what I said, while conceding my main point – that climate science had not been sufficiently transparent:
Sir,
Matt Ridley falls into his own trap in his Opinion column ( Jan 6), though the title “Roll up: cherry pick your research results here” is apposite, because that is exactly what Ridley does with respect to the research evidence for global warming.
There can be no sensible arguments against making available the results of properly conducted research for open scrutiny. The arguments for this have been rehearsed very effectively in health — and, in general, the biomedical research community has accepted these arguments. Indeed UK scientists pioneered the controlled clinical trial and the Cochrane Collaboration led the way in the rigorous meta-analysis of all sources of evidence to reach the most reliable conclusions allowing the implementation of “evidence-based” medicine. The pharmaceutical industry, which can certainly be criticised for past practices in not revealing the results of all clinical studies of new drugs, is now moving towards greater transparency, and drug regulators, such as the EMEA, are rightly pressing hard. Iain Chalmers, Ben Goldacre and others deserve much credit for their campaigning for openness.
The same can be said of the climate science community. Following the controversy over leaked University of East Anglia emails there have been substantial efforts in making source data openly available. It is partly through this openness and replicablility of findings by researchers in different institutes the Berkeley Earth Island Institute analysis published last year is a case in point— that drew the Intergovernmental Panel on Climate Change to the unassailable conclusion in its most recent report that “warming of the climate system is unequivocal”.
This report was a consensus led by 259 scientists, from 39 countries, which assessed the findings of all of the relevant, peer-reviewed scientific literature published between the previous report in 2007 and March of last year. Would that Matt Ridley applied the same rigour when it comes to evidence about the anthropogenic contribution to climate change. The “hockey stick” graphs, prominent as they were at the time, are just one small part of a massive global research effort that provides consistent and overwhelming evidence.
Sir Mark Walport, Chief Scientific Adviser to HM Government
Professor Stephen Belcher, Head of Met Office Hadley Centre
My reply was as follows:
Dear Mark and dear Professor Belcher,
I am glad to see you recognising in your letter to the Times the need for science, as well as industry, to clean up its act with respect to transparency and data withholding. As for the argument relating to the hockey stick that “following the controversy over leaked University of East Anglia emails, there have been substantial efforts to make source data openly available”, it is good that you acknowledge the role that Climategate played in sparking this improved transparency. Indeed, the surmise by Stephen McIntyre of Climate Audit that the University of East Anglia had failed to report a Yamal regional chronology that did not have a Hockey Stick shape was an important issue leading into Climategate. Yet it was not investigated by any of the East Anglia inquiries. As McIntyre says, “The existence of this unreported adverse result was only revealed by subsequent Freedom of Information requests – requests that were fiercely resisted by the University.” It was wrong that those interested in understanding the hockey sticks had to resort to freedom-of-information requests to get publicly funded data that should have been freely published and wrong that the requests were resisted.
You might be interested in McIntyre’s account (to me) of what has happened since: “In 2013, four years after the Climate Audit criticisms, Briffa and coauthors published a re-stated version of their Yamal chronology with a much diminished blade from the previous superstick. Rather than “discrediting” the earlier criticisms, the re-statement implicitly conceded the validity of the earlier criticism, as shown by the measures taken by Briffa and coauthors to avoid repetition of the earlier mistakes. While they have avoided some of their earlier errors, their new attenuated chronology still contains important methodological defects and errors, as discussed at Climate Audit. Nor should much weight be given to findings of the Muir Russell panel. Muir Russell did not even attend the only interview with CRU academics on the Hockey stick. Nor did the panel interview CRU critics. Nor did the Muir Russell panel even ask Briffa and Jones about their destruction of documents to evade FOI requests.”
You then go on to say that global warming is unequivocal, with which I entirely agree (if we take a 30-50 year period) though it is the evidence, not the number of scientists who have put their name to a report, that convinces me. (It is equally unequivocal that warming has been slower than the models forecast.) But this is a straw man. My article did not claim that the hockey stick was necessary to prove the warming unequivocal. “Unequivocal” is not the same as “Unprecedented”, which was the claim made by the hockey-stick graphs. So I did not “fall into my own trap”. May I urge you in future to address the actual arguments of sceptics rather than the almost entirely mythical claim that they think climate change does not happen.
Best wishes
Matt