After our recent Brief Communication Arising (BCA) was published in Nature on Mora et al., several people have asked about the process involved in getting such a comment published.

Nature apparently only publish a small fraction of BCAs received and allow a reply from the original authors. Roughly one BCA is published online each month and they never appear in the print edition. This post describes what happened in our particular example.

On 10th October 2013, Mora et al. was published online by Nature. On 9th October I had been sent the paper by a journalist and asked for my opinion. After quickly reading the paper, I provided a quote which was used by some media outlets (e.g. Climate Central):

As shown by last month’s latest assessment report by the IPCC, the focus of climate science has moved from whether climate change is happening, to when and where it will be most keenly felt. This kind of work is particularly important as it helps to focus the minds of policy makers about when climate change will start to change our environment. It shows that in many parts of the world, climate change will start to have a major impact in our own lifetimes, not just during those of our children and grandchildren.

My first impression was that the findings of this paper were not really very new and that it had not cited any of the extensive literature that I knew about on this topic. This is not entirely the fault of Mora et al. – the scientific literature is vast – the Editors should choose a range of Reviewers who are then responsible for ensuring the relevant literature is discussed. It should be noted that Mahlstein et al., one of the key missing references, had previously been rejected by the same Editor at Nature. However, these ‘novelty’ issues alone would not be grounds for a ‘Comment’.

The same day I also emailed Camilo Mora to mention the previous literature and to ask a few technical questions so that I could more fully understand his methodology. In particular I didn’t immediately understand how his uncertainties could be so small. We then had some very polite email exchanges. Camilo had become aware of some of the relevant previous literature just before publication and subsequently listed them on his website as clarifications, along with an apology. This was much appreciated, but the list of papers and the apology to the missed authors have since disappeared.

However, the uncertainty intervals quoted by Mora et al. were far smaller than all previous studies by about an order of magnitude. I then had email exchanges with colleagues who had also written papers on the topic and had similar concerns about the missing literature and analysis. Over the next two weeks we identified some key oversights in the methodology and prepared a draft Comment which was sent to Mora on 24th October. Nature’s rules on writing Comments are that there must be a discussion with the lead author before submission. Mora did not believe that our Comment nullified any of his conclusions so we then submitted the Comment to Nature.

The process then became rather lengthy. On 15th November we were informed that our Comment and Mora’s initial reply had been sent for review and on 5th December we received the reviews. Two of the three reviewers generally agreed with our Comments, and one did not think it worthwhile publishing the exchange. All of them made some helpful suggestions and we resubmitted a revised Comment on 10th January which was provisionally accepted on 18th February after further reviews. I also sent the Editor a detailed critique of Mora’s Reply on 26th February. However, more delays resulted because Nature decided on 17th April that the Comment was too long, but by 29th April the shortened BCA was finally accepted.

Then followed discussions with the Editors at Nature about highlighting the BCA and Reply, especially given the wide media coverage of the original study. BCAs are never published in print at Nature (something which I strongly disagree with) but Nature did eventually agree to commission a new ‘News & Views’ piece, written by another independent scientist, to appear in print. I was told this was the first time ever that a BCA would be discussed in this way, and I think this decision has been rightly praised. The pre-proofs arrived for checking on 12th June and the proofs a few days later.

There were then some discussions with the Editors about certain statements in Mora et al.’s Reply which we believed to be incorrect. These errors had previously been pointed out to Nature by the Reviewers and myself in February. Some small changes were made to both the BCA and Reply but, in our view, the key error on estimating confidence intervals remained. Nature argued that the original authors have a right to Reply, and that this has a different set of standards from a journal article.

After nearly nine months, the entire exchange was published online by Nature on July 3rd, along with the News & Views. The exchange was highlighted by Carbon Brief, who videoed myself and Rowan Sutton. I also published a summary of the key issues, a rebuttal to Mora’s Reply and a guest post discussing issues with the GDP & population aspects of the original Mora et al. paper, which had been mentioned to me at a visit to Bern in March.

After publication of the exchange, Claudia Tebaldi wrote to the Editor, pointing out that Mora et al.’s Reply incorrectly cited her paper in support of their results, even though her paper argued the opposite. Nothing was changed.

Overall, a rather long and often frustrating process. Ultimately, I hope that future studies will not now make the same mistakes as Mora et al. did.

Was it worth it? Time will tell.