What was the last truly ‘amazing’, ‘novel’ or ‘spectacular’ science paper that you read? According to a recent bibliographic analysis the chance of encountering such ‘ground-breaking’ research has increased almost nine-fold since the 1970s.

Guest post by Rob Wilby, Loughborough University

Vinkers et al. (2015) searched the titles, abstracts and keywords of papers held in the PubMed database for positive and negative words. They found much smaller absolute increases in negative word frequencies, and no change in neutral or random word use.

The 25 positive words were any of: amazing, assuring, astonishing, bright, creative, encouraging, enormous, excellent, favourable, groundbreaking, hopeful, innovative, inspiring, inventive, novel, phenomenal, prominent, promising, reassuring, remarkable, robust, spectacular, supportive, unique, or unprecedented.

The 25 negative words were any of: detrimental, disappointing, disturbing, frustrating, futile, hopeless, impossible, inadequate, ineffective, insignificant, insufficient, irrelevant, mediocre, pessimistic, substandard, unacceptable, unpromising, unsatisfactory, unsatisfying, useless, weak, or worrisome.

Out of curiosity, the same word search is repeated here just for “climate change” OR “global warming” papers. The timespan was restricted to the period 1991 to 2015 to achieve larger samples of papers (>500 per year), and to include only complete years. All searches were performed with the Web of Science data base.

Since 1991, 23,627 papers have been published with positive words and 8,790 with negative words, from a total output of 183,189 papers. Overall, the average incidence of papers with positive or negative words is 12.9% and 4.8% respectively.

These summary statistics do not tell the full story. Over time, as with the medical research literature, there has been an inexorable rise in the frequency of positive words (Figure 1). In contrast, the occurrence of negative words is largely unchanged.

By 2015, the proportion of positive words was 13.7%; 2016 is on target for being a ‘phenomenal’ year as 13.9% of papers so far have positive overtones. This is still less than the 17.5% (in 2014) for medical articles cited by Vinkers et al. (2015), so perhaps climate researchers are a relatively modest bunch? Even so, at the past rate of change, we might expect every climate change paper to contain a superlative by the year 2290.

As with the original analysis of PubMed records, the findings can be interpreted in different ways. The most obvious explanation is that we are shouting louder to get our science heard amidst a rising torrent of output. In the UK, we are certainly incentivised to promote our research as impactful. According to Wolf (2016), exaggeration is also endemic to the promised outcomes of modern grant proposals.

Or maybe we are just more enthusiastic about our research, or linguistic styles of paper-writing have changed, or the quality of our work has simply improved.

No doubt there are limitations to the bibliographic analysis. For instance, both lists could be expanded to count [in brackets] use of even more exuberant positive words such as: “breakthrough” [154]; “revolutionary” [32] or “game changer” [9]. Likewise, the search might include more pejorative words or phrases such as: “junk (science)” [2]; “rubbish” [6]; or “crap” [1]. Thankfully, there is very limited use of these terms.

Additional search criteria, author = “WILBY R”, was added to ground-truth the results and check my own practice. Fourteen out of 82 papers were returned with positive words. Amongst these, six were Hyperbole Hits. They were: “novel” (statistical approaches); “novel” (framework); “novel” (approach to modelling heavy rainfall); “robust” (downscaling technique, twice); and “robust” (hydrochemical model).

Seven of the other results were (self-assessed) as legitimate use of the words: “robust” (catchment management); “prominent” (trend in behaviour); “robust” (climate risk assessment); “robust” (to large uncertainty); “robust” (adaptation to climate change); “robust” (flood protection standards); and “novel” (EU Water Framework Directive approach to environmental protection). One result could not be explained.

Three of my papers came back with negative words: “insufficient” (flood protection standards); “insufficient” (environmental practices); and “inadequate” (climate models as a basis for local climate impact assessment). Again, these were judged as contextual rather than derogatory.

If my profile is representative of the average climate researcher, then it appears that about half of the usage of positive words could be contextual. This still means that ~6% of our community’s papers might be interpreted as self-promoting. My tally of six papers out of 82, gives me a Hyperbole Hit Index (HHI) of 7.3%.

So, in the spirit of shameless marketing of one’s own work, please see Wilby (2016) for more ‘marvellous’ discussion of climate science in practice!

References
Vinkers, C.H., Tijdink, J.K. and Otte, W.M. 2015. Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis. British Medical Journal, 351, h6467, doi: 10.1136/bmj.h6467.
Wilby, R.L. (forthcoming). Who are climate experts? Climate Change in Practice. Cambridge University Press: Cambridge. Chapter 17.
Wolff, J. 2016. University research and the rise of academic bragging contests. The Guardian, 17 May 2016.