On December 13th 2022, a preprint was put up on the ResearchSquare platform. Entitled ‘Absence of effects of widespread badger culling on tuberculosis in cattle’ [1], it was a reanalysis of the Randomized Badger Culling Trials (RBCT). Following a protracted period of review, with a number of very long silences, and relatively few revisions, it was published in Nature Scientific Reports on July 15th 2024 (Torgerson et al 2024) [2]. Finding a major anomaly, it uses a range of statistical models to re-examine the RBCT data and concludes that most standard analytical options did not show any evidence to support an effect of badger culling on bovine TB in cattle.
Torgerson et al 2024 [2] noted that the statistical model selected for use in the original study in 2006 [5] was one of the few models that did show an effect from badger culling. However, various model assessment criteria suggest that the original model was not an optimal model compared to other options available. You can read a short blog on the new Torgerson et al paper here, and the full paper is available here. Essentially, the more appropriate models in the latest study strongly suggest that badger culling does not bring about the disease reduction reported.
Following publication, the new analysis [2] was mentioned in an article in Vet Times on July 24th, and in Vet Record in their 3/10 August edition. Neither publication noted its major significance. No other mainstream media reported on it at all. This is perhaps surprising since the government badger cull policy rests all but entirely on the conclusions from the RBCT. It is the science that DEFRA has used in court to defend their decisions to experiment with culling. It is the science that has resulted in 11 (12 including 2024) years of intensive and supplementary badger culling across huge areas of England, and around 230,000 dead badgers. In other words it is the pivotal piece of work for the decision-making around badger culling policy.
The reluctance of the media to report further on Torgerson et al. is a prelude to work by two of the authors of the original 2006 analysis (Christl Donnelly and Rosie Woodroffe) together with a DPhil Statistics student at Oxford University (Cathal Mills), who had, at the time of publication, two rebuttal papers in press with Royal Society Open Science [3,4]. Unusually, the abstracts and supplementary information for these new papers were posted online and available to view before publication without the main text. Enquiries regarding the posted information and the main papers resulted in in press versions being helpfully forwarded. The new analyses from Mills et al. are entitled An extensive re-evaluation of evidence and analyses of the Randomised Badger Culling Trial (RBCT) I: Within proactive culling areas [3], and An extensive re-evaluation of evidence and analyses of the Randomised Badger Culling Trial II: In neighbouring areas [4].
Published on August 21st, the two new papers largely duplicate the analyses in Torgerson et al 2024, but use different model assessment criterion to come to a different conclusion. In fact they double-down on the conclusions of the original 2006 analysis: “…we estimate substantial beneficial effects of proactive culling within culling areas, consistent with separate, existing, peer-reviewed analyses of the RBCT data.”
So in the year and a half since the posting of the Torgerson pre-print [1], Mills, Woodroffe and Donnelly have been working on their rebuttal to it. Torgerson and his team have looked through the in press versions of the Mills et al (2024) publications [3,4] for a few days and multiple problems stand out. In particular:
- Incorrect statements regarding disease control outcomes since 2010
- Use of a non-peer reviewed publication
- Confusion between the offset and overdispersion leading to incorrect calculation of disease exposure
- Incorrect and confused statements regarding model comparisons
- Exaggerated claims during use of new modelling.
- Failure to address the modern interpretation of SICCT test reactors
- Failure to recognise the onward effect of the analytical failure on multiple subsequent publications and policy outcomes.
Somewhat unusually, the new Mills et al papers do not refer to or cite the published Torgerson et al paper [2], only the first pre-printed version of Torgerson et al. from 2022 [1]. So in essence, the new Mills et al papers [3,4] are out of date at the time of publication, failing to refer to the updated 2023 preprint or the final version in Scientific Reports published 15th July 2024 [2]. Following contact with the Royal Society, a response raising concerns with the newly published papers is being written and will be submitted shortly. An interim report has been put together by Torgerson and Langton with brief observations on the new papers (see here).
The Royal Society says it is committed to reproducibility. Reproducibility is the ability of independent investigators to draw the same conclusions from an experiment by following the documentation shared by the original investigators [6]. The issues identified in [2] and “rebutted” in [3,4] illustrate one of the major issues of the reproducibility crisis: poor statistical inference. It is hoped that the conclusions of this exchange will inform future bovine TB intervention policy this autumn.
References