Godfray ‘science’ report simply adds to the bovine TB chaos

The long-awaited ‘Godfray’ report, a review of scientific evidence since 2018 concerning bovine TB in cattle, was published on 4th September. Commissioned by Defra, it is supposed to feed into a ‘comprehensive new bovine TB strategy’ that was announced in August 2024 by the incoming Labour government. It is the science that the new policy should be based on, so it is an important – and needs to be sound. More details will follow on the entirety of the report, but this is what was said about badgers and bovine TB

Chapter 6: The Disease in Wildlife focuses on badgers. The report does not deal head-on with the Royal Society Open Science pre-publication review by Prof Mark Brewer. This  claimed that models suggesting ‘badger culling works’ were ‘naïve at best’. Instead, a newcomer to the issue, statistician Prof Bernard Silverman (a colleague of Christl Donnelly at Oxford University) has tried to rescue the situation & restore statistical validity to show some positive disease benefit.

Silverman  presents a new set of models. He confirms (para. 65) in a massive ‘wake-up’ finding that a paper by  Prof Torgerson’s study group in June of this year did show that the key 2006 RBCT paper by Donnelly and others, in short, got the modelling wrong. This has massive implications for a wide number of papers that have used that paper’s calculations to build further models. It must cause a tsunami of scientific correction, or the retraction of dozens of publications that have been used to promote  badger culling over the last 20 years. This is potentially one of the biggest shake-up’s in biological science, for a generation. The ‘perturbation effect hypothesis’ evaporates, for example.

Instead of the huge significance claimed in 2006 for badger culling, Silverman has tried to produce a  model that ‘just about’ finds an effect from badger culling, providing ‘weaker evidence for a positive effect’.  But nevertheless, at first glance, it might save Oxford’s blushes. The problem is that he gets to this position in a manner that is both unconvincing, and incorrect.

Firstly, there has been a major howler. Not just from Silverman but the two peer-reviewers asked to check his models, and Defra officials who supervised the process. Annex 4 of the Godfray report outlines the binomial model he has used, but gives the wrong information criteria (ic) outputs. These are standard applications that test how well the model fits the data. He claims to have used what is called Akaike, but instead presents Bayesian outputs. One assumes that this was not intentional, but rather a transcription error.

However, in addition to this, he has failed to address the fact that for binomial models with small sample sizes (the RBCT was an experiment with a small sample size – just ten paired comparisons), his binomal approach should have used a particular type of Akaike information criteria. When applied, results suggest that models showing a benefit from badger culling are those least supported. The best supported models are those that do not include badger culling, indicating badger culling had no effect. Even if he had decided that the Bayesian approach was the correct way to evaluate the models, the small sample size variant should have been used and this too makes badger culling the wrong model to favour from a statistical, hence scientific perspective. A pre-print outling the problems is available here.

So any ‘merit on both sides’ of the current modelling debate, the football equivalent of a score draw, does not apply as he had hoped. Torgerson’s analyses win handsomely on penalties and the RBCT is relegated.

For the benefit of stats geeks wanting to understand more, it is also fair to say that with Poisson regression, it is easy and quite natural (and now accepted by all), to include ‘time at risk’ in the model (as an offset), which is why Poisson regression would be the preferred method for the analysis. With binomial regression, to do this properly you need to have the complementary log-log function in the link. Again, when this is done, there is no effect of culling. Silverman has also not dealt with overfitting – there are too many parameters for the number of data points. He also fails to address the ‘all reactors’ argument that another chapter recommends needs ‘more research’. Could this be because finding in favour of using ‘all reactors’ would be yet another route to showing the data finds badger culling to have had no effect. Saving the RBCT is more important than admitting this essential pointer for cattle management?

The implications of a flawed report to the Minister are huge for Defra and Godfray, who chose Silverman to try to rescue the unsavable. It looks such a crude attempt that the new Minister (Zeichner has now gone) will have some difficult explaining to do. Do farmers get their money back for doing something pointless for the last 12 years, and do wildlife charities get compensation for rightly fighting, at huge cost, a scientifically botched policy? It’s going to be interesting. And if culling doesn’t work, neither will badger vaccination or TVR, which appears to be Defra’s new direction of choice. The second Godfray report could potentially be seen as a back-covering exercise to try to protect Oxford University, but it has not succeeded.

This is grounds for a major inquiry, with standards of scientific integrity and the impartiality of appointments under the spotlight. Badger culling would never have been sanctioned if the RBCT had got its statistics correct back in 2006.