On Friday 26th September, Defra changed the “Bovine TB strategy review update” but simply made things worse by adding another layer of confusion.
An error was reported as “a revision to a line of code”. In the document itself, the change is referred to as a ‘cut and paste error.’ Red text in a box next to the correction (see above) suggests that ‘The rest of the analysis and interpretation is unaffected’.
The replacement red numbers, now show the outputs from their 4 models described, but this time the difference between models 3 and 4 is insignificant. To put it in technical terms, a delta (difference) AICc of 0–2 suggests both models have similar support from the data.
However, far from the analysis and interpretation being unaffected as they suggest, the revision does have serious implications for the inferences drawn. The ‘update’ is actually a correction due to the wrong results being inserted into their models in Annex 4. The results reported as AICc (Akaike Information Criterion) were in fact the results from Baysian analysis (BIC) (which they did not report on).
However, BIC is not the correct IC for model selection, due to the small number of data points and large number of coefficients to estimate. The small sample size equivalent (was this really not tried?) would be appropriate (BICc), especially for logistic regression. Applying BICc favours a model which suggests that the incidence of herd breakdowns is independent of culling (i.e. culling produces no effect).
If AICc is used for model selection, a model suggesting a weak effect of culling is only slightly favoured, model 3: 173.58 vs model 4: 173.87. This difference is too small to draw any inference from; it is effectively a toss-up between the two models. The incorrectly reported IC results showed a bigger difference: model 3:151.58 vs model 4:159.38). It is this greater significant difference, which is what the peer-reviewers seem to have picked up on. Hence the big question, what were the peer reviewers sent? And given that one or both reviewer assumed the code was right, were they commenting on the wrong results?
Apparently Defra have been asked this all-important question and an answer is pending. Maybe they don’t know and need to ask? The answer could alter the Chapter 6 discussion hugely. A very large number of people are interested to find out who knew what and when.
One might pose the question “did the Godfray group try the small sample size BICc?” Given that they obviously tried BIC (as that is the output they incorrectly reported), the correct form of BIC would be the small sample size correction as there are only 20 data points. Interestingly, when BICc is applied, model 4 (without badger culling at 196.00) is favoured over model 3 (with badger culling at 200.96). I.e. badger culling has no effect.
However, notwithstanding all this incredible muddle, it would have been more appropriate to select the most favourable model of a series of over-dispersed models, using the quasi version of AICc, which is QAICc, especially as the quasibinomial model was subsequently used for statistical inference.
When the QAICc is applied to their models, the results strongly favour a model which again suggests bTB incidence in cattle is independent of badger culling (i.e. culling produces no effect).
A pre-print outlining some of the problems with the new analysis is available here.
Reasonable people can disagree….
When dealing with the recent academic publications on the RBCT, (which have been reported on here), and which arrive at different conclusions, the Godfray review has stated at para 6.LXX:
“…reasonable people can disagree about the best way to analyse complex data such as these.”
That is an interesting response to the current differing views. A short article entitled “When Can Reasonable People Disagree?” is quite insightful on this. We borrow a couple of apposite quotes below:
“To say that “reasonable people can disagree” can encourage suspension of judgment in response to important matters of personal and social concern.”
“There is a related tendency to use the expression “reasonable people can disagree” to create an equivalence when it comes to conflicting beliefs that large groups of people hold. A common perception exists that if a large enough group of people believes something, it has met the litmus test for being a “reasonable’ belief and the people involved “reasonable people.” This is a familiar critical thinking error. The fact that a large group of people believes something does not provide us with a compelling reason for thinking it is true.”
“Respect for persons might require engaged listening. It might require careful consideration. It does not require treating weak evidence as if it is strong nor does it require treating unsound arguments as if they are sound. Far from being kind, we are actually doing one another an injustice when we engage one another as if all arguments are equally compelling.”
The point is, and it is a point made by Prof mark Brewer earlier this year in his review of Torgerson et al 2025, it is more a question of can reasonable people agree? And the answer to that appears to be not yet. But there needs to be conversation. So far Defra has refused to have that conversation. Will that continue or does it have to wait until there is an inquiry into the crisis?
In any event, this muddle is so bad that the Godfray review will need to be rewritten. That is unavoidable now. The mistakes are too obvious, too profound and too important to remain as a part of the narrative.
While there is a ‘it doesn’t matter anyway’ type sentence to cover up any exposure, it is quite clear that it really does matter. The mistakes alter everything, including the economic forecast on badger culling benefit.
Its time for Defra to change the way they do business.