This morning I tweeted a rash tweet in response to an article on the BBC website which compared NHS patients having certain operations in NHS hospitals with those who go private but the NHS still pays.

Here is the tweet which drew my attention to the article:

Here is my rash tweet:

And here is a reply:

Now, that tweeter has been on at me for eleven years to start a blog of my own. So, today is the day.

I am an epidemiologist and spend some of my professional life persuading other researchers to tone down their scientific claims. This is a problem for all kinds of reasons, including that academics are inappropriately incentivised to overhype their findings. Imagine the pathology department in your local hospital gave prestige, job security, advancement, praise and more money to the individual pathologists who found the most cancer in the biopsy samples they are sent. Not to the ones who most accurately identify cancer: The ones to detect the most – and the rarer the better. This is how academics are incentivised and it is a disaster. But it is not today’s disaster.

The tweet that caught my eye this morning was about an article by Chris Cook. He is policy editor for Newsnight. The headline claimed NHS patients who went to private hospitals for their NHS operations, paid for by the NHS, had “better results”. The reason I tweeted a sigh about it was because there was far more than 140 characters to say about it.

Before I go on, I should say that data should not be only available to professionals like me. I am in favour of data journalism and I am not lobbing any rocks at Chris Cook or the Newsnight team. This could be a story of misincentivised journalists and for that matter media organisations, but that is not today’s disaster either. I should also say I am not utterly utterly wedded to a public NHS either and am interested in a finding which might throw some light on the issue. Private involvement in healthcare provision is widely thought to be a disaster and I have seen some of that myself, but the ‘proper’ NHS is hardly insulated from being a disaster at times. These two disasters are also not today’s.

There is a simple way to think about studies concerning causality. First think about the exposure, then the outcome, then the process of comparison, then what it means including reasons for caution. The well-known PICO mnemonic is similar: Population, Intervention, Comparison, Outcome, which is really about defining a research question for an experimental study but could be used here.

The exposure was having your operation in a private hospital rather than in the good old NHS hospital. In this case, the operations were hip replacements and knee replacements. Easy! For completeness, we should talk about the population, which is people who had either of two operations: replacements of hips and knees.

The outcome is score on questionnaires called PROMs. This stands for Patient Reported Outcome Measures. These are questionnaires people complete before and after their operations in an effort to capture the improvement gained. Their use is now mandated in the NHS for the above two operations, plus varicose veins and groin hernia repair as well.

So at its heart this analysis looked at the people who had their operations in the NHS and the people who had their operations in the private sector and found the people who had their operations in the private sector scored better. Bingo! More private involvement in the NHS please!

Not so fast! Every young epidemiologist is taught to think about four explanations of any association before considering causality. They are chance, bias, confounding and reverse causality. I am going to focus on confounding.

Confounding is an non-causal alternative explanation for the detected association. My favourite teaching example is the startling relationship between having grey hair and dying in the next ten years. Does this finding mean we should infer causality, and demand Grecian 2000 be available to all on the NHS? Or could it be that the grey hair is associated with being older, and being older, sadly, is associated with greater risk of dying? Suddenly all that hair dye is looking like one of those ‘NHS waste’ scandals the newspapers love.

Could the relationship in the article be confounded? Could there be another explanation? in this case there could be dozens or hundreds: Private hospitals tend to not have intensive care units and are notoriously sparsely staffed, so the patients who get in are likely to be less ill than the ones who do not because the decision over where each patient is to go is taken by a clinician who will be managing clinical risks. Also think on this: Poor health and poor health outcomes tend to be concentrated in areas of high deprivation. Now, do you think areas of high deprivation are more likely or less likely to have a private hospital nearby which could take on operations from the NHS? If you are thinking that poor areas are more likely to have worse health outcomes, but less likely to have private hospitals then that’s good. You have probably worked out already that this could produce the reported results, even though it tells you nothing about whether private provision is better. Welcome to my world. Welcome to my job of constantly telling people they have not found what they think/hope they have found. Yes, I often sit alone at lunchtime.

The article reported exactly this, that the people in the NHS hospitals were already sicker than the ones in the private hospitals, but claimed this had been taken into account by the use of a regression model. I’ll have to blog about this in the future, but to cut a long story short, when the variables capture everything about a thing (male/female is pretty good. ‘looks a bit peaky’ is not), when the relationships between those things are quite well understood and when the mathematical distributions of the various factors are known then it works quite well. Sadly, a complex clinical choice to place someone in one hospital rather than another will not be captured by some simple answers on a questionnaire, so there is likely be so-called ‘residual confounding’ when regression adjustments are made in this kind of setting so the remaining relationship is sill confounded. It is this Giant Jenga of assumptions that made me sigh in my tweet this morning.

The problems above are only the small problems. Really they are only some of the small problems. In real science (when functioning properly), before your work is out there in front of the world, being believed and acted upon by politicians, clinicians, individuals, it undergoes ‘peer review’, where all these details are stress tested by a critical friend, often protected by anonymity. Peer review has its own problems but that is not, perhaps expectedly, today’s disaster either. Authors have to show their working to describe what they did and why. The article lacks the basics by which the underlying study might be judged. What was the sample size? What assumptions were used in the regression model? How many missing data points were there? How did they handle the missing data? What assumptions did the statistical tests make? Without these properly reported, and with no real rationale for why the study was done, its impossible to give it a fair assessment.

We know not what, we know not wherefore and we know not whom. I counsel caution before acting, friends.