What is evidence?

One of the more reliable features of quacks, frauds and charlatans is their misrepresentation of what constitutes evidence. Sometimes this is unconscious – the Dunning-Kruger effect – other times it’s down to wilful ignorance, deceit or perhaps insanity.

Medical science is not totally comprehensive, totally objective and totally infallible, therefore unicorns!

A serial offender in the not getting it department is the amusingly mistitled Homeopathy: Medicine for the 21st Century (H:MC21, great name for peddling 18th Century quackery).

H:MC21 have a page entitled “what is evidence“. They could have saved a lot of time if they had just quoted Terry Fuckwitt: “Fuck me, readers, I haven’t got a clue”

What is evidence?

The wibble begins:

There are two main mistakes made about medical evidence:

  • The first is the idea that evidence has validity in the abstract, rather than being dependent on the context in which it is seen.
  • The second is that the only valid ‘objective’ medical evidence is what can be observed by someone other than the patient, and that the patient’s experience is ‘anecdotal’ and invalid.

Up to a point, Lord Copper. In fact:

  • Yes, evidence does have validity in the abstract, that is the definition of evidence in science. In medical science, evidence is essentially definable as a repeatable experimental finding. The main goal of medical research is to remove the subjectivity and isolate effects from context, so that the effect of the intervention can be separated from the effect of context. A great example is the paper “Homeopathy has clinical benefits in rheumatoid arthritis patients that are attributable to the consultation process but not the homeopathic remedy: a randomized controlled clinical trial” – while this trial is small and low-powered, it clearly shows that the therapeutic context (i.e. placebo effects, expectation effects and the like) is the determinant of outcome, and in fact it is this talk therapy which constitutes the clinical intervention, not the magic sugar pills.
  • No, the hierarchy of evidence in medicine does not exclude case reports, testimonials and the like. They are merely given low weight because they are so unreliable due to human fallibility.

To continue:

The need for context

As Karl Popper has pointed out, evidence is not absolute, and

… the belief that we can start with pure observation alone, without anything in the nature of a theory, is absurd … Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem.

Ah. Quote-mining Karl Popper is a favourite tactic of cranks. Popper posited that science is dividied into “normal science” – fleshing out the details – and “paradigm shifts” where new ground is broken. That’s true as far as it goes, but the charlatan is generally studiously ignoring the fact that (a) most crank theories have no explanatory power, no relevance outside their own walled garden; and (b) most crank theories had some plausibility when they were originated, but subsequent paradigm shifts in knowledge have shown this plausibility to be a result of incomplete understanding.

Homeopathy is a perfect example: the idea that quinine cures malaria because there is some bodily response to it which leads to cinchonism in the extreme but cure in lesser doses, is at least superficially plausible, until you realise that quinine actually works buy killing the plasmodium parasite that causes the disease. The idea that there is no amount of matter which is so small that its essential nature is lost sounds plausible until you discover the atom, and eventually the quantum theory of matter.

Popper is right that observation is selective. The goal of science is to ensure that the selective nature of human interpretation is removed, leaving only objective truth.

Some expectation of outcome is needed in order for a test to be constructed in the first place, and what determines this expectation is the ideas of those conducting the test. As a result the evidence produced by a test will be interpreted on the basis of preconceptions.

Yes it is. The issue here is how the expectation is framed. A scientist will design an experiment to answer the question: “is this hypothesis true?” whereas a crank will design an experiment to answer the question “can I demonstrate this hypothesis?” – the difference may look academic but it’s not. Consider the hypothesis that all swans are white. The scientific approach would be to look for a swan that is not white, the discovery of which would immediately refute the hypothesis. The crank will cunt the number of white swans and assert that hundreds of individual experiments prove that swans are white.

That all swans are white is a falsifiable hypothesis: it can be unambiguously proven false by the existence of a single black swan. Many hypotheses made in medicine are not falsifiable in this way. For this reason, medical science will use qualifying terms such as “is associated with”. You rarely see this in crank literature.

One of the best-known examples of a hypothesis that was tested successfully by many believers is the N-ray of Blondlot. This was proven false through the kind of test that skeptics now use to puncture crank claims, but the case is interesting because of its replication prior to refutation. The observers were not seeing what they thought they were seeing. This is the thing that anyone making a bold hypothesis must guard against.

Thomas Kuhn demonstrates this with reference to John Dalton’s work on the compounds CO and CO2. John Dalton was a meteorologist who was interested in the action and composition of gases at the beginning of the nineteenth century. His contemporaries had noted that… the two oxides of … carbon contained 56 per cent and 72 per cent of oxygen by weight … (2)

This did not reveal any obvious relationship between the two oxides, but when Dalton suggested that

… one weight of carbon would combine either with 1.3 or with 2.6 weights of oxygen … a 2:1 ratio leaped to the eye.

As a result

… the same chemical manipulations assumed a relationship to chemical generalization very different from the one they had had before. 

In other words the same evidence had a different meaning depending on the way it was seen.

Dalton worked out that there were two common oxides of carbon. Yes, indeed, there are. This is an established fact. At the time, when little was known of the nature of matter, this was a clever piece of work and its correctness is borne out by subsequent developments, which also tell us why there are two and under what conditions they form. There is absolutely no parallel with homeopathy, because every claim made by Hahnemann has been refuted by subsequent discoveries in multiple other fields of science.

Incidentally, Thomas Kuhn is another favourite of cranks. Popper and Kuhn are not saying what the cranks think they are saying.

When it comes to testing medical treatments, the same principle holds true. The currently prevailing approach is based not on comparing results with general theoretical predictions (the scientific method), but on simply comparing bodies of evidence with each other. Thus the responses of those in the control arm of a randomised controlled trial (RCT) are compared with the responses in the treatment arm, and the overall outcome is compared with clinical experience. But, as with the oxides of carbon, no matter how accurate the information in this evidence, in the absence of a suitable theoretical framework, its true significance cannot be recognised.

Let’s check the WTF-meter.

The currently prevailing approach – the approach which has dragged medicine screaming and kicking out of the dark ages – is indeed based on the scientific method. Hypothesis, test, refine, repeat. Many tests are absolutely based on theoretical predictions. Dopamine is observed to have a role in neurotransmission, deficiency is observed in Parkinson’s patients, dopamine is prescribed, neurotransmission is measurably improved. I suppose if you want to apply an esoteric definition of theoretical prediction you might argue that the dopamine hypothesis was not one, but that would be pure sophistry.

The idea that there is no theoretical framework in pharmaceuticals, invites the question of whether any such framework exists in homeopathy. The answer, of course, is a resounding no. The doctrines of homeopathy are long refuted, no remotely plausible mechanism has been proposed, no other fields of science provide any justification for these beliefs. To assert that “like cures like” constitutes a theoretical framework while the many and varied biochemical pathways that underpin medical science do not, would be spectacularly stupid. Surely nobody would suggest such a fatuous notion?

‘Objective’ and ‘anecdotal’ evidence

This problem is made worse by claims that only ‘objective’ evidence is important, and that the reports of patients are merely ‘anecdotal’ evidence and therefore an invalid basis for assessing treatments. Such an view depends on a fundamental misrepresentation of illness. Both orthodox medicine and homeopathy recognise that illnesses involve both signs (conditions observable by others, such as a bruise) and symptoms (conditions of which only the patient is aware, such as a pain). What is more there is no absolute relationship between them, since signs can be present without symptoms and symptoms without signs. Thus an athlete may die from a heart attack without there having been any conventional symptoms of heart problems preceding the fatal attack. On the other hand a person may constantly report symptoms of heart problems without there being any observable cause for these symptoms.

The opposite of objective is subjective. Anecdotes are not necessarily subjective, but those asserted by homeopaths invariably are. It’s well known that homeopaths are in denial about many things – from germ theory to atomic physics – and this is one such form of denial. Question: Are the recollections of individual humans, an utterly dependable source?

The word evidence always conjures up images of courts. People give evidence in court -0 actually testimony. There’s a body of legal research showing that eyewitness testimony is undependable. And that’s in a situation where the witness has on vested interest in the outcome, and where the body receiving the testimony is independent and interested in truth, not confirmation. The police are interested in confirmation, not truth. Ask yourself this: would you send someone to prison based solely on the evidence of the prosecution? That is what H:MC21 are basically arguing for.

The entire point of randomised controlled trials is to remove subjective and confounding factors.

Here’s an example: a series of case-control studies showed that combined hormone replacement therapy is protective against heart disease. The result was sufficiently compelling ad repeatable that a randomised controlled trial was run. The result? HRT slightly increases risk of heart disease.

So, why would you elevate testimonies, collected by people with a vested interest form people emotionally vested in the outcome, where there are multiple known confounders such as symptoms that naturally wax and wane, when (a) there is no reason to suppose that homeopathy might work; (b) there is no remotely plausible mechanism by which it could work and (c) the consensus of carefully controlled studies, is that it doesn’t work.

The answer of course is because the unreliable testimonies give the answer you want – in fact, precisely the situation H:MC21 falsely assert is true of medical studies.

Perhaps the most commonly occurring symptom which is treated without any attempt at identifying associated signs is a headache. In this case it is the patient who identifies both the existence of the illness and its disappearance, but no sensible medical practitioner would claim that it is merely an anecdotal illness. Only if headaches are frequent or very severe would a practitioner expect to make more detailed investigations to identify other possible problems associated with the symptoms.

This is a complete red herring. A headache is an illness which is hard to measure objectively, but that doesn’t make it negligible. A headache that goes away after a night’s rest is not serious, one which persists, may well be a symptom of a serious condition. A headache is treated by self-medication because in most cases it will resolve itself, but the self-medication always comes with a label counselling medical advice if the symptom persists (at least it does if you use a reputable product).

The fact that most headaches are considered suitable for self-medication without medical intervention, is hardly surprising. Headache is common and rarely serious, symptomatic relief is normally sufficient (and there is debate over whether the symptomatic relief rises much above placebo effect).  Most common symptoms – aches, pains, even diarrhoea – are not considered serious unless they persist. The human body is not a machine, and pain is by its nature subjective.

In short, it is unscientific to claim that medical effectiveness or efficacy can be determined by evidence alone, and particularly if this is allied to an insistence on narrowly defined ‘objective’ evidence.

That’s a massive leap from headaches, and one not in any way justified by the arguments above. To take a single example: we know that quinine kills plasmodium, that plasmodium causes malaria, and thus that quinine cures malaria. All of this is explicitly objectively testable. The dose-response curve can be accurately calibrated.

What H:MC21 are arguing for here is that they, too, should be allowed to claim to cure malaria, but they should be exempted form the tiresome business of objectively demonstrating that their product kills plasmodium because not every single claim of medicine can be explicitly proven using strictly objective tests.

In other words, medical science is not totally comprehensive, totally objective and totally infallible, therefore unicorns.