On observational studies and conflicting evidence.
There are two areas which I have discussed frequently and at length, where there is a conflict of evidence. These are: cycle helmets and seat belts. The conflict is between observational studies, which try to infer benefits of interventions based on analysis of medical records, and real-world figures from whole populations, where those interventions have been tried, sometimes through compulsion, with results that at best fall far short of those predicted by the studies.
Road safety is not the only place where such conflicts exist. There have been issues with studies on vitamin supplements, the effects of hormone replacement therapy (HRT) on coronary heart disease (CHD) and, most controversially, on the link between the MMR triple vaccine and autism.
A number of observational studies were published which appeared to show a link between HRT and reduced rates of CHD in women. Secondary papers called meta-analyses were also published. A meta-analysis is essentially a statement of the state of present knowledge on a subject, summarising past research. The best quality studies yielded a predicted reduction of up to 50% in CHD for combined HRT. Based on this body of evidence, randomised control studies were commissioned. These took time to produce results, of course, but eventually the results came in and showed... null effect or slightly increased risk.
Reactions were mixed: some researchers tried to rework their data to fit the results of the trials, in order to "prove" that they were right all along. Others denied the validity of the new data. Some tried to explain the discrepancy by reference to differences in the study groups.
Interestingly there are other conclusions from similar studies which have been shown to hold up, such as a reduction in breast cancer incidence. One reason which has been advanced to explain the difference is that CHD is more affected by socioeconomic factors, so will tend to be more influenced by any bias in sample selection. Clearly the issue of sample bias is key to the problems with observational studies.
One of the more obvious similarities between the various areas is the existence of a "snowball effect" where observational studies breed other observational studies, which may actively seek to repeat the conclusion because the researchers are genuinely keen to find a solution to a perceived problem, and meta-analyses (which are often little more than "me too" papers registering agreement with the prevailing view) appear to reinforce the view without actually adding any new data. As these various studies cite each other in turn, the list of references grows exponentially without necessarily adding to the actual sample base under consideration.
This is a particular issue in respect of bicycle helmets, where even the suspect 1989 Thomspon, Rivara and Thompson study is still cited in many papers. Authors like to have a long list of references to add weight to their work, and to demonstrate that their work is supported by others. The risk is that an orthodoxy may grow up which is based on a surprisingly small data set.
In terms of safety interventions the problem is compounded by the fact that the observational studies are published in the medical academic press (and the researchers will naturally be inclined to publish in the first place), whereas the conflicting evidence may be presented in entirely different journals or outside the academic press entirely. Considerable momentum may be built up before the medical establishment becomes aware of the existence of conflicting evidence. The medical press is also dominated by an interventionist approach, which may not be appropriate - some recent road safety innovations have gone in entirely the opposite direction, removing the effects of decades of interventions to produce a road environment devoid of paint and signage with, it seems, remarkanble results.
A Familiar Story
As one who has spent more hours than I like to count poring over observational helmet studies and critiques, and pointing out their weaknesses to others, I found a number of comments struck a chord. For example, from Lawlor et. al.:
"Once a hypothesis is shown to be shaky, the protagonists have several options: either dismiss the new evidence as inadequate, re-adjust the focus of the hypothesis, re-adjust their original data in order to fit with the new evidence or exceptionally, drop the hypothesis. Soon after publication of the RCT demonstrating no CHD benefits from taking HRT, re-analyses of the observational epidemiological data began to appear in attempts to demonstrate that the observational studies were not flawed but showed, essentially the same findings as the trials."
The normal response of helmet promoters to real-world evidence follows the first of Lawlor's alternatives: to dismiss the new evidence. Given that the new evidence is based on much larger populations, and observational epidemiology is wholly reliant on the statistical significance of its sample sizes, this is a curious approach. Much of the argument in defence of these studies shows clear signs of having been "seduced by mechanism" (see below), and this also probably accounts for the failure of seat belt advocates to understand and accept the work of Adams and others.
In both cases the fundamental problem may well have been an excessive focus on the narrow parameters being studies, rather than a holistic consideration of the system under investigation.
Pettiti draws four lessons from the controversy, which seem to me to be doubly relevant for safety interventions, where the option of proper randomised trials is generally unavailable for ethical reasons (nobody is going to do a study where randomly selected cyclists are subjected to blows to the head, after all). These are:
Do not turn a blind eye to contradiction
Do not ignore contradictory evidence (for example from whole population time-series data), and especially do not repudiate it, but try instead to understand the reasons behind the contradictions. Investigating the contradictions may, in the case of safety interventions, be the most productive way of moving the debate forward, undoubtedly more so than the endless cycle of repeating the observational studies and pointing out the contradictory evidence.
Do not be seduced by mechanism
Even where a plausible mechanism exists, do not assume that we know everything about that mechanism and how it might interact with other factors. This has particular relevance with cycle helmets, since there are two views of the mechanism of brain injury: coup / contre-coup (where helmets might be effective) and rotational forces (where they will not). To focus solely on the mechanism which fits the desired outcome risks seriously misjudging the issue.
Of the researchers defending observational studies, Pettiti says this: "belief caused them to be unstrenuous in considering confounding as an explanation for the studies". Helmet promoters are often characterised as "true believers" for their refusal to acknowledge the evidence contradicting observational studies, and there is definite evidence of this in some authors. The lesson is clear: do not be seduced by your desire to prove your case.
Another example of failure to suspend belief is the Isles Report, which concluded that there is no evidence that seat belt laws have resulted in safer roads. This was simply buried by a Department of Transport which had already decided to introduce legislation.
Thompson, Rivara and Thompson conclude that helmets prevent 85% of head injuries and 88% of brain injuries. The sceptic in me questions the likelihood of any protective equipment being more effective against the most serious injuries than against superficial wounds such as cuts and scrapes. In my view it is also perfectly reasonable to be sceptical of the idea that 3/4" of polystyrene foam might make a significant difference to serious or fatal head injuries. This is less polystyrene than the typical computer is packed in, and we would not expect a computer to survive being hit by a car because of that packing.
Figures for injury prevention from observational studies are also commonly extrapolated to potential lives saved, another failure of scepticism, since it is very implausible that protection ratios which apply to the least serious crashes will remain constant in crashes which greatly exceed the design capacity of the helmet.
Observational studies have their uses, but are inherently vulnerable to the effects of sample bias and other confounding factors. Their findings should be treated with caution unless backed up by independent evidence from different kinds of research. The existence of similar studies with similar conclusions cannot be taken as corroboration, since they may all share common flaws.
- ↑ Hormone replacement therapy and coronary heart disease, Pettiti D, International Journal of Epidemiology, 2004;33:461-463
- ↑ 2.02.1 The hormone replacement - coronary heart disease conundrum: is this the death of observational epidemiology? Lawlor DA, Smith GD & Ebrahim S, International Journal of Epidemiology, 2004;33:464-467