A case control study of the effectiveness of bicycle safety helmets, Thompson RS, Rivara FP, Thompson DC, 1989. New England Journal of Medicine: 1989 v320 n21 p1361-7 - US [P139]
Background
This is the grand-daddy of all helmet studies, the source of the widely-quoted 85% / 88% figures. No subsequent study has come close to duplicating these figures, which probably explains why there are so widely used - subsequent figures are probably not deemed impressive enough.
What is quite revealing is that one of the authors, Rivara, was already engaged in surveying and lobbying for helmet use before this study was commenced. In fact, all three were firmly committed to helmet use before starting out, but Rivara is particularly significant because his own figures contradict the helmet wearing rates assumed in this study. Simply substituting his own figures reduces the "helmet effect" to below the level of statistical uncertainty.
Authors' abstract
Center for Health Studies, Group Health Cooperative of Puget Sound, Seattle, WA 98121.
Bicycling accidents cause many serious injuries and, in the United States, about 1300 deaths per year, mainly from head injuries. Safety helmets are widely recommended for cyclists, but convincing evidence of their effectiveness is lacking. Over one year we conducted a case-control study in which the case patients were 235 persons with head injuries received while bicycling, who sought emergency care at one of five hospitals. One control group consisted of 433 persons who received emergency care at the same hospitals for bicycling injuries not involving the head. A second control group consisted of 558 members of a large health maintenance organization who had had bicycling accidents during the previous year. Seven percent of the case patients were wearing helmets at the time of their head injuries, as compared with 24 percent of the emergency room controls and 23 percent of the second control group. Of the 99 cyclists with serious brain injury only 4 percent wore helmets. In regression analyses to control for age, sex, income, education, cycling experience, and the severity of the accident, we found that riders with helmets had an 85 percent reduction in their risk of head injury (odds ratio, 0.15; 95 percent confidence interval, 0.07 to 0.29) and an 88 percent reduction in their risk of brain injury (odds ratio, 0.12; 95 percent confidence interval, 0.04 to 0.40). We conclude that bicycle safety helmets are highly effective in preventing head injury. Helmets are particularly important for children, since they suffer the majority of serious head injuries from bicycling accidents.
General notes
The research upon which this paper is based took place in Seattle between December 1986 and November 1987. This paper has been, and continues to be, by far the most influence research paper in support of the promotion of cycle helmets. It is cited in a considerable proportion of other research papers on helmets, to the extent that many other papers rely fundamentally upon its conclusions in the derivation of their own results.
The figures of 85% for head injury reduction and 88% for brain injury reduction come from only this source, yet are quoted far and wide as gospel by people who know nothing more about cycling, cyclist injuries or cycle helmets. Very few people who cite these figures have ever read the paper or assessed the validity of its conclusions. These two statistics are the keystones of helmet promotion campaigns in the UK by the Department for Transport Road Safety Unit, most local authorities, and the Bicycle Helmet Initiative Trust. In a similar way these statistics have been quoted in support of helmet promotion and mandatory helmet laws around the world.
Authors
The three authors, individually or as a team, are responsible for many other papers on cycle helmets, including meta anaylses of research (in which their own research has sometimes been dominant) including the influential Cochrane Review of helmet effectiveness. The authors have a deep personal commitment to the wearing of cycle helmets, and have also written outspoken campaigning articles pressing for legislation in, for example, the British Medical Journal.
Critique
This paper has been severely criticised by a considerable number of people, both within and outside of the medical profession. At root, its methodology is seen as seriously flawed. In comparison with the paper itself, however, such criticism is scarcely known.
Control groups and helmet use
This paper used two control groups as comparison for the head-injured case group.
Group 1 comprised cyclists seeking emergency room treatment who did not suffer head injuries. There is no reason to suspect that this group differed from the case group in typical membership. Helmet wearing rates in this group were 5.9% for children under 15 and 39.4% for older cyclists.
Group 2 were from families that were members of a single large Seattle healthcare organisation, who filled out a survey form on cycle accidents. On average, these families had higher income and educational achievements than the Seattle population at large. The group was dominated by children under 15 (86%) – adults were too few to be significant. To be included, all members of the group had to do was fall off their bikes during the year. They didn't have to visit hospital and, indeed, only 12% sought medical care for any injuries sustained. The helmet wearing rate for children under 15 in this group was 21.1%.
It so happens that a third control group is also available as a result of a concurrent study in Seattle[1] in May 1987 (of which Frederick P. Rivara was also an author). This observed children under 15 riding around the town and recorded a helmet wearing rate for these children of only 3.1%. Moreover, children wearing helmets were much more often white than black or other races, and riding in parks or on cycle paths than on city streets.
Clearly the population control (Group 2) was nothing like the same as Group 3 in either membership or helmet wearing rates. If Group 2 is considered to represent child cyclists in Seattle, then it may be concluded that helmets prevent 85% of head injuries. However, if the children observed riding around Seattle (Group 3) are considered more typical of child cyclists, then the conclusion would be that helmets have no significant benefit because the helmet wearing rate of head injured child cyclists (2.1%) was well within sampling error of the rate seen on the streets.
Although 5.9% of children under 15 in Group 1 wore a helmet, there were only 12 of these as an absolute number and only 3 helmeted children under 15 in the case group. The numbers wearing helmets are much too small for a valid comparison of whether this control differs significantly from the street study Group 3 in wearing rates. That being so, no conclusion can be reached based on this group about the effectiveness of helmets in reducing head injury.
Sampling bias
The study is not randomised. The people included in both case and control groups are self-selecting samples, in that they choose whether to wear a helmet or not, how they cycle, their attitudes to risk, and many other variables that are therefore beyond the control of the researchers.
Types of injury
Of the 235 head injuries studied, 3 (1.2%) resulted in death and 6 (2.5%) were unconscious for more than 24 hours. Most of the other injuries were minor cuts and scratches. The study does not distinguish facial injuries from other head injuries, although helmets would not prevent the former. 46% of head injuries were to the forehead only. If facial injuries are excluded, the 85% reduction in head injuries comes down to 61%, but the number of cyclists wearing helmets is too small for this to be statistically significant2. In further re-working of Thompson's data, McDermott found that only 40% of head injuries would be reduced using approved helmets, though injury rates increased for the neck, extremities and pelvic region.[2]
Crucially, the study did not include a single case of a helmeted cyclist in collision with a motor vehicle, yet the authors and others have promulgated the results as applicable to all types of cyclist crash.
Authors' acknowledgement of limitations
The authors acknowledged two sources of uncertainty: statistical error due to the fairly small sample, and bias in the sample: "We cannot completely rule out the possibility that more cautious cyclists may have chosen to wear helmets and also had less severe accidents".
In 1996, the authors adjusted their own assessment of the reduction in head injury by helmets to 69% [3], but the original figure is still the one that is widely quoted.
The 85% / 88% figures
These two figures, far and away the most widely quoted in helmet debates, are both misrepresentations.
- The authors calculate an odds ratio and present it as meaning that helmets prevent this proportion of injuries. This is false because odds ratios tend to overstate relative positions. For example, suppose that in a sample of 100 men, 90 have drunk wine in the previous week, while in a sample of 100 women only 20 have drunk wine in the same period. The odds of a man drinking wine are 90 to 10, or 9:1, while the odds of a woman drinking wine are only 20 to 80, or 1:4 = 0.25:1. Now, 9/0.25 = 36, so the odds ratio is 36, showing that men are much more likely to drink wine than women. But actually men are only 4.5 times more likely to have drunk wine than women.
- In the above example it might also be supposed that being female protects you from drinking wine, or drinking wine protects you from being female. Of course, no such causal relationship exists. In the same way, all we can say from this data is that the helmet wearers were less likely to suffer head injury, not that the helmets prevented the injuries, because the populations are clearly different in many important respects, not least where they were riding.
- The figures are in any case an estimate. The tables in the paper give actual figures closer to 75%, but the authors "adjusted" this to give the much higher figure, using some pretty dubious reasoning.
- Analysis by an independent statistician of the full data set showed that the figures for lower-body injuries were similar; in other words, the helmeted riders were "protected" as much from broken legs as from head injuries. The full data set has not been released again since, to my knowledge.
Robinson's critique
This is a critique written by Australian statistician Dorothy Robinson, with some minor amendments for context
There have been several case-control studies of helmeted vs non helmeted cyclists. They generally tend to show fewer head injuries amongst those choosing to wear helmets. They also tend to show helmeted cyclists were involved in fewer bike-motor vehicle accidents.
Other studies have found helmeted cyclists stop at stop signs and traffic lights, make correct hand signals and are generally more likely to obey the rules of the road.
In the US, helmeted cyclists are more often white than other races, and tend to have fewer non-head injuries as well as head injuries. (This is not true in other countries, possibly because of differences in healthcare systems.)
The problem is it's very hard to separate out these fundamental differences and correctly adjust for differences in riding behaviour. Did the helmeted cyclists have fewer head injuries because they were better at obeying the rules of the road, or because of their helmets? Or was it a bit of both? I don't know for certain, but it appears that many of the case-control studies have found an unrealistic and exaggerated effect of helmets because they haven't adequately been able to adjust for all these confounders.
It might be supposed that since the self selecting group also had similar reduction in non-head injuries, the most reasonable explanation is that they visited the hospital with less serious injuries.
Actually no. This group didn't necessarily visit to the hospital at all. They were children of families belonging to a Seattle healthcare organisation who filled out a survey form on bike accidents. To be counted, all they had to do was fall off their bikes in the previous year (ie 1987 ish). Only 12% sought medical care for any injuries sustained. (They tried to count adults as well, but so few actually responded that you can't do much sensible statistical comparison, so I'11 not discuss them).
The helmet wearing rate in this group of kids under 15 was 21.1% Street surveys also took place in Seattle in of helmet wearing in kids with estimated age of 5-1 5 years. Observed helmet wearing in this group of kids riding was 3.1% (May 1987; sample 1957) and 3.3% (September 1987; sample 2544).
The most likely explanations for the differences in helmet wearing between the two groups are that: a) Parents in this particular healthcare organisation were better at persuading their kids to wear helmets than the average parent whose kid was seen riding round Seattle. (On average, healthcare organisation members had higher income and educational achievements than the Seattle population) and / or b) Kids wearing helmets are more likely to fall or admit to falling off their bikes.
Of course, you get completely different interpretations of the benefits of helmets, depending on which group you consider the 'control'. If you consider the healthcare kids as representing (in every detail including helmet wearing rates) the population of child cyclists in Seattle likely to be injured in cycle accidents and require emergency room treatment, then you would estimate that helmets prevent 85% of head injuries. If you consider the kids seen riding round Seattle as representative of the helmet wearing rates of the population of child cyclists likely to be injured and require emergency room treatment at any of the 5 major Seattle hospitals, then you would conclude helmets had no significant benefit, because the helmet wearing rate in head injured child cyclists (2.1% or 3/143) at those hospitals was well within sampling error of the observed helmet wearing rate of the kids seen on the streets.
As you can imagine, these studies are fraught with difficulties!
Thompson et al.'s study in fact used *two * control groups; helmet use at the time of crash was 23.3% and 23.8% for the two control groups. In the T&R study, helmet wearing varied considerably with age.
In the emergency room (ER) control (non-head injured group), 5.9% of children less than 15 wore helmets, but 28.7% of those aged 15-24 and 47.7% of those aged 25 and over.
Now in the second (community) control, 86% were under 15. This means that any statistical comparison between case patients and the community control group will be almost entirely reliant on the comparison for the under 15s. There simply weren't enough older cyclists in this community control group to have much effect on the overall estimate. And, surprise, surprise, helmet wearing for under 15s was 5.9% in the ER control vs. 21.1% in the "Community" control. Given that 86% of the community controls were under 15, most of the information will come from contrasts in this age group. Wearing rates of the two groups are hardly the same!
The study period was December 1, 1986 through November 30, I987. A survey of under 15s riding round the streets of Seattle in May 1987 showed wearing rates of 3.1% (sample size 1957) and in September 1987 3.3% (sample size 2544). This is clearly nothing like T&R's Community control!
Remember that 86% of the community control were under 15 Therefore most of the information using the community control comes from the age group. What was it?
| Riding around control | Emerg Room HI | Emerg Room Control | Community Control | |
|---|---|---|---|---|
| No | 4501 | 143 | 202 | 478 |
| %helmeted | 3.2% | 2.1% | 5.9% | 21.1% |
| number helmeted | 144 | 3 | 12 | 101 |
Now, I wouldn't like to conclude from the above data that those given emerg room treatment for HI were different to the riding around control - the numbers wearing helmets are far too small for any sensible statistical interpretation. I can, however, confidently say that the community control was vastly different in its helmet wearing rate from the riding around control.
Perhaps because of the way they were chosen, as members of a healthcare cooperative who had fallen off their bikes, I'd guess that either falling off your bike, of being a member of that particular cooperative (which may have promoted helmet wearing to its members) can explain the exceptionally high helmet wearing in this group.
However, anyone who claims helmet wearing in the general population was the same as in this second control group, given the survey of street cycling at the time, is almost certainly trying to pull the wool over people's eyes. And anyone who bases an assessment of helmet effectiveness on a comparison of that community control (which was obviously different to the kids riding around Seattle at the time) is also trying to pull the wool over people's eyes. So the statement:
- Thompson et al.'s study in fact used two control groups; helmet use at the time of crash was 23.3% and 23.8?% for the two control groups
is an excellent example of "How to Lie with Statistics"
If you have, (see note at end of posting for further info)
| ER contol | Comm. Control | |
|---|---|---|
| child helmet rate (Sample size) | 5.9% (202) | 21.1% (478) |
| adult helmet rate (Sample size) | 39.4% (231) | 36.2% (80) |
| Combined rate | 23.8% | 23.3% |
The combined rates may look the same, but this is only because helmet wearing differs with age and the ER control has more older cyclists who have a higher helmet wearing rate. (Note: here 15-24s and 25+s have been combined, to simplify the data; Given the small numbers (29 and 5 1) in the comm control helmet wearing rates of 24% and 59% for this group would be subject to large sampling variation).
But no-one can interpret the figures of 23.8% and 23.3% sensibly without being told of the differences in numbers of young and older cyclists in the two samples.
The extraordinary claims by people like Sacks et al. (JAMA 1991) that "universal use of helmets by all bicyclists could have prevented as many as 2500 deaths", (out of a total of 2985 deaths in which some skull or brain injury was recorded) is just plain silly and very detrimental to cycling in general.
If your dataset has the following head injury rates (as reported for Thompson et al's data in a subsequent publication):
| Age |
| 0-4 83%(10/12) |
| 5-9 42% (43/102) |
| 10-14 23% (27/116) |
but you ignore these age categories (as Thompson et a1 did in the NE J Med paper), lumping them as one, you could get a very biased and spurious estimate of the effects of helmets if substantially more of the 10-14s wore helmets than the other age groups.
I don't know the distribution of helmet wearing in this age group, but ignoring the different HI rates for the different age groups has the potential to cause serious errors.
Another example
Let's apply the T&R methodology to their community control group of under 15s who fell off their bikes, compared with the large survey of kids riding around the streets of Seattle. So the case patients are those aged under 15 who fell off their bikes (ie recruited by T&R because they were in / had a bike accident). The group had 21.1% helmet wearing (101 out of 478).
Controls are those of the same age group who DIDN'T fall off their bikes, but were counted in the surveys riding in Seattle in May and September 1997 - contemporaneous with the T&R study. Helmet wearing in controls, was 144/4501
So, the odds of falling for helmet wearers = (No helmeted in case)/(No helmeted in control) = 101/144 Similarly, the odds of falling off your bike for for non wearers (No nonhelmeted in case) / (No nonhelmeted in control) = 377/4357
The odds ratio is therefore 0.1185, Or in T&R terminology, NOT wearing a helmet is associated with an 88% reduction in the risk of falling off your bike.
If you like, I can run this through a logistic regression program to get the confidence intervals, but I assure you it would be HIGHLY significant.
This doesn't necessarily mean that I believe wearing a helmet caused the kids to fall of their bikes. It may have had something to do with the way the sample was chosen.
I think T&R are also blinded by their belief in helmets. Rivara was an author of the paper which reported the survey results for kids riding around Seattle, which, like the NE J Med paper, was published in 1989. He must have been aware of the inconsistency these results.
Non-helmeted kids tended to ride on the streets, helmeted ones on parks or paths. This means that non helmeted kids, riding on streets, may have had relatively more collisons with cars. If true, this matters a great deal. The risk of head injury in a car-bike collision is three to 5 times higher than a bike only accident.
The T&R 1989 paper had incorrect adjustment for age. They lumped all the under 15s together, yet it turns out that 83% of the 0-4s had head injury, 42% of the 5-9s and 23% of the 10-14s. (You can't tell this from the paper, but they published a follow-up a year later on a subset of the data).
References
- ↑ Bicycle helmet use by children. Evaluation of a community-wide campaign. DiGuisseppi CG, Rivara FP, Koepsell TD, Polissar L. Journal of the American Medical Association 1989, vol 262 p2256-2261.
- ↑ The effectiveness of bicycle helmets. McDermott FT, Lane JC, Brazenor GA, Debney EA. Journal of Trauma 1993, vol 34 p834-845.
- ↑ Effectiveness of bicycle safety helmets in preventing head injury: a case-control study Thompson DC, Rivara FP, Thompson RS. Journal of the American Medical Association 1996 vol 276 p1968-73.
