A Tug of War Between Incidence and Mortality
A frequently debated issue in making the case for or against vaccines is on how to measure the disease: Should we use cases/incidence of a disease or should we use mortality rates when measuring vaccine success? The CDC says that you should use incidence to measure vaccine effectiveness, yet many vaccine skeptics often quote the prevalence of measles using death rates or mortality rates. There’s definitely a battle between which is right.
If you look at the following graph, you would conclude that it matters quite a bit! The left panel suggests that vaccines are incredibly effective because as soon as the vaccine is introduced, the number of measles cases drops precipitously from around 400,000 cases per year to close to zero. Who can argue with that?!
The right panel, however, is used to emphasize the fact that vaccines did not save us because mortality rates from measles were declining exponentially leading up to when the Measles Vaccine was introduced and was a relatively trivial cause of deaths. The mortality rate (before vaccines) in 1962 was less than 2 per 1 million people, which is less than the chance of being electrocuted [1] or of being murdered by a serial killer [2]. Measles was generally viewed as a mild disease and officials were concerned that the public would not feel compelled to vaccinate against it [3], even though it can also lead to non-fatal complications (death is not the only bad outcome).
Anyway, given the two possible views, it is no wonder we are so polarized!
Apples to Apples
However, there are are couple of things that are wrong with the comparison of these two charts and it is lazy not to standardize them. One issue is that the cases in the left panel are just absolute numbers (not a rate) and it shows that the number of cases is quite flat even as the total US population is rising. The right-panel is a rate (deaths per 100K), which is the appropriate thing to do when comparing the disease temporally over time because it adjusts for population growth. The second thing is that the x-axes are across different time-periods, so it’s hard to make a comparison. Moreover, the declines are declining exponentially and so we should transform the rates on the y-axis onto a log-axis.
Let’s fix all that! In the two blue lines below, both incidence and mortality are shown divided by the total US population (from census data) and they are shown to span the same time-period. I am using the same data source as above, except that in order to extend the number of cases back to 1927, I downloaded the weekly number of cases from Project Tycho [5]. The red line is the Measles vaccine coverage rate (for 19-35 month old children after 1993 and for 2-year-olds prior to 1993) [4] .
While incidence varies widely from 100 to 800 people per 100,000, the overall range is flat prior to 1962 and when I measured it, the slope was not even closely statistically significant. You cannot make the argument that incidence was on on the decline. Rather, the data unequivocally says that incidence pretty much grew in tandem with population growth until after the vaccine was licensed and that prior to that there was no long-term downward trend.
The CDC criticizes the use of death rates because death-rates were falling due to the advancement of medicine, improved sanitation, and nutrition. Their point is that the advancement of medicine allowed a person afflicted by polio-induced paralysis to live possibly for many years to come (f.ex. by means of an iron lung ) and therefore mortality rates don’t measure exactly what it is that we want to measure because while a lot of things affect a person’s ultimate survival rate, only infection can lead to incidence.
I want to address, however, that using incidence comes with its own host of problems and that there are huge benefits in terms of data-quality for using mortality rates as a proxy. Incidence reporting, it turns out, is whimsical and highly susceptible to scares and the reporting fashions of the day. In other words, it depends a little too much on what is on the news to be a reliable measure and the literature is critical of its use.
The Possible Case Against Incidence
A nice summary of the problem with incidence can be found in a classic Statistics Textbook called “How to Lie With Statistics”. See page 84 for a description of how ‘cases’ fall victim to the era’s fashion of diagnosis:
…a general consciousness of polio was leading to more frequent diagnosis and recording of mild cases. Finally, there was an increased financial incentive, there being more polio insurance and more aid available from the National Foundation for Infantile Paralysis. All this threw considerable doubt on the notion that polio had reached a new high.
What Huff concludes in this Statistics Textbook gem is that
It is an interesting fact that the death rate or number of deaths often is a better measure of the incidence of an ailment than direct incidence figures – simply because the quality of reporting and record-keeping is so much higher on fatalities.
True Incidence Unknown
To be fair, mortality rates share some of the draw-backs to which incidence falls victim. See The Questionable Contribution of Medical Measures to the Decline of Mortality in the US in the Twentieth Century, which points out that mortality rates are also subject to the fashion of the day or that there is a complex set of conditions (rather than just one disease) that results in death. Other issues include changes in disease classification, inaccuracies in recording the cause of death, and changes in the registration area of the United States.
As the study points out, however, some of these errors will be averaged out when pooling or aggregating countries (and the same applies to incidence data as well). If there are weaknesses in mortality data those weaknesses will only be amplified in incidence data and the former therefore has a strong relative advantage over incidence.
Incidence is Fickle?
What we are studying is what health departments call this “completeness of reporting”: the number of reported cases as a percentage of true cases.
Measles surveillance is complex and many factors can influence completeness since 1) the patient must first seek health care, 2) the diagnosis must be recognized by the physician, and finally 3) the case must be reported to health departments. According to this study from the Journal of Infectious Diseases
Estimates of completeness of reporting from the 1980s and 1990s vary widely, from 3% to 58%. One study suggests that 85% of patients with measles sought health care, the proportion of compatible illnesses for which measles was considered varied from 13% to 75%, and the proportion of; suspected cases that were reported varied from 22% to 67%. Few cases were laboratory-confirmed, but all were reported. Surveillance in the United States is responsive, and its sensitivity likely increases when measles is circulating.
If you continue to read the paper, you will see that all disease-reporting requirements come from state laws and regulations. Reporting of measles is required in all states but national criteria for classifying measles fall into the following categories: 1) Suspected 2) Probable and 3) Confirmed and it is not clear or consistent with regard to which of these categories is reportable and when reporting should occur during the diagnostic process. Reporting is a passive process initiated by the reporter, there are no penalties for failure to report, reporting is often incomplete, and few diagnosis are actually laboratory tested and confirmed.
In J A Bean, et al; A comparison of national infection and immunization estimates for measles and rubella. Am J Public Health. 1979 June; 69(6): 611–612, they show how widely measles incidence varies from different data sources. In 1968 were there 200 cases or closer to a 1000 cases?
“The researchers conclude that there is little credence to the validity of any of the data sources and the CDC Morbidity and Mortality Weekly Report (MMWR) figures do not appear to be reliable national estimates.”
In these papers, it was estimated that the CDC was under-reporting incidence by a factor of 10 (i.e. only 1 in 10 true cases reported). I note that this is roughly by how much the measles cases dropped in the top left hand panel.
During “scares” overreporting according to UK Health Officials has been as high as 7,400% or 74 times actual lab-confirmed results (CDR Weekly, Volume 15 Number 12). 1,500% over-reporting shown in CDR Weekly, Volume 16 Number 12.
Case Fatality
Another reason that I think that the decline in incidence is overstated after vaccine licensure is that case-fatality trends upwards. In other words, even as the CDC is claiming that Measles is getting less virulent and deadly over time because of medical advancements and other factors, it was appeared to become twice as deadly leading up to and through the first couple of years after vaccination programs. See the blue line below, which divides mortality by the number of cases. This is a graph and a trend that is not often analyzed.
My hypothesis, given what we know about the variability in completeness of reporting, is that the blue line trends upwards because people simply stopped being vigilant about reporting cases of the disease in that decade, maybe because it was less threatening and people generally felt safe and protected because of the newly introduced vaccination program. Meanwhile, deaths were still being reported fairly accurately. This hypothesis is more palatable to me than other explanations that would mean that medical advancements were failing, or that nutrition and sanitation got worse, or that the virulence of measles increased for other reasons.
And if it is true that the actual case-fatality ratio did not actually double from 1955 to 1970, then the decline in reported incidence may be overstated by a factor of two.
Decline in Reporting Greater than Coverage
Coverage rates refer only to coverage rates among children, not to the global population including adults, adolescents, and other older children. Note that the coverage rate for 1-4 year-olds was 33 percent at the end of 1965 (USIS data [6]), which would have vaccinated 7 million children in that age group because the US population had about 21 million children in that age bracket [7]. It is a little remarkable and difficult to believe that vaccinating one third of a subset of the population caused a 54 percent drop in cases – In the five years before immunizations (from 1958-1962) there were an average of 503.3K cases and the average number of cases in 1965 and 1966 was 233K.
However, the total number of children that received vaccinations was 15 million in the years between licensing in 1963 and continuing until mid-1966 [3]. Hence, the coverage rate against the entire US population of 191.9 million people in 1962 was less than 8% (15M divided by 192M). I am assuming very few adults received the vaccines because the policies were aimed at children [3].
Ultimately, an 8 percent global coverage rate led to a 54 percent drop. Even if all the cases were children, the low 33 to 45 percent coverage rate [4] in that time period among children still makes it hard to imagine a drop that large. I am speculating, again, that the drop came from completeness-of-reporting issues, but more research is needed and we do not have an age break-out of cases prior to 1973 [8].
Conclusion
OK, I have been beating up on incidence, but my conclusion is not what you might think.
Everybody agrees that sanitation, sewage systems, nutrition, the structure of social networks, education, and awareness all play an integral role in controlling disease and mortality. However, we have seen that it could potentially make a huge difference in assessing the effectiveness of vaccines depending on whether we select incidence data or mortality data.
Incidence is better because it includes sick people that don’t die, but it is worse because it is impossible to measure accurately and in some cases officials over or under-reported it by a factor of 10. It is something that we don’t really know. And who is to say that the dramatic drop in incidence following the 1963 licensure wasn’t in-fact a drop in completeness-of-reporting?
Even though incidence is incredibly fickle and is subject to fashions of the day, and even though it fluctuated wildly from 1927 to 1962, it did so within a fairly stable range above 100 per 100,000 population. Only after the introduction of the vaccine did the incidence decline below that. As many as 95 to 98 percent of children [9] had Measles by the time they were 18 years old. It was very common and it makes sense that the incidence rate was stable and that incidence grew in tandem with population growth as shown in this article.
The mortality rate, however, was on the decline, and not necessarily only because of nutrition, sanitation, and medical advances, but possibly also because the population developed natural immunity against the disease. In other words, the virulence of the disease was on the decline as seen through the declining case-fatality ratios. In “primitive” populations, Measles has been known to eradicate 20-30 percent of the population [10]. The virulence of a disease, its morbidity and its deadliness, is not a fixed, static thing.
The drop in incidence following vaccination is probably larger than it should be, but as I will show in a later article, even the drop in mortality can be shown to be statistically significant. I think that the effectiveness of the campaign was effective, but not as effective as health officials would like to believe and not as ineffective as anti-vaxxers would have us believe. The truth is somewhere in the middle, at the intersection of the two narratives.
References
[1] Electrical Injuries: Background, History of the Procedure, Problem. Medscape.com. https://emedicine.medscape.com/article/433682-overview#showall. Published October 5, 2018. Accessed January 23, 2019. ↩
[2] Serial killers may kill more victims than we think: IU News Room: Indiana University. Iu.edu. http://newsinfo.iu.edu/web/page/normal/7225.html. Published January 1, 2012. Accessed January 23, 2019. ↩
[3] Hendriks J, Blume S. Measles vaccination before the measles-mumps-rubella vaccine. Am J Public Health. 2013;103(8):1393-401. (doi: 10.2105/AJPH.2012.301075) ↩
[4] CDC Pinkbook Reported Cases Data Appendix E ↩
[5] Van Panhuis W., Cross A., Burke D., Counts of Measles reported in UNITED STATES OF AMERICA: 1888-2002 (version 2.0, April 1, 2018): Project Tycho data release, DOI: 10.25337/T7/ptycho.v2.0/US.14189004 ↩
[6] Hinman, Alan; Orenstein, W.; Papania, Mark; “Evolution of Measles Strategies in the United States“; Journal of Infectious Diseases 2004; 189 ↩
[7] https://www.populationpyramid.net/united-states-of-america/1962/ ↩
[8] Engelhardt et al, Measles Mortality in the United States 1971-1975, AJPH Nov 1980, Vol. 70, No. 11 ↩
[9] Walter A. Orenstein, Robert T. Perry, Neal A. Halsey; The Clinical Significance of Measles: A Review, The Journal of Infectious Diseases, Volume 189, Issue Supplement_1, 1 May 2004, Pages S4–S16, https://doi.org/10.1086/377712 ↩
[10] Richard Moskowitz, M. D. – The Case Against Immunizations. Vaccine Impact. http://vaccineimpact.com/2015/richard-moskowitz-m-d-the-case-against-immunizations/. Published May 11, 2015. Accessed January 23, 2019.