Measly Timing
In A Tug of War Between Incidence and Mortality, I looked at how to best measure prevalence of a disease from an epidemiological perspective in the case of Measles. I was asking the question, should we look at incidence or mortality statistics? Because if you look at mortality rates, they decline steadily even prior to and leading up to vaccination policies. But if you look at incidence rates, they are quite flat even for decades prior to 1963 and then they drop quite precipitously after the vaccine licensure; it appears to be quite an effective policy.
I pointed out some potential pitfalls to looking at incidence, which is that the quality of reporting, and that the completeness-of-reporting, a measure of how many actual cases actually get reported to data-collecting institutions like the CDC varies tremendously, sometimes even by a few thousand percent, which casts a lot of doubt about looking at incidence rates. Even so, the wide range of fluctuations were still around a level that was an order of magnitude higher prior to 1963 than it was after, which I took as evidence that something more was going on than simply the reporting fashions of the day.
In any case, it is actually very difficult to answer definitively, and my ultimate impression is that we should look at both, and interestingly the ratio between incidence and mortality tells you something about the declining virulence of the disease because of maternal (natural) immunity, sanitation, nutrition, and medical advancements.
We deliberately avoided actual conclusions about vaccine effectiveness, which is a topic to which we now turn. We’ll be looking at a particular vaccine for a particular country, and then revisit effectiveness for other viruses and other countries over time and over various blog-posts.
I’ll be using mortality statistics for this exercise to show that, in spite of the pre-existing declining mortality rates before vaccines, effective vaccine policies can be measured in a statistically meaningful way.
Mortality Statistics
As we saw, the anti-vaccine community often cites data and charts like the following to justify the inefficacy of vaccines. It shows an exponentially decreasing rate in Measles-related mortality. And it looks convincingly like vaccines did nothing to eliminate the disease, although we will find that there is more to this story because the effect is swamped by the scale of previous decades, and that doesn't mean that vaccines are useless or that we shouldn't be taking them. All that it means is that it would be disingenuous to quote mortality levels in 1900 and then attribute the entire drop to the licensure of vaccines, which definitely had very little to do with the overall drop in infectious diseases in the 20th Century; there were a lot of other very important and powerful factors that cannot be underestimated.
Incidence Statistics
Pro-vaccine individuals argue that one should use incidence data (and ignore quality issues) because medical advancements also caused a decline in deaths. The dramatic reduction in incidence following vaccine licensure is shown on a log-scale below.
Level vs. Trend
Both the CDC and anti-vaccine groups are overly focused on levels of their preferred respective data-sets (incidence or mortality). Crucially, however, the main instrument for measuring vaccine efficacy should not be the levels of death-rates, but changes in already declining trends. Below, I take the log-transform of raw mortality-rate data, which allows us to zoom in on the vaccine-licensed era and it also gives us a tractable set of linear models. I compared the slope from 1920 to 1962 with the slope from 1963 to 1978 (after which rate-data become meaningless due to small sample-sizes).
This wasn't a piece-wise linear model; I just compared the slope of two separately computed regressions spanning different time periods and the differences were very statistically significant (-0.04 and -0.11 with a t-stat of 5) .
You can read more about this hypothesis testing here.
This tells us that vaccines played a very strong role in speeding up the decline in Measles-related mortality rates. Visually, the change in slope is clear. The two light-blue lines show upper and lower confidence intervals using a Poisson distribution. Again, the red line is the Measles Vaccine coverage rate.
I used post-1920 data because of possible consequences of World War I, which seem to play a significant role in muting the declining trend in measles mortality rates, and had I incorporated the pre-1920 data, the statistical significance would only have been stronger.
Conclusion
The case for vaccines is looking pretty good here. 408 people died from Measles in 1962 and 24 people died in 1968, and so this represents a reduction of 384 people per year. Given the trajectory before 1963, it would have taken until 2010, instead of 1989, to completely wipe out measles deaths and the difference between actual, vaccine-mediated deaths and that trajectory suggests that a total of about 840 lives were saved from 1963 to 2010. You could argue that 840 lives over that time-span is only 18 people per year and is relatively insignificant compared to a large population that has a lot of other pressing problems. Or you could argue that only a callous monster would not want those nearly 1000 people to be alive and well today.
Going back further however, about 8 in 100,000 people died from Measles in 1920. And in 1968, 5 years after the vaccine coverage rates soared, that number was brought down to 0.02 (people out of 100,000), representing a 99.8% drop. In terms of levels, only 1 percent of that is attributable to the vaccine because the rate was 0.22 in 1962 and it would have been 0.09 in 1968 if it continued to follow its annual trend.
The small impact of medical interventions to overall mortality statistics is scrutinized in The Questionable Contribution of Medical Measures to the Decline of Mortality Statistics in the United States. This paper puts an upper bound of 3.5% contribution to medical measures in curbing infectious diseases [1]. Another paper estimates that clean water was the cause of 2/3rds of declines in child mortality [2].
Sanitation, medical advancements, and education played the predominant role in wiping out measles deaths and the vaccine made a minuscule dent in levels (so small we could not see it at all without taking the log-transformation of the raw data), but it made a dramatic dent in trends and was also helpful in limiting incidence, morbidity, and hospitalization.
It is disappointing that the vaccine was introduced late, but the timing of the vaccine introduction is a separate issue from vaccine efficacy, and it is good to be clear-thinking about those differences. Bad timing does not equal bad vaccine.
Again, these conclusions apply to a particular vaccine in a particular country and they do not incorporate an evaluation of the risks of measles-vaccine administration. I have, f.ex., not found the same efficacy for the whooping cough vaccine. There are many other considerations and here we have evaluated the benefits portion of a cost-benefit analysis, leaving the rest for later.
References
[1]: Cutler DM, Miller G. The Role of Public Health Improvements in Health Advances: The Twentieth-Century United States. Demography. 2005;42(1):1-22. doi:10.1353/dem.2005.0002. https://link.springer.com/article/10.1353/dem.2005.0002
[2]: Kinlay, et al.; The Questionable Contribution of Medical Measures to the Decline of Mortality in the United States in the Twentieth Century; Health and Society 1977. http://www.columbia.edu/itc/hs/pubhealth/rosner/g8965/client_edit/readings/week_2/mckinlay.pdf#page=4