Over- and under-estimation of vaccine effectiveness

• Published in: BMC medical research methodology Vol. 25; no. 1; pp. 163 - 10
• Main Authors: De-Leon, Hilla, Aran, Dvir
• Format: Journal Article
• Language: English
• Published: England BioMed Central Ltd 01.07.2025; BioMed Central; BMC
• Subjects: Agent-based modeling; Clinical trials; COVID-19; COVID-19 - epidemiology; COVID-19 - prevention & control; COVID-19 - transmission; COVID-19 vaccines; COVID-19 Vaccines - immunology; Disease transmission; Effectiveness; Estimates; Humans; Indirect protection; Infections; Influence; Interference; Israel - epidemiology; Population; Population-based studies; SARS-CoV-2 - immunology; Severe acute respiratory syndrome coronavirus 2; Vaccination - statistics & numerical data; Vaccine effectiveness; Vaccine Efficacy - statistics & numerical data; Israel; COVID-19; Agent-based modeling; Indirect protection; Population-based studies; Interference; Vaccine effectiveness
• DOI: 10.1186/s12874-025-02611-4
• ISSN: 1471-2288

The effectiveness of SARS-CoV-2 vaccines against infection has been a subject of debate, with varying results reported in different studies, ranging from 60-95% vaccine effectiveness (VE). This range is striking when comparing two studies conducted in Israel at the same time, as one study reported VE of 90-95%, while the other study reported only ~ 80%. We argue that this variability is due to inadequate accounting for indirect protection provided by vaccines, which can block further transmission of the virus. We developed a novel analytic heterogenous infection model and extended our agent-based model of disease spread to allow for heterogenous interactions between vaccinated and unvaccinated across close-contacts and regions. We applied these models on real-world regional data from Israel from early 2021 to estimate VE using two common study designs: population-based and secondary infections. Our results show that the estimated VE of a vaccine with efficacy of 85% can range from 70-95% depending on the interactions between vaccinated and unvaccinated individuals. Since different study designs capture different levels of interactions, we suggest that this interference explains the variability across studies. Finally, we propose a methodology for more accurate estimation without knowledge of interactions. DISCUSSIONS AND CONCLUSIONS: Our study highlights the importance of considering indirect protection when estimating vaccine effectiveness, explains how different study designs may report biased estimations, and propose a method to overcome this bias. We hope that our models will lead to more accurate understanding of the impact of vaccinations and inform public health policy. Not applicable.