Bias, precision and timeliness of historical (background) rate comparison methods for vaccine safety monitoring: an empirical multi-database analysis
Bias, precision and timeliness of historical (background) rate comparison methods for vaccine safety monitoring: an empirical multi-database analysis
ABSTRACT Using real-world data and past vaccination data, we conducted a large-scale experiment to quantify bias, precision and timeliness of different study designs to estimate historical background (expected) compared to post-vaccination (observed) rates of safety events for several vaccines. We used negative (not causally related) and positive control outcomes. The latter were synthetically generated true safety signals with incident rate ratios ranging from 1.5 to 4. Observed vs. expected analysis using within-database historical background rates is a sensitive but unspecific method for the identification of potential vaccine safety signals. Despite good discrimination, most analyses showed a tendency to overestimate risks, with 20%-100% type 1 error, but low (0% to 20%) type 2 error in the large databases included in our study. Efforts to improve the comparability of background and post-vaccine rates, including age-sex adjustment and anchoring background rates around a visit, reduced type 1 error and improved precision but residual systematic error persisted. Additionally, empirical calibration dramatically reduced type 1 to nominal but came at the cost of increasing type 2 error. Our study found that within-database background rate comparison is a sensitive but unspecific method to identify vaccine safety signals. The method is positively biased, with low (<=20%) type 2 error, and 20% to 100% of negative control outcomes were incorrectly identified as safety signals due to type 1 error. Age-sex adjustment and anchoring background rate estimates around a healthcare visit are useful strategies to reduce false positives, with little impact on type 2 error. Sufficient sensitivity was reached for the identification of safety signals by month 1-2 for vaccines with quick uptake (e.g., seasonal influenza), but much later (up to month 9) for vaccines with slower uptake (e.g., varicella-zoster or papillomavirus). Finally, we reported that empirical calibration using negative control outcomes reduces type 1 error to nominal at the cost of increasing type 2 error.
Suchard Marc A、Casajust Paula、Lai Lana YH、Ostropolets Anna、Arshad Faaizah、Areia Carlos、Prieto-Alhambra Daniel、Minty Evan P、Schuemie Martijn J、Li Xintong、Ryan Patrick B、Pratt Nicole、Alshammari Thamir M、Hripcsak George、Tan Eng Hooi、Duarte-Salles Talita
Department of Biostatistics, University of California||Observational Health Data Sciences and InformaticsReal-World Evidence, Trial Form SupportSchool of Medical Sciences, University of ManchesterDepartment of Biomedical Informatics, Columbia UniversityDepartment of Biostatistics, University of CaliforniaNuffield Department of Clinical Neurosciences, University of OxfordCentre for Statistics in Medicine, NDORMS, University of Oxford||Health Data Sciences, Medical Informatics, Erasmus Medical Center UniversityO?ˉBrien Institute for Public Health, Faculty of Medicine, University of CalgaryDepartment of Biostatistics, University of California||Observational Health Data Sciences and Informatics||Observational Health Data Analytics, Janssen R&DCentre for Statistics in Medicine, NDORMS, University of OxfordObservational Health Data Sciences and Informatics||Observational Health Data Analytics, Janssen R&DQuality Use of Medicines and Pharmacy Research Centre, University of South AustraliaCollege of Pharmacy, Riyadh Elm UniversityDepartment of Biomedical Informatics, Columbia University||Medical Informatics Services, NewYork-Presbyterian HospitalCentre for Statistics in Medicine, NDORMS, University of OxfordInstitut Universitari d?ˉInvestigaci¨? en Atenci¨? Prim¨¤ria Jordi Gol (IDIAPJGol)
医学研究方法预防医学
vaccine safetybackground ratesincidenceadverse eventsurveillance
Suchard Marc A,Casajust Paula,Lai Lana YH,Ostropolets Anna,Arshad Faaizah,Areia Carlos,Prieto-Alhambra Daniel,Minty Evan P,Schuemie Martijn J,Li Xintong,Ryan Patrick B,Pratt Nicole,Alshammari Thamir M,Hripcsak George,Tan Eng Hooi,Duarte-Salles Talita.Bias, precision and timeliness of historical (background) rate comparison methods for vaccine safety monitoring: an empirical multi-database analysis[EB/OL].(2025-03-28)[2025-07-16].https://www.medrxiv.org/content/10.1101/2021.07.10.21258463.点此复制
评论