EMR-Based Prediction Models in HIV Care
Introduction
The integration of electronic medical record (EMR)-based prediction models into HIV care is an exciting development poised to enhance clinical decision-making and patient outcomes. However, many studies in this area struggle with issues such as bias from missing data and a lack of external validation. Despite these challenges, the use of EMR-based models has shown promising improvements in the HIV care continuum. This review aims to evaluate the methodological robustness, applicability, and clinical implications of these models in HIV care.
Methods
We conducted a systematic review of studies published between January 1, 2010, and January 17, 2022, using PubMed and Google Scholar. The review adhered to CHARMS, PROBAST, and TRIPOD guidelines to assess methodological quality and consulted early-stage clinical evaluation guidelines for decision support systems to determine clinical impact.
Results and discussion
Over the past decade, 35 studies on EMR-based modeling in HIV care have been identified, with 24 focused on model development and 11 on deployment. Most studies (88.6%) originated from high-income countries, predominantly the United States.
Trends in Research
The number of studies employing EMR-based prediction models has steadily increased, reflecting the growing interest in their potential to support clinical decision-making. Between 2014 and 2017, 13 studies were published, compared to 10 between 2018 and 2021 and only 4 between 2010 and 2013. This could be due to the increased interest in prediction model development and its potential applications as clinical decision support, followed by the rise of EMRs in healthcare settings. The other argument could be narrated as the need for risk stratification. Since resources are getting scarce and a classical, “one-size-fits-all” intervention strategy is clinically inefficient, researchers are looking for a solution like predicting and stratifying patients into risk levels and tailoring intervention for those who may benefit most.
Outcomes in HIV Care
The reviewed studies focused on various outcomes within the HIV care continuum:
- 34.3% predicted individual risk of carrying HIV
- 25.7% predicted the risk of care interruption.
- 20% predicted virological failure.
- Other outcomes included phenotype prediction, new HIV diagnoses, clinical status, CD4 count changes, comorbidity burden, and risk of readmission and death.
Model Development
We evaluated 24 studies focused on model development and validation. Among these, 15 (62.5%) were diagnostic models designed to predict the current risk of event outcomes, and 9 (37.5%) were prognostic models aimed at predicting future event risks within the HIV care continuum. Regarding data sources, 14 (58.3%) studies used EMRs from multiple centers, while 10 (41.7%) relied on EMR data from a single site.
In terms of methodologies, 11 (45.8%) of the studies utilized machine learning algorithms, 9 (37.5%) employed generalized linear models like logistic regression and Cox regression, and 5 (20.8%) used chart review-based algorithm classification [Table 1]. The median sample size for model development was 2633 (interquartile range: 795, 138, 806), with 7 studies (29.2%) having a sample size of less than 1000. The median number of predictors was 20 (interquartile range: 12, 94), with 7 studies using 50 or more predictors and 4 studies using fewer than 10 predictors.
Model Validation and Performance Metrics
Model validation is a critical stage in predictive modeling. Most included studies performed internal validation: 12 (50%) used cross-validation techniques, 4 (16.7%) employed sample splitting, and 3 (12.5%) used bootstrapping. Notably, 4 studies (16.7%) used more than one validation technique [Table 1].
For performance evaluation, 14 (58.3%) studies used the c-statistic or area under the receiver operating characteristic curve (AUC) to measure discrimination. Additional measures included sensitivity (reported in 11 studies, 45.8%) and positive predictive value (PPV) (reported in 8 studies, 33.3%). Model calibration was assessed in 6 (25%) studies, with one study reporting both an F1 score and an AUC. However, none of the studies included decision curve analysis (DCA) to evaluate the clinical value and implications of the models.
Table 1|Model development methods, and model validation techniques (n=24).
|
Items |
Number of studies (%) |
|
|
Clinical context of models |
||
|
|
Diagnostic |
15(62.5) |
|
|
Prognostic |
9(37.5) |
|
Model development methods |
|
|
|
|
Traditional regression models |
9(37.5) |
|
Machine learning models |
10(41.7) |
|
|
Chart review-based algorithm/classification |
5(20.8) |
|
|
Model selection strategy |
|
|
|
|
Predictor selection before modeling |
3(12.5) |
|
Predictor selection during modeling |
14(58.3) |
|
|
Full model approach |
7(29.2) |
|
|
Model validation techniques |
|
|
|
|
Cross-validation |
12(50) |
|
Split sample |
4(16.7) |
|
|
Bootstrapping Validation |
3(12.5) |
|
|
Multiple forms of validation |
4(16.7) |
|
|
Other forms (e.g. Z-score) |
1(4.2) |
|
Methodological Pitfalls
Common methodological issues included insufficient reporting of outcome events, handling of missing data, and inadequate statistical performance metrics such as discrimination and calibration. These pitfalls hinder the generalizability and performance of prediction models in new settings or populations.
Early-Stage Clinical Impact
Despite methodological concerns, deployed EMR-based models have shown positive early-stage impacts. These include increased HIV screening rates, improved linkage to care, reduced care interruption, and enhanced patient engagement. For instance, one study reported a rise in monthly HIV screenings from 7 to 550 and increased linkage to care from 15% to 100%. Another noted a 10% reduction in loss to follow-up, a 3.8% improvement in clinical appointments, and a 15% increase in achieving viral suppression.
Conclusions
EMR-based prediction models are being developed and deployed to support clinical decision-making in HIV care. While early-stage impacts are promising, most studies face significant methodological challenges that need addressing. External validation in different settings is crucial to ensuring the models' robustness and reliability.
Recommendations and Policy Implications
To harness the full potential of EMR-based prediction models in HIV care:
- Address methodological concerns during model development to enhance predictive accuracy.
- Conduct external validations to test model performance in varied settings and populations.
- Evaluate the clinical consequences comprehensively before widespread implementation to ensure improved patient outcomes and informed clinical decisions.
By overcoming these challenges, EMR-based prediction models can become a cornerstone of personalized and efficient HIV care.
Please sign in or register for FREE
If you are a registered user on Research Communities by Springer Nature, please sign in