EMR-based prediction models developed and deployed in the HIV care continuum: a systematic review

EMR-based prediction models are now used in clinical decision support for HIV care. However, there is limited information about the methodological issues and clinical implications in the HIV care settings.
Published in Healthcare & Nursing
Like

Share this post

Choose a social network to share with, or copy the shortened URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

Read the paper

SpringerLink
SpringerLink SpringerLink

Electronic medical record-based prediction models developed and deployed in the HIV care continuum: a systematic review - Discover Health Systems

Objective To assess the methodological issues in prediction models developed using electronic medical records (EMR) and their early-stage clinical impact on the HIV care continuum. Methods A systematic search of entries in PubMed and Google Scholar was conducted between January 1, 2010, and January 17, 2022, to identify studies developing and deploying EMR-based prediction models. We used the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies), PROBAST (Prediction Model Risk of Bias Assessment Tool), and TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) statements to assess the methodological issues. In addition, we consulted reporting guidelines for early-stage clinical evaluation of decision support systems to assess the clinical impact of the models. Results The systematic search yielded 35 eligible articles: 24 (68.6%) aimed at model development and 11 (31.4%) for model deployment. The majority of these studies predicted an individual’s risk of carrying HIV (n = 12/35, 34.3%), the risk of interrupting HIV care (n = 9/35), and the risk of virological failure (n = 7/35). The methodological assessment for those 24 studies found that they were rated as high risk (n = 6/24), some concerns (n = 14/24), and a low risk of bias (n = 4/24). Several studies didn’t report the number of events (n = 14/24), missing data management (n = 12/24), inadequate reporting of statistical performance (n = 18/24), or lack of external validation (n = 21/24) in their model development processes. The early-stage clinical impact assessment for those 9/11-deployed models showed improved care outcomes, such as HIV screening, engagement in care, and viral load suppression. Conclusions EMR-based prediction models have been developed, and some are practically deployed as clinical decision support tools in the HIV care continuum. Overall, while early-stage clinical impact is observed with those deployed models, it is important to address methodological concerns and assess their potential clinical impact before widespread implementation. Systematic review registration: PROSPERO CRD42023454765.

EMR-Based Prediction Models in HIV Care

Introduction

The integration of electronic medical record (EMR)-based prediction models into HIV care is an exciting development poised to enhance clinical decision-making and patient outcomes. However, many studies in this area struggle with issues such as bias from missing data and a lack of external validation. Despite these challenges, the use of EMR-based models has shown promising improvements in the HIV care continuum. This review aims to evaluate the methodological robustness, applicability, and clinical implications of these models in HIV care.

Methods

We conducted a systematic review of studies published between January 1, 2010, and January 17, 2022, using PubMed and Google Scholar. The review adhered to CHARMS, PROBAST, and TRIPOD guidelines to assess methodological quality and consulted early-stage clinical evaluation guidelines for decision support systems to determine clinical impact.

Results and discussion

Over the past decade, 35 studies on EMR-based modeling in HIV care have been identified, with 24 focused on model development and 11 on deployment. Most studies (88.6%) originated from high-income countries, predominantly the United States.

Trends in Research

The number of studies employing EMR-based prediction models has steadily increased, reflecting the growing interest in their potential to support clinical decision-making. Between 2014 and 2017, 13 studies were published, compared to 10 between 2018 and 2021 and only 4 between 2010 and 2013. This could be due to the increased interest in prediction model development and its potential applications as clinical decision support, followed by the rise of EMRs in healthcare settings. The other argument could be narrated as the need for risk stratification. Since resources are getting scarce and a classical, “one-size-fits-all” intervention strategy is clinically inefficient, researchers are looking for a solution like predicting and stratifying patients into risk levels and tailoring intervention for those who may benefit most.

Outcomes in HIV Care

The reviewed studies focused on various outcomes within the HIV care continuum:

  • 34.3% predicted individual risk of carrying HIV
  • 25.7% predicted the risk of care interruption.
  • 20% predicted virological failure.
  • Other outcomes included phenotype prediction, new HIV diagnoses, clinical status, CD4 count changes, comorbidity burden, and risk of readmission and death.

Model Development
We evaluated 24 studies focused on model development and validation. Among these, 15 (62.5%) were diagnostic models designed to predict the current risk of event outcomes, and 9 (37.5%) were prognostic models aimed at predicting future event risks within the HIV care continuum. Regarding data sources, 14 (58.3%) studies used EMRs from multiple centers, while 10 (41.7%) relied on EMR data from a single site.
In terms of methodologies, 11 (45.8%) of the studies utilized machine learning algorithms, 9 (37.5%) employed generalized linear models like logistic regression and Cox regression, and 5 (20.8%) used chart review-based algorithm classification [Table 1]. The median sample size for model development was 2633 (interquartile range: 795, 138, 806), with 7 studies (29.2%) having a sample size of less than 1000. The median number of predictors was 20 (interquartile range: 12, 94), with 7 studies using 50 or more predictors and 4 studies using fewer than 10 predictors.

Model Validation and Performance Metrics
Model validation is a critical stage in predictive modeling. Most included studies performed internal validation: 12 (50%) used cross-validation techniques, 4 (16.7%) employed sample splitting, and 3 (12.5%) used bootstrapping. Notably, 4 studies (16.7%) used more than one validation technique [Table 1].
For performance evaluation, 14 (58.3%) studies used the c-statistic or area under the receiver operating characteristic curve (AUC) to measure discrimination. Additional measures included sensitivity (reported in 11 studies, 45.8%) and positive predictive value (PPV) (reported in 8 studies, 33.3%). Model calibration was assessed in 6 (25%) studies, with one study reporting both an F1 score and an AUC. However, none of the studies included decision curve analysis (DCA) to evaluate the clinical value and implications of the models.

Table 1|Model development methods, and model validation techniques (n=24).

Items

Number of studies (%)

Clinical context of  models 

 

Diagnostic

15(62.5)

 

Prognostic

9(37.5)

Model development methods

 

 

Traditional regression models

9(37.5)

Machine learning  models

10(41.7)

Chart review-based algorithm/classification

5(20.8)

Model selection strategy

 

 

Predictor selection before modeling

3(12.5)

Predictor selection during modeling

14(58.3)

Full model approach

7(29.2)

Model validation techniques

 

 

Cross-validation

12(50)

Split sample

4(16.7)

Bootstrapping Validation

3(12.5)

Multiple forms of validation

4(16.7)

Other forms (e.g. Z-score)

1(4.2)

 

 Methodological Pitfalls

Common methodological issues included insufficient reporting of outcome events, handling of missing data, and inadequate statistical performance metrics such as discrimination and calibration. These pitfalls hinder the generalizability and performance of prediction models in new settings or populations.

Early-Stage Clinical Impact

Despite methodological concerns, deployed EMR-based models have shown positive early-stage impacts. These include increased HIV screening rates, improved linkage to care, reduced care interruption, and enhanced patient engagement. For instance, one study reported a rise in monthly HIV screenings from 7 to 550 and increased linkage to care from 15% to 100%. Another noted a 10% reduction in loss to follow-up, a 3.8% improvement in clinical appointments, and a 15% increase in achieving viral suppression.

Conclusions

EMR-based prediction models are being developed and deployed to support clinical decision-making in HIV care. While early-stage impacts are promising, most studies face significant methodological challenges that need addressing. External validation in different settings is crucial to ensuring the models' robustness and reliability.

Recommendations and Policy Implications

To harness the full potential of EMR-based prediction models in HIV care:

  1. Address methodological concerns during model development to enhance predictive accuracy.
  2. Conduct external validations to test model performance in varied settings and populations.
  3. Evaluate the clinical consequences comprehensively before widespread implementation to ensure improved patient outcomes and informed clinical decisions.

By overcoming these challenges, EMR-based prediction models can become a cornerstone of personalized and efficient HIV care.

 

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Subscribe to the Topic

Health Informatics
Life Sciences > Health Sciences > Health Care > Health Informatics

Related Collections

With collections, you can get published faster and increase your visibility.

Conflict and Health in Africa

Armed conflict poses a major threat to health care system. Africa has been the hardest hit continent in armed conflict, fragile health care system, and poor population’s health outcomes. Armed conflict poses threat to entire health care system through attacks on health infrastructure and personnel, interrupting health service provision, leading to poor service delivery and health outcomes.

This Collection will bring together quantitative and qualitative studies, case reports and viewpoints on broad range of health topics, such as attacks on health infrastructure and health personnel, evaluation of the environmental, social, economic, and cultural impacts of war, civil unrest and conflicts, health systems’ preparedness, response, and resilience to address war/conflict-related inequities, wars/conflict, schooling and the impact on individuals/families and community functioning, the role of the United Nation peace operations to counter war/conflicts and civil unrest, infectious disease prevention and control, sexual and gender-based violence, reproductive and maternal health, mental health, non-communicable diseases, health systems, and ethics in conflicts and humanitarian crises.

Keywords: perilous medicine, conflict and health, armed conflicts and health, health crisis, health/social systems preparedness and responses, methodology in crisis, conflict and ethics, Sub-Saharan Africa

Publishing Model: Open Access

Deadline: Oct 31, 2024

(Trans)nationally mandated digital solutions in healthcare: Development, implementation, adoption and failure

The digital transformation of healthcare is a complex undertaking. Integration of diverse stakeholders, (trans-)national interdependencies and ethical considerations are among the challenges in the digitalization process. Various digital instruments, such as electronic health records, have been developed at national levels, especially with regard to the COVID-19 pandemic (e.g., tracing apps). However, health information technology projects often fail. Power relations, negotiation processes and collaborative meaning making are central challenges for successful implementation, which in turn may result into resistances and frictions. This is particularly relevant for (trans)nationally mandated digital solutions, where implementation most often is slow and subject to various resistances.

This call, therefore, invites researchers from around the globe to examine the development, implementation, and adoption processes of various (trans)nationally mandated digital solutions and their consequences on societal levels from an empirical, theoretical, or conceptual perspective. The focus lies on interdisciplinary assessments of the current situation or the socio-historical course to show a transnational picture through this collection.

The following questions are of interest to this call, but do not represent an exclusive or definitive list:

• Which visions and specific goals are addressed and achieved with (trans)nationally mandated digital solutions in healthcare?

• How are (trans)nationally mandated digital solutions in healthcare designed and implemented?

• How are these solutions embedded in overall digitalization strategies, theoretical assumptions, as well as in existing structures and processes?

• Which actors participated in the development process of such digital solutions? And what are their responsibilities during the phases of the implementation process?

• What role do negotiation processes play in the processes?

• What can be learned from the development, implementation, or failure of these innovation processes?

• What are factors of success and barriers for fast and widespread implementation?

• Which societal impact do these (trans)nationally mandated digital solutions have?

Keywords: digitalization, eHealth, telematics, digital infrastructure, innovation, implementation

Publishing Model: Open Access

Deadline: Jun 30, 2024