Drug administration errors are a leading cause of preventable patient harm with 5-10% of all drugs being given in error. Adverse events associated with these errors are estimated to impact 1.2 million people annually with 5.1 billion dollars in associated costs. These errors have been shown to occur across the spectrum of clinical care including anesthesiology, pediatrics, the emergency department, medical wards, and the intensive care unit.

Syringe and vial swap errors often occur when a clinician must transfer a medication from a vial into a syringe. Vial swaps occur when the wrong vial is selected or a syringe is mislabeled, syringe swaps occur when the drug is labeled correctly but administered in error. These two errors comprise about 40% of medication errors.

While today’s operating rooms include safety measures such as barcode scanning of drug labels, a busy or stressed clinician may forget to perform this extra step. In a survey across 109 anesthesiology providers, we find that the median percentage of time providers scan or manually record the drug label is only 20%. Furthermore, providers believe that only 68% of manually recorded medical records accurately reflect drug name, dose, and administration time. 

AI-enabled wearable camera system attached to head strap and recording drug preparation events.

In this work, we create the first AI-enabled wearable camera system that can automatically detect medication errors using deep learning algorithms. Our system is designed to detect syringes and vials in a provider’s hand, classify the drug type on the label and check if they match in order to detect vial swap errors. Such an automated system can serve as a second set of ‘eyes’ to verify that medication errors have not occurred and provide a visual or auditory warning before the wrong drug is given to a patient. As clinicians in operating rooms often wear protective eyewear, this system could potentially be integrated into smart eyewear similar to augmented reality glasses that can visually flag medication errors before they occur.

Vial swap error during a drug preparation event. Ondansetron is incorrectly drawn up into a syringe with a rocuronium label. 

Detecting these errors are challenging as text on the drug labels are small and can be obscured by the clinician’s fingers which can move while preparing the drug. Furthermore, real world drug preparation events occur against the busy backdrop of the operating room which can also contain syringes and vials, vary in lighting condition, and whether the clinician is wearing gloves. 

To tackle these challenges, we trained a neural network on a large annotated dataset of 4K videos across 13 anesthesiology providers, 2 hospitals and 17 operating rooms over 55 days. This dataset comprised a variety of different lighting conditions and backgrounds. This dataset was further augmented using image transformations to improve model performance. As drug labels were often obscured, instead of reading the text from the drug label, we designed the neural network to pick up on visual cues: vial and syringe size and shape, vial cap color, label print size, and classify the drug label. 

Real-world operating room environments are busy with syringes and vials in the background. Clinicians’ fingers can obscure the label, and move quickly during a drug drawup.

The system was evaluated on 418 drug drawup events and achieved 99.6% sensitivity and 98.8% specificity at detecting vial swap errors. To evaluate the generalization capabilities of our model, the system was tested on data collected from an unseen hospital and obtained similar performance at classifying drug labels. In our survey across anesthesiology providers, the majority desired the system to be at least 95% accurate, which has been achieved by the system. 

Confusion matrix demonstrating system performance across 418 drug drawup events.

To estimate if our system could alert a clinician in time before drug delivery, we measured the mean time between when a clinician had selected a syringe to when it was delivered to a patient. Our analysis of 212 drug delivery events show that this time period is on average 9.9 ± 7.2 s (with a minimum time of 2 s). The inference time of our deep learning system took less than 25 ms on a NVIDIA GPU which could be integrated as part of an edge server in an operating room environment. 

This work demonstrates the potential for AI-based camera systems to improve patient safety practices across a variety of healthcare practices. Integration of the system with an electronic medical record also opens up opportunities for automatic documentation of drug information and can reduce the overhead of manual record-keeping.

Check out our npj Digital Medicine paper for more details.