Application of Machine Learning Models to Biomedical and Information System Signals From Critically Ill Adults

Background: Machine learning (ML)-derived notifications for impending episodes of hemodynamic instability and respiratory failure events are interesting because they can alert physicians in time to intervene before these complications occur. Research Question: Do ML alerts, telemedicine system (TS)-generated alerts, or biomedical monitors (BMs) have superior performance for predicting episodes of intubation or administration of vasopressors? Study Design and Methods: An ML algorithm was trained to predict intubation and vasopressor initiation events among critically ill adults. Its performance was compared with BM alarms and TS alerts. Results: ML notifications were substantially more accurate and precise, with 50-fold lower alarm burden than TS alerts for predicting vasopressor initiation and intubation events. ML notifications of internal validation cohorts demonstrated similar performance for independent academic medical center external validation and COVID-19 cohorts. Characteristics were also measured for a control group of recent patients that validated event detection methods and compared TS alert and BM alarm performance. The TS test characteristics were substantially better, with 10-fold less alarm burden than BM alarms. The accuracy of ML alerts (0.87-0.94) was in the range of other clinically actionable tests; the accuracy of TS (0.28-0.53) and BM (0.019-0.028) alerts were not. Overall test performance (F scores) for ML notifications were more than fivefold higher than for TS alerts, which were higher than those of BM alarms. Interpretation: ML-derived notifications for clinically actioned hemodynamic instability and respiratory failure events represent an advance because the magnitude of the differences of accuracy, precision, misclassification rate, and pre-event lead time is large enough to allow more proactive care and has markedly lower frequency and interruption of bedside physician work flows.