If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
A multicentre validation study of the deep learning-based early warning score for predicting in-hospital cardiac arrest in patients admitted to general wards
Department of Critical Care Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of KoreaDivision of Pulmonary and Critical Care Medicine, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
Corresponding author at: Department of Emergency Medicine, Seoul National University Bundang Hospital, 82, Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do 13620, Republic of Korea.
Department of Emergency Medicine, Seoul National University Bundang Hospital, Gyeonggi-do, Republic of KoreaDepartment of Emergency Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea
The recently developed deep learning (DL)-based early warning score (DEWS) has shown potential in predicting deteriorating patients. We aimed to validate DEWS in multiple centres and compare the prediction, alarming and timeliness performance with the modified early warning score (MEWS) to identify patients at risk for in-hospital cardiac arrest (IHCA).
Method/research design
This retrospective cohort study included adult patients admitted to the general wards of five hospitals during a 12-month period. The occurrence of IHCA within 24 h of vital sign observation was the outcome of interest. We assessed the discrimination using the area under the receiver operating characteristic curve (AUROC).
Results
The study population consists of 173,368 patients (224 IHCAs). The predictive performance of DEWS was superior to that of MEWS in both the internal (AUROC: 0.860 vs. 0.754, respectively) and external (AUROC: 0.905 vs. 0.785, respectively) validation cohorts. At the same specificity, DEWS had a higher sensitivity than MEWS, and at the same sensitivity, DEWS reduced the mean alarm count by nearly half of MEWS. Additionally, DEWS was able to predict more IHCA patients in the 24–0.5 h before the outcome, and DEWS was reasonably calibrated.
Conclusion
Our study showed that DEWS was superior to MEWS in three key aspects (IHCA predictive, alarming, and timeliness performance). This study demonstrates the potential of DEWS as an effective, efficient screening tool in rapid response systems (RRSs) to identify high-risk patients.
A rapid response system (RRS) is a strategy for preventing cardiac arrest (CA) or deterioration in the general ward by providing immediate and efficient interventions by monitoring patients’ conditions.
To effectively identify these at-risk patients, several early warning scores (EWSs) have been developed. Because of the limited RRS resources, an ideal EWS should have high specificity and sensitivity, ensuring the correct identification of the at-risk patients while avoiding excessive alarm, which can increase RRS staff desensitization and decrease quality of care.
Worthing physiological scoring system: derivation and validation of a physiological early-warning system for medical admissions. An observational, population-based single-centre study.
The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
In 2018, a DL-based early warning score, called DEWS, was developed which considers only 4 basic vital signs: systolic blood pressure (SBP), heart rate (HR), respiratory rate (RR), and body temperature (BT).
We have extended from this version of DEWS by adding diastolic blood pressure (DBP), age, and the recorded time of each vital sign.
DEWS measures the risk of CA within 24 h from vital sign observation. DEWS showed potential in predicting in-hospital CA (IHCA) by showing higher sensitivity, and a lower false alarm rate than MEWS in the original development.
The original study was performed in 2 hospitals with approximately 300 beds each; one was a cardiovascular-specific hospital, and the other was a community general hospital. Therefore, we aimed to validate DEWS in a large multicentre cohort and compare the IHCA predictive performance of DEWS with that of MEWS.
Methods
Study population and design
A retrospective cohort study was performed in 5 hospitals located in South Korea: Mediplex Sejong Hospital (323 beds), Sejong Hospital (301 beds), Inha University Hospital (925 beds), Seoul National University Bundang Hospital (1324 beds) and Samsung Medical Center (1989 beds). The two hospitals (A: Sejong Hospital and B: Mediplex Sejong Hospital), where the original DEWS was developed,
were included for internal validation, and the other three hospitals (C: Inha University Hospital, D: Seoul National University Bundang Hospital, and E: Samsung Medical Center) were included for external validation. All hospitals had a mature RRS except hospital A. The structure of the RRS in each hospital are described in supplement Table 1.
The study population included adult patients (≥18 years old) admitted to the general ward over a 12-month period. We excluded patients with data recorded less than 30 min of admission duration, no vital signs measured 24 h before the CA event, and erroneous patient demographics. The specific details on the participant selection process is reported in supplemental Fig. 1. Since there exists no established method in determining sample size for prognostic models using DL methods, we have chosen the sample size appropriate for our experiments.
The primary outcome of interest was IHCA (defined as lack of a palpable pulse with attempted resuscitation). All vital signs used to predict the outcome of interest were collected for every patient. As the vital signs were measured multiple times per patient, the DEWS and MEWS were calculated at each point of measurement. Finally, performances of DEWS and MEWS were compared by the predictive performance of IHCA within 24 h of vital sign measurement.
Data collection and preprocessing
We collected data including age, sex, occurrence of events, time and location of event occurrences, and five time-stamped vital signs (SBP, DBP, HR, RR, and BT) recorded during hospitalization of the patients abstracted from the electronic medical records (EMRs). From the initial data collected, erroneous values with extreme deviations from the vital specific normal ranges and non-numeric values were treated as missing values. The missing values were imputed to the most recent previous value and the missing rates of each variable are presented in supplemental Table 2.
Deep learning-based early warning system
The DEWS architecture includes three long short-term memory (LSTM) layers and three fully connected (FC) layers with the rectified linear unit. To reflect the trend of the vital signs for each patient, 20 consecutive series of vital signs are used as inputs of the LSTM layers.
To address the class imbalance problem, we adjusted the ratio of nonevent/event data in the training process by duplicating the event data. From the original DEWS model, we have extended the model by adding DBP along with age and the recorded time of each vital sign.
Performance evaluation and statistical analysis
We have compared the performance of DEWS and MEWS in terms of the following three main key questions:
•
Key question 1: How accurate is DEWS in terms of predicting IHCA compared with MEWS (predictive performance)?
The predictive performance was measured by comparing the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC).
The AUROC is one of the most commonly used metrics and represents the area under the sensitivity-false positive rate curve. Compared with the AUROC, the AUPRC accounts for the class imbalance in data by measuring the area under the plot of the precision-sensitivity curve. Additionally, we compared DEWS with MEWS in terms of the positive predictive value (PPV = true positive/(true positive + false positive)), the negative predictive value (NPV = true negative/(true negative + false negative)), F measure (2 × (precision × recall)/(precision + recall)), the net reclassification index (NRI), the mean alarm count per day per 1000 beds (MACPD), and the number needed to examine (NNE) at the same specificity as MEWS.
The study concept is demonstrated graphically in supplemental Fig. 2.
•
Key question 2: Does DEWS produce a lower false alarm rate than MEWS with the same sensitivity level (alarming performance)?
The alarm rate is an important criterion for validating the feasibility upon implementation of EWS because excessive false alarms can cause alarm fatigue.
Excessive false alarms and alarm fatigue can result to staff desensitization and missed responses to alerts of clinical significance, putting patient safety and quality of care at substantial risk.
Therefore, an ideal EWS should have high sensitivity and a low false alarm rate, so we compared the alarming rate of DEWS and MEWS using the MACPD at the same sensitivity level.
•
Key question 3: Does DEWS predict IHCA earlier than MEWS at the same specificity level (timeliness performance)?
It is already well known that a delayed RRS response is associated with a poor patient outcome.
Recently, in the study “Quality metrics for the evaluation of RRS” defined predictable IHCA as CAs occurring in hospitalized ward patients who met the hospital's escalation threshold at least 30 min prior to and within 24 h of the event.
However, compared to intensive care units (ICUs) where vital signs are measured continuously, vital signs are usually measured only 3∼4 times a day (every 6 or 8 h) in the general wards. Therefore, it is important that the RRS staff are aware of the at-risk patients as early as possible so that they can prepare in advance and perform suitable action in enough time before the event. In this respect, we compared the cumulative prediction percentage of IHCA at the same time point within 24 h of the event (supplemental Fig. 2).
We assessed the calibration of DEWS using a calibration plot and the average absolute error between the actual outcome and the estimated probabilities.
The x-axis of the calibration plot is the means of decile-binned predictions, and the y-axis is the means of the observed outcomes in each bin so that well calibrated model will fall close to the diagonal. Additionally, to mitigate the black-box prediction problem, we applied a Shapley additive explanations (SHAP) algorithm to our prediction model to obtain interpretability of the features that drive predictions.
SHAP is a game theoretic approach designed to explain the output of a machine learning model where the influence of each feature on a prediction is described using Shapley values.
Ethics statement
The Institutional Review Board of each hospital approved the study protocol and waived the requirement of informed consent because of the retrospective study design. The IRB number of each participating hospital is as follows: B-1806-477-002 (Seoul National University Bundang Hospital), 2018-054 (Mediplex Sejong Hospital), 2018-0689 (Sejong General Hospital), 2019-09-001-000 (Inha University Hospital), and SMC-2019-09-129 (Samsung Medical Center).
Results
Baseline characteristics
During the study period of 12 months, 173,368 patients from the five hospitals was examined. The internal validation cohort contained 14,365 patients with 23 IHCAs, and the external validation cohort contained 159,003 patients with 201 IHCAs. The incidence rate of IHCAs in the overall cohort was 1.29 per 1000 admissions. We plotted the DEWS and MEWS distributions in the IHCA cases (supplemental Fig. 3) using the average DEWS or MEWS within 24 h of the event. Among the event cases, only 19 cases had a MEWS greater than five points, which was quite a low number. When the number of event cases is compared, more cases are distributed at higher score ranges for DEWS than for MEWS, especially in the external validation cohort. The baseline characteristics of the overall cohort are depicted in Table 1.
As shown in Fig. 1, the performance of DEWS for predicting IHCA was superior to that of MEWS in both the internal (AUROC: 0.860 vs. 0.754, respectively) and external (AUROC: 0.905 vs. 0.785, respectively) validation cohorts. Additionally, the AUPRC for DEWS was higher than that of MEWS in both the internal (0.012 vs. 0.003, respectively) and external (0.017 vs. 0.005, respectively) validation cohorts. We validated MEWS at the most commonly used cut-off scores of 3, 4, 5, and 6 in terms of the sensitivity, specificity, PPV, F measure, NPV, NNE, NRI and compared these values to those of DEWS at the same specificity.
Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions.
As shown in Table 2, DEWS achieved higher sensitivity for all the cut-off scores and achieved at most 228.3% and 63.2% higher sensitivity than MEWS in the internal validation and external validation cohorts, respectively. The predictive performance of each hospital is shown in supplemental Fig. 4, and DEWS outperformed MEWS in each of five hospitals.
Fig. 1Performance of the early warning scores for predicting in-hospital cardiac arrest. DEWS indicates the deep learning-based early warning score, MEWS indicates the modified early warning score, AUROC indicates the area under the receiver operating characteristic curve, AUPRC indicates the area under precision-recall curve, and CI indicates the confidence interval. P-value was calculated using Delong test. Power was calculated according to the formula by Obuchowski and McClish, 1997.
Table 2Comparison of accuracy of in-hospital cardiac arrest prediction model with same specificity point.
Characteristics
Sensitivity
Specificity
PPV
NPV
F-measure
NRI
MACPD
NNE
Internal validation cohort
MEWS ≥ 3
0.484
0.932
0.0011
1
0.002
104
391
DEWS ≥ 53.1
0.548
0.932
0.0029
1
0.006
0.0011
103
342
MEWS ≥ 4
0.419
0.953
0.0032
1
0.006
71
308
DEWS ≥ 60.5
0.484
0.953
0.0037
1
0.007
0.0015
71
269
MEWS ≥ 5
0.234
0.992
0.0106
1
0.007
12
94
DEWS ≥ 87.5
0.306
0.992
0.0136
1
0.026
0.0032
12
73
External validation cohort
MEWS ≥ 3
0.551
0.908
0.0033
1
0.007
335
302
DEWS ≥ 69.9
0.700
0.908
0.0042
1
0.008
0.0014
334
236
MEWS ≥ 4
0.386
0.958
0.0050
1
0.010
154
191
DEWS ≥ 83.2
0.560
0.958
0.0073
1
0.032
0.0024
154
137
MEWS ≥ 5
0.230
0.989
0.0117
1
0.022
39
85
DEWS ≥ 94.1
0.338
0.989
0.0166
1
0.032
0.0052
41
60
PPV positive predictive value, NPV negative predictive value, NRI net reclassification improvement, MACPD mean alarm count per day per 1000 beds, NNE number needed to examine, MEWS modified early warning score, DEWS deep learning-based early warning score.
We compared DEWS and MEWS by the MACPD at the same sensitivity level. As shown in Fig. 2, DEWS achieved a lower MACPD than MEWS. This result indicates that DEWS can detect the same number of deteriorating patients with a much lower false alarm rate than MEWS. For example, at MEWS cut-offs of 3, DEWS produced 62.5% and 44.2% fewer alarms than MEWS in the internal and external validation cohorts, respectively.
Fig. 2Comparison of the mean alarm count per day per 1000 beds at the same sensitivity point for predicting in-hospital cardiac arrest. MACPD indicates the mean alarm count per day per 1000 beds, DEWS indicates the deep learning-based early warning score, and MEWS indicates the modified early warning score.
We validated DEWS and MEWS by enrolling IHCA patients at the time point where the early warning score first triggered the alarm from 24 to 0.5 h before the CA occurred. As shown in Fig. 3, DEWS detected more patients with CA in this period than MEWS. Especially in the external validation cohort, DEWS detected 10 and 20 more IHCA patients 20 and 15 h before the event, respectively. This finding indicates that DEWS can not only predict more IHCA patients within 24 h but can also detect more patients in advance and thus save time for the medical team to effectively manage patients at risk.
Fig. 3Comparison of the cumulative percentages of cardiac arrest patients. DEWS indicates the deep learning-based early warning score, and MEWS indicates the modified early warning score.
We assessed the calibration of DEWS on the entire cohort. As shown in supplemental Fig. 5, DEWS was reasonably calibrated where the curve approaches close to the diagonal. Quantitatively, the average absolute error between the outcome and the estimated probabilities was 0.046, indicating that the prediction scores and the absolute risk are close to perfect concordance.
Inspection of model features
In supplemental Fig. 6, the overall importance of the predictor variables of DEWS shows HR as the most important feature. The second most important feature was RR, but in the case of other features, it was found that the importance was relatively low. Additionally, the feature importance of DEWS according to the order of consecutive time steps shows a rapid increase in the SHAP value at the most recent time point.
Discussion
We evaluated the ability of DEWS in predicting IHCA in general ward-admitted patients of a large multicentre cohort. The results of all three key questions (predictive performance of IHCA, alarming performance, timeliness performance) were superior for DEWS compared to those of MEWS. In both cohorts, DEWS achieved better performance in predicting IHCA within 24 h of vital sign observation than MEWS: DEWS achieved 14.0% (300%) and 15.2% (240%) higher AUROCs (AUPRCs) than MEWS, respectively. The number of alarms is an important issue for RRS teams because they are eventually associated with the team's workload. In this study, the alarming rate of DEWS was 44.2% of that of MEWS for a cut-off score of 3, 37.0% of that of MEWS for a cut-off score of 4, and of 48.7% that of MEWS for a cut-off score of 5 in the external validation cohort. In summary, DEWS had nearly half of the alarming rate of MEWS. The third key question was the timeliness of the prediction. When examined for every time point from 24 h to 30 min before the event, DEWS detected more IHCA cases than MEWS. As result of such an advantage, it enables RRSs to evaluate and care for deteriorating patients with more time to respond. Therefore, better predictions with fewer alarms and earlier predictions indicate that DEWS has the potential to be an effective alternative screening tool than conventional early warning systems.
Various studies have attempted to predict mortality in critically ill patients (i.e., those in ICUs) using machine learning (ML).
ICUs, in particular, have many databases for continuous vital sign monitoring and large numbers of diagnostic tests, including laboratory tests, imaging tests, microbiologic reports, medical history panels, patient demographics, ordered fluids, drugs, transfusions, etc. This large database enables ICUs to be an adequate setting for which to conduct artificial intelligence (AI)-based studies. Most AI-based ICU studies have studied mortality or major event prediction (such as hypotension, sepsis, readmission), and in general, algorithm-based prediction achieved better performances than conventional prognostic systems.
showed that a ML (i.e., random forest) algorithm (AUROC 0.80) predicted clinical deterioration more accurately than MEWS (AUROC 0.70) in general ward patients. Both ML and DL methods analyse data through self-learning to solve the task or problem. ML requires feature engineering, whereas DL does not; rather, it learns the representation of the raw data in multiple levels of abstractions by itself, which is the essence of why DL methods achieve higher accuracy than most ML methods.
However, this study did not focus on general ward patients and sudden CA but rather on the entire length of stay, including both the general ward and the ICU. The outcomes of interest were inpatient mortality, readmission, length of stay and discharge diagnoses. Thus, to the best of our knowledge, our study is the first to apply DL to detect deteriorating patients in general wards in a large multicentre cohort.
One of the critical strengths of DEWS is that it consists of a limited number of predictor variables. In this validation study, DEWS used only five basic vital signs: SBP, DBP, HR, RR and BT along with age and recorded time of vital signs. The two previous AI-based studies
in general ward patients used a variety of predictor variables, including demographics, vital signs, laboratory values, etc. Prediction models with more variables would have better predictability, but there are significant limitations to the scalability and applicability of models with many variables. The predictor variables used in DEWS are basic but essential vital signs that are almost always checked in admitted patients and with low missing rates. Therefore, DEWS can be applied worldwide without any difficulties in technical implementation.
Five hospitals in South Korea participated in this validation study. The characteristics of each hospital are quite different in terms of the locations, hospital sizes, admitted patients and operating policies. The two hospitals involved in the internal validation have more than 300 beds; one is a cardiovascular-specific hospital, and the other is a community general hospital. The hospitals in the external validation set have more than 900 beds, and all three hospitals are tertiary teaching hospitals, which are affiliated with each of the three different medical universities. Since the original DL model was developed and trained from the two hospitals with 300 beds, the results on the external validation cohort are important in terms of generalization. As a result, DEWS achieved superior performance in the external validation cohort (AUROC 0.905, 95% CI [0.901 - 0.910]) compared to the internal validation cohort (AUROC 0.860, 95% CI [0.832–0.888]), which suggests that DEWS is robust across multiple hospitals.
Our study has several limitations. We consider only the first CA for each patient admission, although second and third CAs are also important for the patient's prognosis. Nonetheless, the first CA has the highest priority because the care level the patient receives after CA will be maximal. Additionally, since this study was performed in a retrospective manner, a well-designed prospective clinical trial is necessary to apply DEWS in clinical practices as an alternative to other triggering score systems in RRS.
Conclusion
We compared DEWS and MEWS in multiple centres via extensive experiments. The results showed that DEWS not only predicts IHCA more accurately than MEWS but also reduces the false alarm rate. Additionally, DEWS was able to predict more CA patients in the period from 24 h to 0.5 h before the event than MEWS. These findings demonstrate the potential of DEWS as an effective screening tool in RRSs that can be efficiently applied to identify high-risk patients.
Conflict of interest
All authors have reported that no potential conflicts of interest exist with any companies/organizations whose products or services may be discussed in this article.
Funding
None.
Authors’ contribution
YHJ and YJL had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: YJL, KJC, OK, HP, YL, JMK, JP, JSK, MJL, AJK, REK, KJ, YHC. Acquisition, analysis, or interpretation of data: KJC, OK, YJL, YHJ. Drafting of the manuscript: YJL, KJC. Critical revision of the manuscript for important intellectual content: YJL, KJC, OK, HP, YL, JMK, JP, JSK, MJL, AJK, REK, KJ, YHC. Statistical analysis: KJC, OK, HP, YL. Administrative, technical, or material support: YJL, JMK, JP, JSK, MJL, AJK, REK, KJ. Supervision: OK, HP, YL, JMK, JP, JSK, MJL, AJK, REK, KJ, YHC. Image analysis: YJL, KJC, OK, HP.
Appendix A. Supplementary data
The following are Supplementary data to this article:
Worthing physiological scoring system: derivation and validation of a physiological early-warning system for medical admissions. An observational, population-based single-centre study.
The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions.
00154-4&id=gr1.jpg){kind=link}
00154-4&id=gr2.jpg){kind=link}
00154-4&id=gr3.jpg){kind=link}
00154-4&id=mmc3.jpg){kind=link}
00154-4&id=mmc4.jpg){kind=link}
00154-4&id=mmc5.jpg){kind=link}
00154-4&id=mmc6.jpg){kind=link}
00154-4&id=mmc7.jpg){kind=link}
00154-4&id=mmc8.jpg){kind=link}