If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author at: Department of Department of Intensive Care, Emergency Medicine and Anaesthesiology - Fondazione Policlinico Universitario A. Gemelli, IRCCS, Università Cattolica del Sacro Cuore – Roma, Largo Francesco Vito, 1, 00168 Rome, Italy.
Department of Intensive Care, Emergency Medicine and Anaesthesiology, Fondazione Policlinico Universitario A. Gemelli-IRCCS, Rome, ItalyInstitute of Anaesthesiology and Intensive Care Medicine, Università Cattolica del Sacro Cuore, Rome, Italy
Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, ItalyDepartment of Internal Medicine and Gastroenterology, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
Department of Intensive Care, Emergency Medicine and Anaesthesiology, Fondazione Policlinico Universitario A. Gemelli-IRCCS, Rome, ItalyDepartment of Internal Medicine and Gastroenterology, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
Department of Internal Medicine and Gastroenterology, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, ItalyInstitute of Internal Medicine and Gastroenterology, Università Cattolica del Sacro Cuore, Rome, Italy
To identify the most accurate early warning score (EWS) for predicting an adverse outcome in COVID-19 patients admitted to the emergency department (ED).
Methods
In adult consecutive patients admitted (March 1-April 15, 2020) to the ED of a major referral centre for COVID-19, we retrospectively calculated NEWS, NEWS2, NEWS-C, MEWS, qSOFA, and REMS from physiological variables measured on arrival. Sensitivity, specificity, positive (PPV) and negative predictive value (NPV), and the area under the receiver operating characteristic (AUROC) curve of each EWS for predicting admission to the intensive care unit (ICU) and death at 48 h and 7 days were calculated.
Results
We included 334 patients (119 [35.6%] females, median age 66 [54-78] years). At 7 days, the rates of ICU admission and death were 56/334 (17%) and 26/334 (7.8%), respectively. NEWS was the most accurate predictor of ICU admission within 7 days (AUROC 0.783 [95% CI, 0.735-0.826]; sensitivity 71.4 [57.8-82.7]%; NPV 93.1 [89.8-95.3]%), while REMS was the most accurate predictor of death within 7 days (AUROC 0.823 [0.778–0.863]; sensitivity 96.1 [80.4-99.9]%; NPV 99.4[96.2–99.9]%). Similar results were observed for ICU admission and death at 48 h. NEWS and REMS were as accurate as the triage system used in our ED. MEWS and qSOFA had the lowest overall accuracy for both outcomes.
Conclusion
In our single-centre cohort of COVID-19 patients, NEWS and REMS measured on ED arrival were the most sensitive predictors of 7-day ICU admission or death. EWS could be useful to identify patients with low risk of clinical deterioration.
After the first cases were identified in Wuhan (China) in December 2019, the novel coronavirus SARS-CoV-2 has caused a pandemic of respiratory illness named COVID-19.
Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases From the Chinese Center for Disease Control and Prevention.
In hospitalized COVID-19 patients, the prevalence of hypoxemic respiratory failure is around 20%, and more than 25% of them may require intensive care treatment.
The increasing number of COVID-19 cases has challenged healthcare systems worldwide. In this context of overwhelming demand for medical assessment and triage in emergency departments (EDs), early warning scores (EWS) may be useful. EWS are physiological scoring systems based on a rapid and quantitative assessment of changes in vital signs,
and were initially developed to identify and track hospitalised patients at risk of deterioration outside critical care areas in order to ensure early stabilisation and transfer to intensive care unit (ICU) when appropriate, and to prevent avoidable cardiac arrests.
Can binary early warning scores perform as well as standard early warning scores for discriminating a patient’s risk of cardiac arrest, death or unanticipated intensive care unit admission?.
suggested using a score greater than 6 of a modified version of NEWS (hereafter referred as to NEWS-C) for deciding admission to ICU of COVID-19 patients.
A Risk Classification Model to Predict Mortality Among Laboratory-Confirmed Avian Influenza A H7N9 Patients: A Population-Based Observational Cohort Study.
A Comparison of the Quick Sequential (Sepsis-Related) Organ Failure Assessment Score and the National Early Warning Score in Non-ICU Patients With/Without Infection.
and it has not been validated so far. At present, it is not clear which EWS is the most accurate for triaging COVID-19 patients in the ED.
The aim of the present study was to compare the ability of EWS assessed on ED arrival to identify adult COVID-19 patients at risk for ICU admission or death within seven days. The EWS assessed in this study included Modified EWS (MEWS),
and REMS. Moreover, in order to measure the added value of EWS to the conventional ED triage process, we also compared the accuracy of EWS with that of the Regional Triage System for Infectious Diseases in use in our hospital for COVID-19 patients.
Methods
Study design and setting
This is a retrospective observational study (clinicaltrials.gov NCT04371562) conducted in the ED of the Fondazione Policlinico Universitario Agostino Gemelli, in Rome. Policlinico Gemelli is the largest teaching hospital in Rome and a referral centre for COVID-19 in central Italy. Our ED has a catchment area of 1,8 M inhabitants, with an average of 80k access/year.
The Regional Triage System for Infectious Diseases in our ED is based on a protocol issued by the Health Service of Lazio Region, where our hospital is located. The protocol, summarised on ESM Tables 1a-b, follows an ABCDE scheme (airway, breathing, circulation, disability, exposure) with additional variables such as age, blood glucose level, plus additional symptoms and signs. The triage system assigns a priority value from 1 to 5, in descending order of severity. Triage is performed by a nurse certified in a full time 7-day course followed by a 6-month supervised training.
The criteria for ICU admission of COVID-19 patients in our hospital during the study period included circulatory shock requiring vasopressors and the need for positive pressure respiratory support, including non-invasive ventilation. High-flow oxygen treatment is provided in general wards. ICU admission criteria were consistent throughout the study period. At present, no EWS is in use in our hospital.
Eligibility criteria
From March 1 to April 15, 2020 we included all adult (>18y) patients assessed in our ED for suspected COVID-19 whose diagnosis was confirmed with real-time reverse-transcriptase–polymerase-chain-reaction assay of nasal and pharyngeal swab specimens. Testing for COVID-19 was conducted according to the World Health Organization (WHO) interim guidance.
We excluded pregnant women and patients who were on circulatory shock and positive pressure support on arrival to ED. Patients with a do-not-resuscitate/do-not-intubate (DNR/DNI) order in place were also excluded.
Data collection and measurements
Two authors blinded to patients’ outcome retrieved clinical information from the electronic medical records (EMR) of the ED database. Extracted data included age, sex, clinical history and presentation, temperature, heart rate (HR), respiratory rate (RR), blood pressure (BP), Glasgow Coma Scale (GCS) score, oxygen therapy, and peripheral oxygen saturation (SpO2). All clinical signs, including SpO2, were measured upon ED arrival. EWS were calculated for each patient using Microsoft Excel 2010 for Windows (Microsoft Inc., Redmond, WA, USA) from available physiological measurements. Variables included in each of these scores are listed on ESM Table 2. For NEWS2 calculation, patients were considered at risk of type 2 respiratory failure (T2RF) if they had a confirmed history of chronic obstructive pulmonary disease.
Outcomes
The primary study outcomes were death and ICU admission within seven days from ED arrival. Secondary outcomes were death and ICU admission within 48 h from ED arrival. Outcomes were retrospectively assessed by reviewing EMRs of the hospital medical database.
Statistical analysis
Continuous variables were reported as median [interquartile range, IQR] and were compared with univariate analysis by Mann-Whitney U test. Categorical variables were reported as absolute number (percentage) and are compared by Chi-square test (with Fisher’s test if appropriate).
Receiver operating characteristic (ROC) curve analysis was used to estimate the performance of the evaluated scores in predicting the study outcomes. For each score threshold value, the following was calculated: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (+LR), and negative likelihood ratio (-LR). The Youden index was used to estimate the optimal thresholds for sensitivity and specificity. The comparison between the areas under the ROC curve (AUROCs) was made according to the DeLong method.
The results of imputation were added to the original study cohort with the complete dataset of vital parameters and EWS were recalculated. The resulting AUROCs were compared to the AUROC obtained in the study population by using the z-statistics (2 sided). Further data on imputation are available in Appendix 1.
Data were analysed using IBM SPSS statistics for Windows, Version 25 (IBM Corp. Armonk, NY, USA) and MedCalc Statistical Software version 19.2.1 (MedCalc Software Ltd, Ostend, Belgium; https://www.medcalc.org; 2020).
Ethical approval
This study has been approved by the local Ethics Committee (authorization #001705520) and has been performed in accordance with the ethical standards established in the 1964 Declaration of Helsinki and its later amendments. All patients gave their informed consent for clinical data analysis.
Results
Characteristics of the study subjects
During the study period, 3955 patients were seen in the ED, of whom 1801 were evaluated for suspect COVID-19 and 512 were found positive to the COVID-19 test. Of these, 336 had sufficient data on EMR for calculating EWS. Two patients with a DNR/DNI order in place were excluded, leaving 334 patients for final analysis (Fig. 1). Patients’ median [IQR] age was 66 [54 – 78] years (range 20 -98 years) and 215 (64.4%) were male (Table 1).
Fig. 1Flow-chart of study selection. DNR/DNI: Do-not-resuscitate/Do-not-intubate order. EMR: Electronic Medical Record; EWS: Early Warning Score.
Table 1Characteristics of the study population and distribution of score values according to patient outcome. Data for ICU admission and death are referred to day 7. Data are presented as median (25th – 75th percentile) or number (percentage).
All Patients (n° 334)
Admitted to ICU
P
Survived (308 [90.9%])
Dead (26 [7.8%])
p
No (278 [83.2%])
Yes (56 [16.8%])
Age, years
66 [54–78]
63 [51–77]
71.5 [64–78]
0.002
64 [53–77]
79.5 [74–86]
<0.001
Sex, male
215 (64.4%)
173 (62.2%)
42 (75.0%)
0.069
196 (63.6%)
19 (73.1%)
0.334
Heart rate, beats · min-1
90 [79–105]
88 [78–103]
98 [86–105]
0.058
90 [79–104]
95 [81–113]
0.265
Respiratory rate, breaths · min-1
18 [16–21]
18 [16–21]
20 [17–21]
0.027
18 [16–21]
20 [17–23]
0.068
Glasgow Coma Score
15 [15–15]
15 [15–15]
15 [15–15]
0.793
15 [15–15]
15 [15–15]
0.002
Blood pressure, mmHg
Systolic
127 [114–140]
130 [119–138]
129 [117–140]
0.838
127 [114–144]
128 [115–139]
0.728
Diastolic
80 [70–89]
80 [70–89]
77 [70–84]
0.064
80 [70–88]
73 [64–89]
0.089
Mean
96 [86–103]
96 [86–104]
93 [84–100]
0.351
94 [85–103]
92 [84–101]
0.268
SpO2, %
95 [92–97]
95 [93–97]
88.4 [81–92]
<0.001
95 [92–97]
89 [79–92]
<0.001
Body Temperature, °C
37.0 [36.7–37.5]
37.0 [36.7–37.5]
37.1 [36.9–37.4]
0.213
36.5 [36.0–37.2]
37.1 [37.0–37.5]
0.614
Comorbidities
Hypertension
140 (41.9%)
107 (38.5%)
33 (58.9%)
0.005
123 (39.9%)
17 (65.4%)
0.012
CAD, CHF
81 (24.3%)
57 (20.5%)
24 (42.9%)
<0.001
70 (22.7%)
11 (42.3%)
0.025
Diabetes
36 (10.7%)
30 (10.8%)
6 (10.7%)
1.000
34 (11.0%)
2 (7.7%)
0.597
COPD
19 (5.7%)
14 (5.0%)
5 (8.9%)
0.251
16 (5.2%)
3 (11.5%)
0.180
Dementia
12 (3.6%)
12 (4.3%)
0
0.113
9 (2.9%)
3 (11.5%)
0.023
Malignancy
14 (4.2%)
12 (4.3%)
2 (3.6%)
0.800
12 (3.9%)
2 (7.7%)
0.354
Chronic Kidney disease
11 (3.3%)
9 (3.2%)
2 (3.6%)
0.898
8 (2.6%)
3 (11.5%)
0.046
Severe Obesity
6 (1.8%)
3 (1.1%)
3 (5.4%)
0.028
6 (1.9%)
0
1.000
Living in an institution
29 (8.6%)
27 (9.6%)
2 (3.6%)
0.140
26 (8.4%)
3 (11.5%)
0.483
TRIAGE code
3 [3–4]
4 [3–4]
2 [1–3]
<0.001
3 [3–4]
2 [1–3]
<0.001
Early Warning Scores
MEWS
5 [4–7]
5 [4–7]
6 [5–7]
0.004
5 [4–7]
6 [4–8]
0.136
NEWS
3 [1–5]
2 [1–4]
6 [4–8]
<0.001
3 [1–5]
6 [3–9]
<0.001
NEWS2
2 [1–5]
2 [1–4]
6 [3–8]
<0.001
2 [1–5]
6 [2–9]
<0.001
NEWS – C
6 [4–9]
6 [4–8]
9 [6–11]
<0.001
6 [4–8]
10 [6–12]
<0.001
qSOFA
0 [0–0]
0 [0–0]
0 [0–1]
0.066
0 [0–0]
1 [0–1]
<0.001
REMS
5 [3–7]
5 [3–6]
7 [5–9]
<0.001
5 [3–7]
8 [6–11]
<0.001
Outcome
Admitted to ICU at ≤7 days
56 (16.7%)
-
-
38 (12.3%)
18 (69.2%)
<0.001
Deceased at ≤7 Days
26 (7.8%)
8 (2.9%)
18 (32.1%)
<0.001
-
-
Abbreviations: CAD Coronary Artery Disease; CHF Congestive Heart Failure; COPD Chronic Obstructive Pulmonary Disease; ICU Intensive Care Unit. TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modified NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score.
Thirty-eight (11%) COVID-19 patients were discharged from ED and quarantined. These patients were contacted daily to assess their clinical condition. The remaining 296 patients were admitted to hospital, of whom 56 (17% of the total) were admitted to ICU. Fifty patients were admitted to ICU within 48 hours, while the remaining six patients were admitted on days 3-7 after ED arrival. Patients who needed ICU admission were older, had lower SpO2 and higher respiratory rate on ED arrival, and more frequently reported a history of hypertension, obesity, and chronic heart failure or coronary artery disease (Table 1).
Overall, 26 patients (7.8%) deceased within seven days after ED arrival, among whom 11 (3.3%) deceased by 48 hours. These patients were significantly older, had a higher respiratory rate, and a lower SpO2 on ED arrival. As for ICU admission, in patients who died a history of hypertension and chronic heart failure or coronary artery disease was more commonly reported (Table 1).
Accuracy of warning scores
Concerning ICU admission, among all the EWS we assessed, NEWS had the highest overall accuracy both at 48 h and at 7 days (AUROC 0.802 [0.756 – 0.844] and 0.783 [0.735 – 0.826], respectively), and the highest NPV, both at 48 h and at 7 days (93.4 [90.6 – 95.4]% for NEWS ≤ 5 and 93.1 [89.8 – 95.3]% for NEWS ≤ 4, respectively) along with NEWS2 (NPV 93.7 [90.6 – 95.8] and 92.4 [89.3 – 94.7] at 48 h and 7 days, respectively). The accuracy was lower, although not significantly, than that of the triage system of our hospital (Table 2).
Table 2Sensitivity, specificity, negative and positive predictive value, and positive and negative likelihood ratio of the hospital triage score and of the EWS for predicting ICU admission within 48 hours and 7 days from ED admission. Optimal cut off values for sensitivity and specificity were chosen according to Youden index. Values of the EWS with the largest AUROC are in bold.
AUROC
cut-off
Sensitivity (%)
Specificity (%)
+LR
-LR
PPV
NPV
Admission to ICU within 48 hours
TRIAGE
0.839 [0.795 – 0.877]
≤ 2
66.0 [51.2 – 78.8]
89.8 [85.7 – 93.1]
6.5 [4.3 – 9.6]
0.4 [0.3 – 0.6]
53.2 [43.3 – 62.9]
93.7 [91.1 – 95.7]
MEWS
0.652 [0.598 – 0.703]
> 5
70.0 [55.4 – 82.1]
64.8 [58.9 – 70.3]
2.0 [1.6 – 2.5]
0.5 [0.3 – 0.7]
25.9 [21.6 – 30.8]
92.5 [88.8 – 95.0]
NEWS
0.802 [0.756 – 0.844]
> 5
66.0 [51.2 – 78.8]
84.9 [80.2 – 88.8]
4.4 [3.1 – 6.1]
0.4 [0.3 – 0.6]
43.4 [35.3 – 51.9]
93.4 [90.6 – 95.4]
NEWS2
0.780 [0.731 – 0.823]
> 4
70.0 [55.4 – 82.1]
78.5 [73.3 – 83.2]
3.3 [2.4 – 4.3]
0.4 [0.2 – 0.6]
36.5 [30.1 – 43.3]
93.7 [90.6 – 95.8]
NEWS-C
0.771 [0.701 – 0.831]
> 8
62.9 [44.9 – 78.5]
82.7 [75.4 – 88.6]
3.6 [2.3 – 5.7]
0.5 [0.3 – 0.7]
47.8 [37.0 – 58.8]
89.8 [85.1 – 93.2]
qSOFA
0.566 [0.511 – 0.620]
> 0
34.0 [21.2 – 48.8]
79.6 [74.4 – 84.1]
1.7 [1.1 – 2.6]
0.8 [0.7 – 1.0]
22.7 [15.8 – 31.5]
87.3 [84.8 – 89.4]
REMS
0.742 [0.692 – 0.788]
> 6
60.0 [45.2 – 73.6]
75.0 [69.5 – 79.9]
2.4 [1.8 – 3.2]
0.5 [0.4 – 0.8]
29.7 [23.8 – 36.4]
91.4 [88.3 – 93.8]
Admission to ICU within 7 days
TRIAGE
0.818 [0.773 – 0.858]
≤ 2
58.9 [45.0 – 71.9]
89.6 [85.4 – 92.9]
5.6 [3.8 – 8.5]
0.5 [0.3 – 0.6]
53.2 [43.1 – 63.1]
91.5 [88.8 – 93.7]
MEWS
0.618 [0.564 – 0.671]
> 5
66.1 [52.2 – 78.2]
64.8 [58.8 – 70.4]
1.9 [1.5 – 2.4]
0.5 [0.4 – 0.8]
27.4 [22.8 – 32.6]
90.5 [86.7 – 93.2]
NEWS
0.783 [0.735 – 0.826]
> 4
71.4 [57.8 – 82.7]
77.3 [72.1 – 82.1]
3.2 [2.4 – 4.1]
0.4 [0.2 – 0.6]
38.8 [32.6 – 45.5]
93.1 [89.8 – 95.3]
NEWS2
0.762 [0.714 – 0.807]
> 4
67.8 [54.0 – 79.7]
79.1 [73.9 – 83.8]
3.2 [2.4 – 4.4]
0.4 [0.3 – 0.6]
39.6 [32.9 – 46.7]
92.4 [89.3 – 94.7]
NEWS-C
0.755 [0.684 – 0.817]
> 8
60.1 [51.9 – 81.9]
84.2 [76.9 – 90.0]
3.9 [2.4 – 6.1]
0.5 [0.3 – 0.7]
54.3 [42.8 – 65.4]
87.5 [82.6 – 91.2]
qSOFA
0.557 [0.501 – 0.611]
> 0
32.1 [20.3 – 46.0]
79.5 [74.3 – 84.1]
1.6 [1.0 – 2.4]
0.8 [0.7 – 1.0]
24.0 [16.8 – 33.0]
85.3 [82.8 – 87.5]
REMS
0.735 [0.685 – 0.782]
> 6
58.9 [45.0 – 71.9]
75.5 [70.0 – 80.5]
2.4 [1.8 – 3.3]
0.5 [0.4 – 0.7]
32.7 [26.4 – 39.6]
90.1 [86.9 – 92.6]
Abbreviations: TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modi0fied NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score; +LR Positive Likelihood Ratio; −LR Negative Likelihood Ratio; PPV Positive Predictive Value; NPV Negative Predictive Value.
Concerning in-hospital death, REMS showed the highest overall accuracy both at 48 h and 7 days (AUROC 0.882 [0.843 – 0.915] and 0.823 [0.778 – 0.863], respectively). These AUROCs were larger, although not significantly, than those of the triage system of our hospital at both time points. REMS had also the highest NPV for in-hospital death (99.6 [97.3 – 99.9]% for REMS > 6 at 48 h and 99.4 [96.2 – 99.9]% for REMS > 5 at 7 days; Table 3).
Table 3Sensitivity, specificity, negative and positive predictive value, and positive and negative likelihood ratio of the hospital triage score and of the EWS for predicting death within 48 hours and 7 days from ED admission. Optimal cut off values for sensitivity and specificity were chosen according to Youden index. Values of the EWS with the largest AUROC are in bold.
AUROC
cut-off
Sensitivity (%)
Specificity (%)
+LR
-LR
PPV
NPV
Death within 48 hours
TRIAGE
0.849 [0.806 – 0.886]
≤ 2
81.8 [48.2 – 97.7]
83.6 [79.1 – 87.5]
5.0 [4.1 – 15.7]
0.5 [0.3 – 0.9]
14.5 [10.5 – 19.8]
99.3 [97.5 – 99.8]
MEWS
0.630 [0.576 – 0.682]
> 5
63.6 [30.8 – 89.1]
60.4 [54.8 – 65.7]
1.6 [1.0 – 2.6]
0.6 [0.3 – 1.3]
5.2 [3.3 – 8.0]
98.0 [95.7 – 99.1]
NEWS
0.829 [0.742 – 0.832]
>4
81.8 [48.2 – 97.7]
70.9 [65.6 – 75.8]
2.8 [2.0 – 3.9]
0.3 [0.1 – 0.9]
8.7 [6.5 – 11.7]
99.1 [97.0 – 99.8]
NEWS2
0.753 [0.703 – 0.798]
> 4
72.7 [39.0 – 94.0]
72.7 [67.6 – 77.5]
2.7 [1.8 – 4.0]
0.4 [0.1 – 1.0]
8.3 [5.7 – 12.0]
98.7 [96.8 – 99.5]
NEWS-C
0.773 [0.703 – 0.833]
> 9
70.0 [34.8 – 93.3]
82.3 [75.6 – 87.8]
4.0 [2.3 – 6.7]
0.4 [0.1 – 0.9]
19.4 [12.5 – 28.9]
97.8 [94.6 – 99.1]
qSOFA
0.808 [0.762 – 0.849]
> 0
81.8 [48.2 – 97.7]
79.6 [74.8 – 83.8]
4.0 [2.8 – 5.7]
0.2 [0.1 – 0.8]
12.0 [8.8 – 16.2]
99.2 [97.3 – 99.8]
REMS
0.882 [0.843 – 0.915]
> 6
90.9 [58.7 – 99.8]
71.8 [66.6 – 76.7]
3.2 [2.5 – 4.2]
0.1 [0.1 – 0.8]
9.9 [7.8 – 12.4]
99.6 [97.3 – 99.9]
Death within 7 days
TRIAGE
0.819 [0.773 – 0.858]
≤ 2
61.5 [40.6 – 79.8]
85.1 [80.6 – 88.9]
4.1 [2.8 – 6.2]
0.5 [0.3 – 0.7]
25.8 [18.8 – 34.3]
94.4 [92.8 – 95.7]
MEWS
0.586 [0.531 – 0.640]
> 5
57.7 [36.9 – 76.6]
61.0 [55.3 – 66.5]
1.5 [1.0 – 2.1]
0.7 [0.4 – 1.1]
11.1 [8.0 – 15.2]
94.5 [91.5 – 96.4]
NEWS
0.768 [0.719 – 0.812]
> 5
65.4 [44.3 – 82.8]
80.8 [76.0 – 85.1]
3.4 [2.4 – 4.9]
0.4 [0.3 – 0.7]
22.4 [16.7 – 29.3]
96.5 [94.2 – 97.9]
NEWS2
0.727 [0.676 – 0.774]
> 5
61.5 [40.6 – 79.8]
81.5 [76.7 – 85.7]
3.3 [2.3 – 4.9]
0.5 [0.3 – 0.8]
17.7 [13.3 – 23.2]
96.2 [93.9 – 97.6]
NEWS-C
0.767 [0.697 – 0.828]
> 8
65.2 [42.7 – 83.6]
79.5 [72.1 – 85.6]
3.1 [2.1 – 4.9]
0.4 [0.2 – 0.8]
32.6 [23.9 – 42.7]
93.7 [89.5 – 96.4]
qSOFA
0.673 [0.620 – 0.723]
> 0
53.8 [33.4 – 73.3]
80.2 [75.3 – 84.5]
2.7 [1.8 – 4.1]
0.6 [0.4 – 0.9]
18.7 [13.1 – 25.9]
95.4 [93.1 – 96.9]
REMS
0.823 [0.778 – 0.863]
> 5
96.1 [80.4 – 99.9]
56.8 [51.1 – 62.4]
2.3 [1.9 – 2.6]
0.1 [0.0 – 0.5]
15.8 [13.9 – 17.9]
99.4 [96.2 – 99.9]
Abbreviations: TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modified NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score; +LR Positive Likelihood Ratio; −LR Negative Likelihood Ratio; PPV Positive Predictive Value; NPV Negative Predictive Value.
For both outcomes, the overall accuracy of the NEWS-C was consistently lower than that of NEWS in our population. Among all EWS, MEWS and qSOFA showed the lowest overall accuracy. Their AUROCs were significantly lower than that of the triage system of our hospital, except for prediction of death at 48 h for qSOFA (Table 3).
ESM figures 1 and 2 include the ROC curves for prediction of ICU admission and death, respectively, along with a statistical comparison of the AUROCs of the scores included in the study.
Among the 176 patients excluded because of missing parameters, 78 (44.3%) had less than three missing parameters and were eligible for imputation. Their available parameters are shown in Appendix 1, Table A1. Among these 78 patients, six deaths (7.7%) and nine ICU admissions (11.5%) occurred within 7 days.
The EWS AUROCs in the extended population including 78 patients with imputed vital parameters (total 412 patients) were not significantly different from those of the 334 patients with complete original data. (Appendix 1, Tables A2 and A3).
Discussion
Our study showed that in COVID-19 patients NEWS and REMS assessed on arrival at the emergency department were the most accurate scores for predicting the risk of ICU admission and death, respectively, both at 48 h and at 7 days. Moreover, the accuracy of these scores was comparable with that of the comprehensive triage score used in our department.
NEWS is among the most accurate tools for predicting patient deterioration outside ICU.
The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
NEWS was more strongly associated with ICU admission or death compared to 33 other scoring systems (not including REMS). Moreover, in the specific setting of acute care patients admitted for suspected infection in ED or hospital wards, NEWS has proven to be more accurate for predicting death or ICU transfer than other tools that were purposely designed to assess infected patients, such as qSOFA and systemic inflammatory response syndrome (SIRS) criteria.
Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit.
even if the added value of this modification in improving discrimination of adverse outcomes in patients with documented T2RF has not been demonstrated yet.
A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study.
In our study, NEWS2 did not perform better than NEWS in predicting ICU admission or death. However, the prevalence of T2RF patients in our population was very low (19/334 patients; 5.7%). This is in accordance with data from other studies showing that hypercapnia is uncommon in COVID-19 patients, even in those admitted to ICU,
Age ≥65 years was added to NEWS-C as an independent risk factor based on recent reports in patients with influenza or other non-COVID infectious diseases.
A Risk Classification Model to Predict Mortality Among Laboratory-Confirmed Avian Influenza A H7N9 Patients: A Population-Based Observational Cohort Study.
A Comparison of the Quick Sequential (Sepsis-Related) Organ Failure Assessment Score and the National Early Warning Score in Non-ICU Patients With/Without Infection.
However, NEWS-C did not perform better than NEWS in predicting ICU admission or mortality in our study. This result can be explained considering that our population was relatively old. In fact, the median age of survivors vs. non-survivors was 64 years vs. 79.5 years in our COVID-19 population, as compared with 51 vs.68 years respectively in a recent report from China.
United Nations Department of Economic and Social Affairs Population Dynamics. World Population Prospects 2019. Accessed May 24, 2020 at https://population.un.org/wpp/.
) and in policies for ICU admission and triage. When weighting for age, REMS may be particularly appropriate, since – unlike other EWS – it stratifies age (five groups from <45 to >74 years; see also ESM Table 2). In our COVID-19 population, REMS was the most accurate score for predicting mortality. Among the scoring systems we evaluated, REMS was the only one specifically developed to predict mortality in ED patients.
The qSOFA has been proposed as a prompt to consider possible sepsis outside the ICU. Criteria to calculate qSOFA include altered mental status (GCS < 15), tachypnoea (RR ≥ 22), and hypotension (systolic BP ≤ 100). In the original validation study, a qSOFA score of 2 or higher was associated with a significantly increased risk of hospital mortality.
Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit.
Notably, most patients in our cohort did not have any of the three qSOFA criteria, and none of them had a qSOFA score of 2 or higher. Among the three qSOFA criteria, the prevalence of hypotension and altered mental status on ED admission in our patient population was particularly low (6.6% and 3.3%; data not shown). This is in line with recent reports showing that extrapulmonary organ dysfunction occurs later in the clinical course of COVID-19.
When compared with other scoring systems, including our institutional triage score, NEWS and REMS had the highest sensitivity and NPV for adverse outcomes. In other words, they performed particularly well in identifying COVID-19 patients who did not require ICU admission or were at low risk of death, respectively. When triaging patients in ED, having a high NPV (i.e., a low rate of false negative predictions) for adverse outcomes is paramount in order to avoid inappropriately admitting a patient at risk of deterioration to a non-critical care area. However, because of the relatively low incidence of adverse outcomes in our population, all the EWS we evaluated had a low PPV (i.e., a low rate of true positive predictions), which implies a risk of overtriage when these scores are used to decide admission to a critical care area. Although consensus suggests that overtriage is generally preferable to undertriage,
ICU Admission, Discharge, and Triage Guidelines: A Framework to Enhance Clinical Operations, Development of Institutional Policies, and Further Research.
this may not be the case when meeting the demands of healthcare resources is difficult. The prognostic indications from physiological scores should be interpreted within a more comprehensive clinical approach including patient’s comorbidities, functional status, and preferences.
In our study we assessed EWS only once, on patient arrival to the ED. Since EWS have been designed as track-and-trigger systems to detect deterioration of vital signs over time, their sequential measurement in COVID-19 patients admitted to non-monitored wards may represent the logical continuation of their use in ED. The value of EWS in this context deserves further investigation.
Some limitations of our study should be acknowledged. First, this was a single-centre study conducted on a relatively small population, and its confirmation on a larger, multicentre cohort is warranted. Secondly, the EWS were retrospectively calculated from data recorded on the patients’ EMR that were not specifically collected for this purpose. This resulted in missing data necessitating imputation. Thirdly, ICU admission is an institution-dependent outcome, which may limit the generalizability of our results.
In summary, our study showed that in patients with confirmed COVID-19 assessed on ED arrival, REMS and NEWS were the most accurate indices for predicting in-hospital death and ICU admission within seven days, respectively. Due to their high NPV, these indices could be particularly useful for identifying COVID-19 patients who are at lower risk of poor outcome and can be admitted to a non-monitored ward. This supports their use as an adjunct to the process of triage and disposition of COVID-19 patients in the ED. Prospective multicentre cohort studies are needed to provide an external validation of our results.
Credit author statement
All authors have made substantial contributions to all of the following: (1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
The manuscript, including related data, figures and tables, has not been published previously and it is not under consideration elsewhere.
Conflict of interests
None Declared.
Acknowledgements and Funding
None Declared.
Appendix A. Supplementary data
The following is Supplementary data to this article:
Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases From the Chinese Center for Disease Control and Prevention.
Can binary early warning scores perform as well as standard early warning scores for discriminating a patient’s risk of cardiac arrest, death or unanticipated intensive care unit admission?.
A Risk Classification Model to Predict Mortality Among Laboratory-Confirmed Avian Influenza A H7N9 Patients: A Population-Based Observational Cohort Study.
A Comparison of the Quick Sequential (Sepsis-Related) Organ Failure Assessment Score and the National Early Warning Score in Non-ICU Patients With/Without Infection.
The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit.
A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study.
United Nations Department of Economic and Social Affairs Population Dynamics. World Population Prospects 2019. Accessed May 24, 2020 at https://population.un.org/wpp/.
ICU Admission, Discharge, and Triage Guidelines: A Framework to Enhance Clinical Operations, Development of Institutional Policies, and Further Research.
30439-1&id=gr1.jpg){kind=link}