Advertisement

Predicting intensive care unit admission and death for COVID-19 patients in the emergency department using early warning scores

  • Marcello Covino
    Affiliations
    Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
    Search for articles by this author
  • Claudio Sandroni
    Correspondence
    Corresponding author at: Department of Department of Intensive Care, Emergency Medicine and Anaesthesiology - Fondazione Policlinico Universitario A. Gemelli, IRCCS, Università Cattolica del Sacro Cuore – Roma, Largo Francesco Vito, 1, 00168 Rome, Italy.
    Affiliations
    Department of Intensive Care, Emergency Medicine and Anaesthesiology, Fondazione Policlinico Universitario A. Gemelli-IRCCS, Rome, Italy

    Institute of Anaesthesiology and Intensive Care Medicine, Università Cattolica del Sacro Cuore, Rome, Italy
    Search for articles by this author
  • Michele Santoro
    Affiliations
    Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
    Search for articles by this author
  • Luca Sabia
    Affiliations
    Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
    Search for articles by this author
  • Benedetta Simeoni
    Affiliations
    Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
    Search for articles by this author
  • Maria Grazia Bocci
    Affiliations
    Department of Intensive Care, Emergency Medicine and Anaesthesiology, Fondazione Policlinico Universitario A. Gemelli-IRCCS, Rome, Italy
    Search for articles by this author
  • Veronica Ojetti
    Affiliations
    Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy

    Department of Internal Medicine and Gastroenterology, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
    Search for articles by this author
  • Marcello Candelli
    Affiliations
    Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
    Search for articles by this author
  • Massimo Antonelli
    Affiliations
    Department of Intensive Care, Emergency Medicine and Anaesthesiology, Fondazione Policlinico Universitario A. Gemelli-IRCCS, Rome, Italy

    Department of Internal Medicine and Gastroenterology, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy
    Search for articles by this author
  • Antonio Gasbarrini
    Affiliations
    Department of Internal Medicine and Gastroenterology, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy

    Institute of Internal Medicine and Gastroenterology, Università Cattolica del Sacro Cuore, Rome, Italy
    Search for articles by this author
  • Francesco Franceschi
    Affiliations
    Emergency Department, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy

    Institute of Emergency Medicine, Università Cattolica del Sacro Cuore, Rome, Italy
    Search for articles by this author

      Abstract

      Aims

      To identify the most accurate early warning score (EWS) for predicting an adverse outcome in COVID-19 patients admitted to the emergency department (ED).

      Methods

      In adult consecutive patients admitted (March 1-April 15, 2020) to the ED of a major referral centre for COVID-19, we retrospectively calculated NEWS, NEWS2, NEWS-C, MEWS, qSOFA, and REMS from physiological variables measured on arrival. Sensitivity, specificity, positive (PPV) and negative predictive value (NPV), and the area under the receiver operating characteristic (AUROC) curve of each EWS for predicting admission to the intensive care unit (ICU) and death at 48 h and 7 days were calculated.

      Results

      We included 334 patients (119 [35.6%] females, median age 66 [54-78] years). At 7 days, the rates of ICU admission and death were 56/334 (17%) and 26/334 (7.8%), respectively. NEWS was the most accurate predictor of ICU admission within 7 days (AUROC 0.783 [95% CI, 0.735-0.826]; sensitivity 71.4 [57.8-82.7]%; NPV 93.1 [89.8-95.3]%), while REMS was the most accurate predictor of death within 7 days (AUROC 0.823 [0.778–0.863]; sensitivity 96.1 [80.4-99.9]%; NPV 99.4[96.2–99.9]%). Similar results were observed for ICU admission and death at 48 h. NEWS and REMS were as accurate as the triage system used in our ED. MEWS and qSOFA had the lowest overall accuracy for both outcomes.

      Conclusion

      In our single-centre cohort of COVID-19 patients, NEWS and REMS measured on ED arrival were the most sensitive predictors of 7-day ICU admission or death. EWS could be useful to identify patients with low risk of clinical deterioration.

      Keywords

      Introduction

      After the first cases were identified in Wuhan (China) in December 2019, the novel coronavirus SARS-CoV-2 has caused a pandemic of respiratory illness named COVID-19.
      • Wu Z.
      • McGoogan J.M.
      Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases From the Chinese Center for Disease Control and Prevention.
      To date, more than 29 million cases have been reported worldwide, including more than 900,000 deaths.

      European Centre for Disease Prevention and Control COVID-19: Situation update worldwide Accessed September 10, 2020 at https://www.ecdc.europa.eu/en/geographical-distribution-2019-ncov-cases.

      Typical COVID-19 patients present with fever, myalgia, fatigue, and dry cough.
      • Guan W.J.
      • Ni Z.Y.
      • Hu Y.
      • et al.
      Clinical Characteristics of Coronavirus Disease 2019 in China.
      • Huang C.
      • Wang Y.
      • Li X.
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
      Severe cases progress to severe dyspnoea and hypoxemia within one week after the onset of symptoms.
      • Guan W.J.
      • Ni Z.Y.
      • Hu Y.
      • et al.
      Clinical Characteristics of Coronavirus Disease 2019 in China.
      • Huang C.
      • Wang Y.
      • Li X.
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
      • Wu C.
      • Chen X.
      • Cai Y.
      • et al.
      Risk Factors Associated With Acute Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease 2019 Pneumonia in Wuhan, China.
      In hospitalized COVID-19 patients, the prevalence of hypoxemic respiratory failure is around 20%, and more than 25% of them may require intensive care treatment.
      • Wang D.
      • Hu B.
      • Hu C.
      • et al.
      Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China.
      The increasing number of COVID-19 cases has challenged healthcare systems worldwide. In this context of overwhelming demand for medical assessment and triage in emergency departments (EDs), early warning scores (EWS) may be useful. EWS are physiological scoring systems based on a rapid and quantitative assessment of changes in vital signs,
      • Lyons P.G.
      • Edelson D.P.
      • Churpek M.M.
      Rapid response systems.
      and were initially developed to identify and track hospitalised patients at risk of deterioration outside critical care areas in order to ensure early stabilisation and transfer to intensive care unit (ICU) when appropriate, and to prevent avoidable cardiac arrests.
      • Sandroni C.
      • D’Arrigo S.
      • Antonelli M.
      Rapid response systems: are they really effective?.
      • Jarvis S.
      • Kovacs C.
      • Briggs J.
      • et al.
      Can binary early warning scores perform as well as standard early warning scores for discriminating a patient’s risk of cardiac arrest, death or unanticipated intensive care unit admission?.
      However, these scores have also subsequently been investigated as a tool to predict critical illness in ED patients,
      • Nannan Panday R.S.
      • Minderhoud T.C.
      • Alam N.
      • Nanayakkara P.W.B.
      Prognostic value of early warning scores in the emergency department (ED) and acute medical unit (AMU): A narrative review.
      and some, as the Rapid Emergency Medicine Score (REMS),
      • Olsson T.
      • Terent A.
      • Lind L.
      Rapid Emergency Medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients.
      have specifically been developed for this purpose. In studies performed on the general ED population, the National Early Warning Score (NEWS)
      • McGinley A.
      • Pearse R.
      A national early warning score for acutely ill patients.
      was the most accurate in predicting both in-hospital mortality and ICU admission.
      • Nannan Panday R.S.
      • Minderhoud T.C.
      • Alam N.
      • Nanayakkara P.W.B.
      Prognostic value of early warning scores in the emergency department (ED) and acute medical unit (AMU): A narrative review.
      A recent document from the Swiss Society of Intensive Care Medicine
      • Swiss Society of Intensive Care Medicine
      Recommendations for the admission of patients with COVID-19 to intensive care and intermediate care units (ICUs and IMCUs).
      suggested using a score greater than 6 of a modified version of NEWS (hereafter referred as to NEWS-C) for deciding admission to ICU of COVID-19 patients.
      • Liao X.
      • Wang B.
      • Kang Y.
      Novel coronavirus infection during the 2019-2020 epidemic: preparing intensive care units-the experience in Sichuan Province, China.
      However, this recommendation is based on previous evidence on non-COVID-19 patients,
      • Martinez L.
      • Cheng W.
      • Wang X.
      • et al.
      A Risk Classification Model to Predict Mortality Among Laboratory-Confirmed Avian Influenza A H7N9 Patients: A Population-Based Observational Cohort Study.
      • Redfern O.C.
      • Smith G.B.
      • Prytherch D.R.
      • Meredith P.
      • Inada-Kim M.
      • Schmidt P.E.
      A Comparison of the Quick Sequential (Sepsis-Related) Organ Failure Assessment Score and the National Early Warning Score in Non-ICU Patients With/Without Infection.
      and it has not been validated so far. At present, it is not clear which EWS is the most accurate for triaging COVID-19 patients in the ED.
      The aim of the present study was to compare the ability of EWS assessed on ED arrival to identify adult COVID-19 patients at risk for ICU admission or death within seven days. The EWS assessed in this study included Modified EWS (MEWS),
      • Burch V.C.
      • Tarr G.
      • Morroni C.
      Modified early warning score predicts the need for hospital admission and inhospital mortality.
      NEWS, NEWS2,
      • Smith G.B.
      • Redfern O.C.
      • Pimentel M.A.
      • et al.
      The National Early Warning Score 2 (NEWS2).
      NEWS-C, Quick Sepsis Related Organ Failure Assessment (qSOFA),
      • Seymour C.W.
      • Liu V.X.
      • Iwashyna T.J.
      • et al.
      Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).
      and REMS. Moreover, in order to measure the added value of EWS to the conventional ED triage process, we also compared the accuracy of EWS with that of the Regional Triage System for Infectious Diseases in use in our hospital for COVID-19 patients.

      Methods

      Study design and setting

      This is a retrospective observational study (clinicaltrials.gov NCT04371562) conducted in the ED of the Fondazione Policlinico Universitario Agostino Gemelli, in Rome. Policlinico Gemelli is the largest teaching hospital in Rome and a referral centre for COVID-19 in central Italy. Our ED has a catchment area of 1,8 M inhabitants, with an average of 80k access/year.
      The Regional Triage System for Infectious Diseases in our ED is based on a protocol issued by the Health Service of Lazio Region, where our hospital is located. The protocol, summarised on ESM Tables 1a-b, follows an ABCDE scheme (airway, breathing, circulation, disability, exposure) with additional variables such as age, blood glucose level, plus additional symptoms and signs. The triage system assigns a priority value from 1 to 5, in descending order of severity. Triage is performed by a nurse certified in a full time 7-day course followed by a 6-month supervised training.
      The criteria for ICU admission of COVID-19 patients in our hospital during the study period included circulatory shock requiring vasopressors and the need for positive pressure respiratory support, including non-invasive ventilation. High-flow oxygen treatment is provided in general wards. ICU admission criteria were consistent throughout the study period. At present, no EWS is in use in our hospital.

      Eligibility criteria

      From March 1 to April 15, 2020 we included all adult (>18y) patients assessed in our ED for suspected COVID-19 whose diagnosis was confirmed with real-time reverse-transcriptase–polymerase-chain-reaction assay of nasal and pharyngeal swab specimens. Testing for COVID-19 was conducted according to the World Health Organization (WHO) interim guidance.

      World Health Organization. Laboratory testing of human suspected cases of novel coronavirus (nCoV) infectionInterim guidance Accessed April 8, 2020 at https://apps.who.int/iris/bitstream/handle/10665/330374/WHO-2019-nCoV-laboratory-2020.1-eng.pdf.

      We excluded pregnant women and patients who were on circulatory shock and positive pressure support on arrival to ED. Patients with a do-not-resuscitate/do-not-intubate (DNR/DNI) order in place were also excluded.

      Data collection and measurements

      Two authors blinded to patients’ outcome retrieved clinical information from the electronic medical records (EMR) of the ED database. Extracted data included age, sex, clinical history and presentation, temperature, heart rate (HR), respiratory rate (RR), blood pressure (BP), Glasgow Coma Scale (GCS) score, oxygen therapy, and peripheral oxygen saturation (SpO2). All clinical signs, including SpO2, were measured upon ED arrival. EWS were calculated for each patient using Microsoft Excel 2010 for Windows (Microsoft Inc., Redmond, WA, USA) from available physiological measurements. Variables included in each of these scores are listed on ESM Table 2. For NEWS2 calculation, patients were considered at risk of type 2 respiratory failure (T2RF) if they had a confirmed history of chronic obstructive pulmonary disease.

      Outcomes

      The primary study outcomes were death and ICU admission within seven days from ED arrival. Secondary outcomes were death and ICU admission within 48 h from ED arrival. Outcomes were retrospectively assessed by reviewing EMRs of the hospital medical database.

      Statistical analysis

      Continuous variables were reported as median [interquartile range, IQR] and were compared with univariate analysis by Mann-Whitney U test. Categorical variables were reported as absolute number (percentage) and are compared by Chi-square test (with Fisher’s test if appropriate).
      Receiver operating characteristic (ROC) curve analysis was used to estimate the performance of the evaluated scores in predicting the study outcomes. For each score threshold value, the following was calculated: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (+LR), and negative likelihood ratio (-LR). The Youden index was used to estimate the optimal thresholds for sensitivity and specificity. The comparison between the areas under the ROC curve (AUROCs) was made according to the DeLong method.
      • DeLong E.R.
      • DeLong D.M.
      • Clarke-Pearson D.L.
      Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
      A p value ≤ 0.05 was regarded as significant.
      To check the robustness of our analysis we imputed data in patients with less than three missing parameters using a multiple imputation approach.
      • Sterne J.A.
      • White I.R.
      • Carlin J.B.
      • et al.
      Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls.
      The results of imputation were added to the original study cohort with the complete dataset of vital parameters and EWS were recalculated. The resulting AUROCs were compared to the AUROC obtained in the study population by using the z-statistics (2 sided). Further data on imputation are available in Appendix 1.
      Data were analysed using IBM SPSS statistics for Windows, Version 25 (IBM Corp. Armonk, NY, USA) and MedCalc Statistical Software version 19.2.1 (MedCalc Software Ltd, Ostend, Belgium; https://www.medcalc.org; 2020).

      Ethical approval

      This study has been approved by the local Ethics Committee (authorization #001705520) and has been performed in accordance with the ethical standards established in the 1964 Declaration of Helsinki and its later amendments. All patients gave their informed consent for clinical data analysis.

      Results

      Characteristics of the study subjects

      During the study period, 3955 patients were seen in the ED, of whom 1801 were evaluated for suspect COVID-19 and 512 were found positive to the COVID-19 test. Of these, 336 had sufficient data on EMR for calculating EWS. Two patients with a DNR/DNI order in place were excluded, leaving 334 patients for final analysis (Fig. 1). Patients’ median [IQR] age was 66 [54 – 78] years (range 20 -98 years) and 215 (64.4%) were male (Table 1).
      Fig. 1
      Fig. 1Flow-chart of study selection. DNR/DNI: Do-not-resuscitate/Do-not-intubate order. EMR: Electronic Medical Record; EWS: Early Warning Score.
      Table 1Characteristics of the study population and distribution of score values according to patient outcome. Data for ICU admission and death are referred to day 7. Data are presented as median (25th – 75th percentile) or number (percentage).
      All Patients

      (n° 334)
      Admitted to ICUPSurvived (308 [90.9%])Dead (26 [7.8%])p
      No (278 [83.2%])Yes (56 [16.8%])
      Age, years66 [54–78]63 [51–77]71.5 [64–78]0.00264 [53–77]79.5 [74–86]<0.001
      Sex, male215 (64.4%)173 (62.2%)42 (75.0%)0.069196 (63.6%)19 (73.1%)0.334
      Heart rate, beats · min-190 [79–105]88 [78–103]98 [86–105]0.05890 [79–104]95 [81–113]0.265
      Respiratory rate, breaths · min-118 [16–21]18 [16–21]20 [17–21]0.02718 [16–21]20 [17–23]0.068
      Glasgow Coma Score15 [15–15]15 [15–15]15 [15–15]0.79315 [15–15]15 [15–15]0.002
      Blood pressure, mmHg
      Systolic127 [114–140]130 [119–138]129 [117–140]0.838127 [114–144]128 [115–139]0.728
      Diastolic80 [70–89]80 [70–89]77 [70–84]0.06480 [70–88]73 [64–89]0.089
      Mean96 [86–103]96 [86–104]93 [84–100]0.35194 [85–103]92 [84–101]0.268
      SpO2, %95 [92–97]95 [93–97]88.4 [81–92]<0.00195 [92–97]89 [79–92]<0.001
      Body Temperature, °C37.0 [36.7–37.5]37.0 [36.7–37.5]37.1 [36.9–37.4]0.21336.5 [36.0–37.2]37.1 [37.0–37.5]0.614
      Comorbidities
      Hypertension140 (41.9%)107 (38.5%)33 (58.9%)0.005123 (39.9%)17 (65.4%)0.012
      CAD, CHF81 (24.3%)57 (20.5%)24 (42.9%)<0.00170 (22.7%)11 (42.3%)0.025
      Diabetes36 (10.7%)30 (10.8%)6 (10.7%)1.00034 (11.0%)2 (7.7%)0.597
      COPD19 (5.7%)14 (5.0%)5 (8.9%)0.25116 (5.2%)3 (11.5%)0.180
      Dementia12 (3.6%)12 (4.3%)00.1139 (2.9%)3 (11.5%)0.023
      Malignancy14 (4.2%)12 (4.3%)2 (3.6%)0.80012 (3.9%)2 (7.7%)0.354
      Chronic Kidney disease11 (3.3%)9 (3.2%)2 (3.6%)0.8988 (2.6%)3 (11.5%)0.046
      Severe Obesity6 (1.8%)3 (1.1%)3 (5.4%)0.0286 (1.9%)01.000
      Living in an institution29 (8.6%)27 (9.6%)2 (3.6%)0.14026 (8.4%)3 (11.5%)0.483
      TRIAGE code3 [3–4]4 [3–4]2 [1–3]<0.0013 [3–4]2 [1–3]<0.001
      Early Warning Scores
      MEWS5 [4–7]5 [4–7]6 [5–7]0.0045 [4–7]6 [4–8]0.136
      NEWS3 [1–5]2 [1–4]6 [4–8]<0.0013 [1–5]6 [3–9]<0.001
      NEWS22 [1–5]2 [1–4]6 [3–8]<0.0012 [1–5]6 [2–9]<0.001
      NEWS – C6 [4–9]6 [4–8]9 [6–11]<0.0016 [4–8]10 [6–12]<0.001
      qSOFA0 [0–0]0 [0–0]0 [0–1]0.0660 [0–0]1 [0–1]<0.001
      REMS5 [3–7]5 [3–6]7 [5–9]<0.0015 [3–7]8 [6–11]<0.001
      Outcome
      Admitted to ICU at ≤7 days56 (16.7%)--38 (12.3%)18 (69.2%)<0.001
      Deceased at ≤7 Days26 (7.8%)8 (2.9%)18 (32.1%)<0.001--
      Abbreviations: CAD Coronary Artery Disease; CHF Congestive Heart Failure; COPD Chronic Obstructive Pulmonary Disease; ICU Intensive Care Unit. TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modified NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score.
      Thirty-eight (11%) COVID-19 patients were discharged from ED and quarantined. These patients were contacted daily to assess their clinical condition. The remaining 296 patients were admitted to hospital, of whom 56 (17% of the total) were admitted to ICU. Fifty patients were admitted to ICU within 48 hours, while the remaining six patients were admitted on days 3-7 after ED arrival. Patients who needed ICU admission were older, had lower SpO2 and higher respiratory rate on ED arrival, and more frequently reported a history of hypertension, obesity, and chronic heart failure or coronary artery disease (Table 1).
      Overall, 26 patients (7.8%) deceased within seven days after ED arrival, among whom 11 (3.3%) deceased by 48 hours. These patients were significantly older, had a higher respiratory rate, and a lower SpO2 on ED arrival. As for ICU admission, in patients who died a history of hypertension and chronic heart failure or coronary artery disease was more commonly reported (Table 1).

      Accuracy of warning scores

      Concerning ICU admission, among all the EWS we assessed, NEWS had the highest overall accuracy both at 48 h and at 7 days (AUROC 0.802 [0.756 – 0.844] and 0.783 [0.735 – 0.826], respectively), and the highest NPV, both at 48 h and at 7 days (93.4 [90.6 – 95.4]% for NEWS ≤ 5 and 93.1 [89.8 – 95.3]% for NEWS ≤ 4, respectively) along with NEWS2 (NPV 93.7 [90.6 – 95.8] and 92.4 [89.3 – 94.7] at 48 h and 7 days, respectively). The accuracy was lower, although not significantly, than that of the triage system of our hospital (Table 2).
      Table 2Sensitivity, specificity, negative and positive predictive value, and positive and negative likelihood ratio of the hospital triage score and of the EWS for predicting ICU admission within 48 hours and 7 days from ED admission. Optimal cut off values for sensitivity and specificity were chosen according to Youden index. Values of the EWS with the largest AUROC are in bold.
      AUROCcut-offSensitivity (%)Specificity (%)+LR-LRPPVNPV
      Admission to ICU within 48 hours
      TRIAGE0.839 [0.795 – 0.877]≤ 266.0 [51.2 – 78.8]89.8 [85.7 – 93.1]6.5 [4.3 – 9.6]0.4 [0.3 – 0.6]53.2 [43.3 – 62.9]93.7 [91.1 – 95.7]
      MEWS0.652 [0.598 – 0.703]> 570.0 [55.4 – 82.1]64.8 [58.9 – 70.3]2.0 [1.6 – 2.5]0.5 [0.3 – 0.7]25.9 [21.6 – 30.8]92.5 [88.8 – 95.0]
      NEWS0.802 [0.756 – 0.844]> 566.0 [51.2 – 78.8]84.9 [80.2 – 88.8]4.4 [3.1 – 6.1]0.4 [0.3 – 0.6]43.4 [35.3 – 51.9]93.4 [90.6 – 95.4]
      NEWS20.780 [0.731 – 0.823]> 470.0 [55.4 – 82.1]78.5 [73.3 – 83.2]3.3 [2.4 – 4.3]0.4 [0.2 – 0.6]36.5 [30.1 – 43.3]93.7 [90.6 – 95.8]
      NEWS-C0.771 [0.701 – 0.831]> 862.9 [44.9 – 78.5]82.7 [75.4 – 88.6]3.6 [2.3 – 5.7]0.5 [0.3 – 0.7]47.8 [37.0 – 58.8]89.8 [85.1 – 93.2]
      qSOFA0.566 [0.511 – 0.620]> 034.0 [21.2 – 48.8]79.6 [74.4 – 84.1]1.7 [1.1 – 2.6]0.8 [0.7 – 1.0]22.7 [15.8 – 31.5]87.3 [84.8 – 89.4]
      REMS0.742 [0.692 – 0.788]> 660.0 [45.2 – 73.6]75.0 [69.5 – 79.9]2.4 [1.8 – 3.2]0.5 [0.4 – 0.8]29.7 [23.8 – 36.4]91.4 [88.3 – 93.8]
      Admission to ICU within 7 days
      TRIAGE0.818 [0.773 – 0.858]≤ 258.9 [45.0 – 71.9]89.6 [85.4 – 92.9]5.6 [3.8 – 8.5]0.5 [0.3 – 0.6]53.2 [43.1 – 63.1]91.5 [88.8 – 93.7]
      MEWS0.618 [0.564 – 0.671]> 566.1 [52.2 – 78.2]64.8 [58.8 – 70.4]1.9 [1.5 – 2.4]0.5 [0.4 – 0.8]27.4 [22.8 – 32.6]90.5 [86.7 – 93.2]
      NEWS0.783 [0.735 – 0.826]> 471.4 [57.8 – 82.7]77.3 [72.1 – 82.1]3.2 [2.4 – 4.1]0.4 [0.2 – 0.6]38.8 [32.6 – 45.5]93.1 [89.8 – 95.3]
      NEWS20.762 [0.714 – 0.807]> 467.8 [54.0 – 79.7]79.1 [73.9 – 83.8]3.2 [2.4 – 4.4]0.4 [0.3 – 0.6]39.6 [32.9 – 46.7]92.4 [89.3 – 94.7]
      NEWS-C0.755 [0.684 – 0.817]> 860.1 [51.9 – 81.9]84.2 [76.9 – 90.0]3.9 [2.4 – 6.1]0.5 [0.3 – 0.7]54.3 [42.8 – 65.4]87.5 [82.6 – 91.2]
      qSOFA0.557 [0.501 – 0.611]> 032.1 [20.3 – 46.0]79.5 [74.3 – 84.1]1.6 [1.0 – 2.4]0.8 [0.7 – 1.0]24.0 [16.8 – 33.0]85.3 [82.8 – 87.5]
      REMS0.735 [0.685 – 0.782]> 658.9 [45.0 – 71.9]75.5 [70.0 – 80.5]2.4 [1.8 – 3.3]0.5 [0.4 – 0.7]32.7 [26.4 – 39.6]90.1 [86.9 – 92.6]
      Abbreviations: TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modi0fied NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score; +LR Positive Likelihood Ratio; −LR Negative Likelihood Ratio; PPV Positive Predictive Value; NPV Negative Predictive Value.
      Concerning in-hospital death, REMS showed the highest overall accuracy both at 48 h and 7 days (AUROC 0.882 [0.843 – 0.915] and 0.823 [0.778 – 0.863], respectively). These AUROCs were larger, although not significantly, than those of the triage system of our hospital at both time points. REMS had also the highest NPV for in-hospital death (99.6 [97.3 – 99.9]% for REMS > 6 at 48 h and 99.4 [96.2 – 99.9]% for REMS > 5 at 7 days; Table 3).
      Table 3Sensitivity, specificity, negative and positive predictive value, and positive and negative likelihood ratio of the hospital triage score and of the EWS for predicting death within 48 hours and 7 days from ED admission. Optimal cut off values for sensitivity and specificity were chosen according to Youden index. Values of the EWS with the largest AUROC are in bold.
      AUROCcut-offSensitivity (%)Specificity (%)+LR-LRPPVNPV
      Death within 48 hours
      TRIAGE0.849 [0.806 – 0.886]≤ 281.8 [48.2 – 97.7]83.6 [79.1 – 87.5]5.0 [4.1 – 15.7]0.5 [0.3 – 0.9]14.5 [10.5 – 19.8]99.3 [97.5 – 99.8]
      MEWS0.630 [0.576 – 0.682]> 563.6 [30.8 – 89.1]60.4 [54.8 – 65.7]1.6 [1.0 – 2.6]0.6 [0.3 – 1.3]5.2 [3.3 – 8.0]98.0 [95.7 – 99.1]
      NEWS0.829 [0.742 – 0.832]>481.8 [48.2 – 97.7]70.9 [65.6 – 75.8]2.8 [2.0 – 3.9]0.3 [0.1 – 0.9]8.7 [6.5 – 11.7]99.1 [97.0 – 99.8]
      NEWS20.753 [0.703 – 0.798]> 472.7 [39.0 – 94.0]72.7 [67.6 – 77.5]2.7 [1.8 – 4.0]0.4 [0.1 – 1.0]8.3 [5.7 – 12.0]98.7 [96.8 – 99.5]
      NEWS-C0.773 [0.703 – 0.833]> 970.0 [34.8 – 93.3]82.3 [75.6 – 87.8]4.0 [2.3 – 6.7]0.4 [0.1 – 0.9]19.4 [12.5 – 28.9]97.8 [94.6 – 99.1]
      qSOFA0.808 [0.762 – 0.849]> 081.8 [48.2 – 97.7]79.6 [74.8 – 83.8]4.0 [2.8 – 5.7]0.2 [0.1 – 0.8]12.0 [8.8 – 16.2]99.2 [97.3 – 99.8]
      REMS0.882 [0.843 – 0.915]> 690.9 [58.7 – 99.8]71.8 [66.6 – 76.7]3.2 [2.5 – 4.2]0.1 [0.1 – 0.8]9.9 [7.8 – 12.4]99.6 [97.3 – 99.9]
      Death within 7 days
      TRIAGE0.819 [0.773 – 0.858]≤ 261.5 [40.6 – 79.8]85.1 [80.6 – 88.9]4.1 [2.8 – 6.2]0.5 [0.3 – 0.7]25.8 [18.8 – 34.3]94.4 [92.8 – 95.7]
      MEWS0.586 [0.531 – 0.640]> 557.7 [36.9 – 76.6]61.0 [55.3 – 66.5]1.5 [1.0 – 2.1]0.7 [0.4 – 1.1]11.1 [8.0 – 15.2]94.5 [91.5 – 96.4]
      NEWS0.768 [0.719 – 0.812]> 565.4 [44.3 – 82.8]80.8 [76.0 – 85.1]3.4 [2.4 – 4.9]0.4 [0.3 – 0.7]22.4 [16.7 – 29.3]96.5 [94.2 – 97.9]
      NEWS20.727 [0.676 – 0.774]> 561.5 [40.6 – 79.8]81.5 [76.7 – 85.7]3.3 [2.3 – 4.9]0.5 [0.3 – 0.8]17.7 [13.3 – 23.2]96.2 [93.9 – 97.6]
      NEWS-C0.767 [0.697 – 0.828]> 865.2 [42.7 – 83.6]79.5 [72.1 – 85.6]3.1 [2.1 – 4.9]0.4 [0.2 – 0.8]32.6 [23.9 – 42.7]93.7 [89.5 – 96.4]
      qSOFA0.673 [0.620 – 0.723]> 053.8 [33.4 – 73.3]80.2 [75.3 – 84.5]2.7 [1.8 – 4.1]0.6 [0.4 – 0.9]18.7 [13.1 – 25.9]95.4 [93.1 – 96.9]
      REMS0.823 [0.778 – 0.863]> 596.1 [80.4 – 99.9]56.8 [51.1 – 62.4]2.3 [1.9 – 2.6]0.1 [0.0 – 0.5]15.8 [13.9 – 17.9]99.4 [96.2 – 99.9]
      Abbreviations: TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modified NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score; +LR Positive Likelihood Ratio; −LR Negative Likelihood Ratio; PPV Positive Predictive Value; NPV Negative Predictive Value.
      For both outcomes, the overall accuracy of the NEWS-C was consistently lower than that of NEWS in our population. Among all EWS, MEWS and qSOFA showed the lowest overall accuracy. Their AUROCs were significantly lower than that of the triage system of our hospital, except for prediction of death at 48 h for qSOFA (Table 3).
      ESM figures 1 and 2 include the ROC curves for prediction of ICU admission and death, respectively, along with a statistical comparison of the AUROCs of the scores included in the study.
      Among the 176 patients excluded because of missing parameters, 78 (44.3%) had less than three missing parameters and were eligible for imputation. Their available parameters are shown in Appendix 1, Table A1. Among these 78 patients, six deaths (7.7%) and nine ICU admissions (11.5%) occurred within 7 days.
      The EWS AUROCs in the extended population including 78 patients with imputed vital parameters (total 412 patients) were not significantly different from those of the 334 patients with complete original data. (Appendix 1, Tables A2 and A3).

      Discussion

      Our study showed that in COVID-19 patients NEWS and REMS assessed on arrival at the emergency department were the most accurate scores for predicting the risk of ICU admission and death, respectively, both at 48 h and at 7 days. Moreover, the accuracy of these scores was comparable with that of the comprehensive triage score used in our department.
      NEWS is among the most accurate tools for predicting patient deterioration outside ICU.
      • Nannan Panday R.S.
      • Minderhoud T.C.
      • Alam N.
      • Nanayakkara P.W.B.
      Prognostic value of early warning scores in the emergency department (ED) and acute medical unit (AMU): A narrative review.
      In a comparative study including 198,755 observation sets collected from 35,585 consecutive acute medical admissions in UK,
      • Smith G.B.
      • Prytherch D.R.
      • Meredith P.
      • Schmidt P.E.
      • Featherstone P.I.
      The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
      NEWS was more strongly associated with ICU admission or death compared to 33 other scoring systems (not including REMS). Moreover, in the specific setting of acute care patients admitted for suspected infection in ED or hospital wards, NEWS has proven to be more accurate for predicting death or ICU transfer than other tools that were purposely designed to assess infected patients, such as qSOFA and systemic inflammatory response syndrome (SIRS) criteria.
      • Churpek M.M.
      • Snyder A.
      • Han X.
      • et al.
      Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit.
      In 2017, NEWS has been updated to NEWS2 by adding a new SpO2 scoring scale for patients with type 2 respiratory failure (T2RF),

      NHS England National Early Warning Score (NEWS). Accessed May 24, 2020 at https://www.england.nhs.uk/ourwork/clinical-policy/sepsis/nationalearlywarningscore/.

      even if the added value of this modification in improving discrimination of adverse outcomes in patients with documented T2RF has not been demonstrated yet.
      • Pimentel M.A.F.
      • Redfern O.C.
      • Gerry S.
      • et al.
      A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study.
      NEWS2 is recommended by the Royal College of Physicians for use in COVID-19 patients.

      Royal College of Physicians NEWS2 and deterioration in COVID-19. Accessed May 24, 2020 at https://www.rcplondon.ac.uk/news/news2-and-deterioration-covid-19.

      In our study, NEWS2 did not perform better than NEWS in predicting ICU admission or death. However, the prevalence of T2RF patients in our population was very low (19/334 patients; 5.7%). This is in accordance with data from other studies showing that hypercapnia is uncommon in COVID-19 patients, even in those admitted to ICU,
      • Grasselli G.
      • Zangrillo A.
      • Zanella A.
      • et al.
      Baseline Characteristics and Outcomes of 1591 Patients Infected With SARS-CoV-2 Admitted to ICUs of the Lombardy Region, Italy.
      while hypoxia from early pulmonary involvement is the most common presentation feature.
      • Huang C.
      • Wang Y.
      • Li X.
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
      • Shi H.
      • Han X.
      • Jiang N.
      • et al.
      Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study.
      NEWS-C, another modified version of NEWS, has recently been suggested for assessment of COVID-19 patients.
      • Liao X.
      • Wang B.
      • Kang Y.
      Novel coronavirus infection during the 2019-2020 epidemic: preparing intensive care units-the experience in Sichuan Province, China.
      Age ≥65 years was added to NEWS-C as an independent risk factor based on recent reports in patients with influenza or other non-COVID infectious diseases.
      • Martinez L.
      • Cheng W.
      • Wang X.
      • et al.
      A Risk Classification Model to Predict Mortality Among Laboratory-Confirmed Avian Influenza A H7N9 Patients: A Population-Based Observational Cohort Study.
      • Redfern O.C.
      • Smith G.B.
      • Prytherch D.R.
      • Meredith P.
      • Inada-Kim M.
      • Schmidt P.E.
      A Comparison of the Quick Sequential (Sepsis-Related) Organ Failure Assessment Score and the National Early Warning Score in Non-ICU Patients With/Without Infection.
      Indeed, older age is associated with increased mortality in COVID-19 patients.
      • Chen T.
      • Wu D.
      • Chen H.
      • et al.
      Clinical characteristics of 113 deceased patients with coronavirus disease 2019: retrospective study.
      However, NEWS-C did not perform better than NEWS in predicting ICU admission or mortality in our study. This result can be explained considering that our population was relatively old. In fact, the median age of survivors vs. non-survivors was 64 years vs. 79.5 years in our COVID-19 population, as compared with 51 vs.68 years respectively in a recent report from China.
      • Chen T.
      • Wu D.
      • Chen H.
      • et al.
      Clinical characteristics of 113 deceased patients with coronavirus disease 2019: retrospective study.
      These data may reflect differences in demographics (median population age is 34.4 years in China vs. 42.5 years in Europe and 47.3 years in Italy

      United Nations Department of Economic and Social Affairs Population Dynamics. World Population Prospects 2019. Accessed May 24, 2020 at https://population.un.org/wpp/.

      ) and in policies for ICU admission and triage. When weighting for age, REMS may be particularly appropriate, since – unlike other EWS – it stratifies age (five groups from <45 to >74 years; see also ESM Table 2). In our COVID-19 population, REMS was the most accurate score for predicting mortality. Among the scoring systems we evaluated, REMS was the only one specifically developed to predict mortality in ED patients.
      • Olsson T.
      • Terent A.
      • Lind L.
      Rapid Emergency Medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients.
      • Ha D.T.
      • Dang T.Q.
      • Tran N.V.
      • Vo N.Y.
      • Nguyen N.D.
      • Nguyen T.V.
      Prognostic performance of the Rapid Emergency Medicine Score (REMS) and Worthing Physiological Scoring system (WPS) in emergency department.
      The qSOFA has been proposed as a prompt to consider possible sepsis outside the ICU. Criteria to calculate qSOFA include altered mental status (GCS < 15), tachypnoea (RR ≥ 22), and hypotension (systolic BP ≤ 100). In the original validation study, a qSOFA score of 2 or higher was associated with a significantly increased risk of hospital mortality.
      • Seymour C.W.
      • Liu V.X.
      • Iwashyna T.J.
      • et al.
      Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).
      However, the accuracy of qSOFA was low in our study, confirming results from previous studies conducted in non-COVID potentially septic patients.
      • Churpek M.M.
      • Snyder A.
      • Han X.
      • et al.
      Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit.
      • Cleek W.R.
      • Johnson N.J.
      • Watsjold B.K.
      • Hall M.K.
      • Henning D.J.
      Comparing Mortality Prediction by Quick Sequential Organ Failure Assessment With Emergency Physician Judgment.
      • Mellhammar L.
      • Linder A.
      • Tverring J.
      • et al.
      NEWS2 is Superior to qSOFA in Detecting Sepsis with Organ Dysfunction in the Emergency Department.
      Notably, most patients in our cohort did not have any of the three qSOFA criteria, and none of them had a qSOFA score of 2 or higher. Among the three qSOFA criteria, the prevalence of hypotension and altered mental status on ED admission in our patient population was particularly low (6.6% and 3.3%; data not shown). This is in line with recent reports showing that extrapulmonary organ dysfunction occurs later in the clinical course of COVID-19.
      • Zhou F.
      • Yu T.
      • Du R.
      • et al.
      Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
      When compared with other scoring systems, including our institutional triage score, NEWS and REMS had the highest sensitivity and NPV for adverse outcomes. In other words, they performed particularly well in identifying COVID-19 patients who did not require ICU admission or were at low risk of death, respectively. When triaging patients in ED, having a high NPV (i.e., a low rate of false negative predictions) for adverse outcomes is paramount in order to avoid inappropriately admitting a patient at risk of deterioration to a non-critical care area. However, because of the relatively low incidence of adverse outcomes in our population, all the EWS we evaluated had a low PPV (i.e., a low rate of true positive predictions), which implies a risk of overtriage when these scores are used to decide admission to a critical care area. Although consensus suggests that overtriage is generally preferable to undertriage,
      • Nates J.L.
      • Nunnally M.
      • Kleinpell R.
      • et al.
      ICU Admission, Discharge, and Triage Guidelines: A Framework to Enhance Clinical Operations, Development of Institutional Policies, and Further Research.
      this may not be the case when meeting the demands of healthcare resources is difficult. The prognostic indications from physiological scores should be interpreted within a more comprehensive clinical approach including patient’s comorbidities, functional status, and preferences.
      In our study we assessed EWS only once, on patient arrival to the ED. Since EWS have been designed as track-and-trigger systems to detect deterioration of vital signs over time, their sequential measurement in COVID-19 patients admitted to non-monitored wards may represent the logical continuation of their use in ED. The value of EWS in this context deserves further investigation.
      Some limitations of our study should be acknowledged. First, this was a single-centre study conducted on a relatively small population, and its confirmation on a larger, multicentre cohort is warranted. Secondly, the EWS were retrospectively calculated from data recorded on the patients’ EMR that were not specifically collected for this purpose. This resulted in missing data necessitating imputation. Thirdly, ICU admission is an institution-dependent outcome, which may limit the generalizability of our results.
      In summary, our study showed that in patients with confirmed COVID-19 assessed on ED arrival, REMS and NEWS were the most accurate indices for predicting in-hospital death and ICU admission within seven days, respectively. Due to their high NPV, these indices could be particularly useful for identifying COVID-19 patients who are at lower risk of poor outcome and can be admitted to a non-monitored ward. This supports their use as an adjunct to the process of triage and disposition of COVID-19 patients in the ED. Prospective multicentre cohort studies are needed to provide an external validation of our results.

      Credit author statement

      All authors have made substantial contributions to all of the following: (1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
      The manuscript, including related data, figures and tables, has not been published previously and it is not under consideration elsewhere.

      Conflict of interests

      None Declared.

      Acknowledgements and Funding

      None Declared.

      References

        • Wu Z.
        • McGoogan J.M.
        Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases From the Chinese Center for Disease Control and Prevention.
        JAMA. 2020; 323: 1239-1242
      1. European Centre for Disease Prevention and Control COVID-19: Situation update worldwide Accessed September 10, 2020 at https://www.ecdc.europa.eu/en/geographical-distribution-2019-ncov-cases.

        • Guan W.J.
        • Ni Z.Y.
        • Hu Y.
        • et al.
        Clinical Characteristics of Coronavirus Disease 2019 in China.
        N Engl J Med. 2020; 382: 1708-1720
        • Huang C.
        • Wang Y.
        • Li X.
        • et al.
        Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
        Lancet. 2020; 395: 497-506
        • Wu C.
        • Chen X.
        • Cai Y.
        • et al.
        Risk Factors Associated With Acute Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease 2019 Pneumonia in Wuhan, China.
        JAMA Intern Med. 2020; 180: 934-943
        • Wang D.
        • Hu B.
        • Hu C.
        • et al.
        Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China.
        JAMA. 2020; 323: 1061-1069
        • Lyons P.G.
        • Edelson D.P.
        • Churpek M.M.
        Rapid response systems.
        Resuscitation. 2018; 128: 191-197
        • Sandroni C.
        • D’Arrigo S.
        • Antonelli M.
        Rapid response systems: are they really effective?.
        Crit Care. 2015; 19: 104
        • Jarvis S.
        • Kovacs C.
        • Briggs J.
        • et al.
        Can binary early warning scores perform as well as standard early warning scores for discriminating a patient’s risk of cardiac arrest, death or unanticipated intensive care unit admission?.
        Resuscitation. 2015; 93: 46-52
        • Nannan Panday R.S.
        • Minderhoud T.C.
        • Alam N.
        • Nanayakkara P.W.B.
        Prognostic value of early warning scores in the emergency department (ED) and acute medical unit (AMU): A narrative review.
        Eur J Intern Med. 2017; 45: 20-31
        • Olsson T.
        • Terent A.
        • Lind L.
        Rapid Emergency Medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients.
        J Intern Med. 2004; 255: 579-587
        • McGinley A.
        • Pearse R.
        A national early warning score for acutely ill patients.
        BMJ. 2012; 345: e5310
        • Swiss Society of Intensive Care Medicine
        Recommendations for the admission of patients with COVID-19 to intensive care and intermediate care units (ICUs and IMCUs).
        Swiss Med Wkly. 2020; 150w20227
        • Liao X.
        • Wang B.
        • Kang Y.
        Novel coronavirus infection during the 2019-2020 epidemic: preparing intensive care units-the experience in Sichuan Province, China.
        Intensive Care Med. 2020; 46: 357-360
        • Martinez L.
        • Cheng W.
        • Wang X.
        • et al.
        A Risk Classification Model to Predict Mortality Among Laboratory-Confirmed Avian Influenza A H7N9 Patients: A Population-Based Observational Cohort Study.
        J Infect Dis. 2019; 220: 1780-1789
        • Redfern O.C.
        • Smith G.B.
        • Prytherch D.R.
        • Meredith P.
        • Inada-Kim M.
        • Schmidt P.E.
        A Comparison of the Quick Sequential (Sepsis-Related) Organ Failure Assessment Score and the National Early Warning Score in Non-ICU Patients With/Without Infection.
        Crit Care Med. 2018; 46: 1923-1933
        • Burch V.C.
        • Tarr G.
        • Morroni C.
        Modified early warning score predicts the need for hospital admission and inhospital mortality.
        Emerg Med J. 2008; 25: 674-678
        • Smith G.B.
        • Redfern O.C.
        • Pimentel M.A.
        • et al.
        The National Early Warning Score 2 (NEWS2).
        Clin Med (Lond). 2019; 19: 260
        • Seymour C.W.
        • Liu V.X.
        • Iwashyna T.J.
        • et al.
        Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).
        JAMA. 2016; 315: 762-774
      2. World Health Organization. Laboratory testing of human suspected cases of novel coronavirus (nCoV) infectionInterim guidance Accessed April 8, 2020 at https://apps.who.int/iris/bitstream/handle/10665/330374/WHO-2019-nCoV-laboratory-2020.1-eng.pdf.

        • DeLong E.R.
        • DeLong D.M.
        • Clarke-Pearson D.L.
        Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
        Biometrics. 1988; 44: 837-845
        • Sterne J.A.
        • White I.R.
        • Carlin J.B.
        • et al.
        Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls.
        BMJ. 2009; 338: b2393
        • Smith G.B.
        • Prytherch D.R.
        • Meredith P.
        • Schmidt P.E.
        • Featherstone P.I.
        The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
        Resuscitation. 2013; 84: 465-470
        • Churpek M.M.
        • Snyder A.
        • Han X.
        • et al.
        Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit.
        Am J Respir Crit Care Med. 2017; 195: 906-911
      3. NHS England National Early Warning Score (NEWS). Accessed May 24, 2020 at https://www.england.nhs.uk/ourwork/clinical-policy/sepsis/nationalearlywarningscore/.

        • Pimentel M.A.F.
        • Redfern O.C.
        • Gerry S.
        • et al.
        A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study.
        Resuscitation. 2019; 134: 147-156
      4. Royal College of Physicians NEWS2 and deterioration in COVID-19. Accessed May 24, 2020 at https://www.rcplondon.ac.uk/news/news2-and-deterioration-covid-19.

        • Grasselli G.
        • Zangrillo A.
        • Zanella A.
        • et al.
        Baseline Characteristics and Outcomes of 1591 Patients Infected With SARS-CoV-2 Admitted to ICUs of the Lombardy Region, Italy.
        JAMA. 2020; (In press)
        • Shi H.
        • Han X.
        • Jiang N.
        • et al.
        Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study.
        Lancet Infect Dis. 2020; 20: 425-434
        • Chen T.
        • Wu D.
        • Chen H.
        • et al.
        Clinical characteristics of 113 deceased patients with coronavirus disease 2019: retrospective study.
        BMJ. 2020; 368 (m1091)
      5. United Nations Department of Economic and Social Affairs Population Dynamics. World Population Prospects 2019. Accessed May 24, 2020 at https://population.un.org/wpp/.

        • Ha D.T.
        • Dang T.Q.
        • Tran N.V.
        • Vo N.Y.
        • Nguyen N.D.
        • Nguyen T.V.
        Prognostic performance of the Rapid Emergency Medicine Score (REMS) and Worthing Physiological Scoring system (WPS) in emergency department.
        Int J Emerg Med. 2015; 8: 18
        • Cleek W.R.
        • Johnson N.J.
        • Watsjold B.K.
        • Hall M.K.
        • Henning D.J.
        Comparing Mortality Prediction by Quick Sequential Organ Failure Assessment With Emergency Physician Judgment.
        Shock. 2019;
        • Mellhammar L.
        • Linder A.
        • Tverring J.
        • et al.
        NEWS2 is Superior to qSOFA in Detecting Sepsis with Organ Dysfunction in the Emergency Department.
        J Clin Med. 2019; 8: E1128
        • Zhou F.
        • Yu T.
        • Du R.
        • et al.
        Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
        Lancet. 2020; 395: 1054-1062
        • Nates J.L.
        • Nunnally M.
        • Kleinpell R.
        • et al.
        ICU Admission, Discharge, and Triage Guidelines: A Framework to Enhance Clinical Operations, Development of Institutional Policies, and Further Research.
        Crit Care Med. 2016; 44: 1553-1602