Advertisement

The performance of the National Early Warning Score and National Early Warning Score 2 in hospitalised patients infected by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)

      Abstract

      Introduction

      Since the introduction of the UK’s National Early Warning Score (NEWS) and its modification, NEWS2, coronavirus disease 2019 (COVID-19), has caused a worldwide pandemic. NEWS and NEWS2 have good predictive abilities in patients with other infections and sepsis, however there is little evidence of their performance in COVID-19.

      Methods

      Using receiver-operating characteristics analyses, we used the area under the receiver operating characteristic (AUROC) curve to evaluate the performance of NEWS or NEWS2 to discriminate the combined outcome of either death or intensive care unit (ICU) admission within 24 h of a vital sign set in five cohorts (COVID-19 POSITIVE, n = 405; COVID-19 NOT DETECTED, n = 1716; COVID-19 NOT TESTED, n = 2686; CONTROL 2018, n = 6273; CONTROL 2019, n = 6523).

      Results

      The AUROC values for NEWS or NEWS2 for the combined outcome were: COVID-19 POSITIVE, 0.882 (0.868−0.895); COVID-19 NOT DETECTED, 0.875 (0.861−0.89); COVID-19 NOT TESTED, 0.876 (0.85−0.902); CONTROL 2018, 0.894 (0.884−0.904); CONTROL 2019, 0.842 (0.829−0.855).

      Conclusions

      The finding that NEWS or NEWS2 performance was good and similar in all five cohorts (range = 0.842−0.894) suggests that amendments to NEWS or NEWS2, such as the addition of new covariates or the need to change the weighting of existing parameters, are unnecessary when evaluating patients with COVID-19. Our results support the national and international recommendations for the use of NEWS or NEWS2 for the assessment of acute-illness severity in patients with COVID-19.

      Keywords

      Introduction

      The National Early Warning Score
      • Royal College of Physicians (London)
      National Early Warning Score (NEWS). Standardising the Assessment of Acute-Illness Severity in the NHS.
      (NEWS) and its modification, NEWS2
      • Royal College of Physicians
      National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS.
      , are validated for the identification of patient deterioration in a range of clinical conditions and settings.
      • Smith G.B.
      • Prytherch D.R.
      • Meredith P.
      • Schmidt P.E.
      • Featherstone P.I.
      The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
      • Hydes T.J.
      • Aspinall R.J.
      • Prytherch D.R.
      • Meredith P.
      • Schmidt P.E.
      • Smith G.B.
      National Early Warning Score Accurately Discriminates Risk of serious adverse events in patients with liver disease.
      • Pimentel M.A.F.
      • Redfern O.C.
      • Collins G.S.
      • et al.
      A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: a multi-centre database study.
      • Kovacs C.
      • Jarvis S.W.
      • Prytherch D.R.
      • et al.
      Comparison of the National Early Warning Score in non-elective medical and surgical patients.
      • Scott L.J.
      • Redmond N.M.
      • Tavaré A.
      • Little H.
      • Srivastava S.
      • Pullyblank A.
      Association between National Early Warning Scores in primary care and clinical outcomes: an observational study in UK primary and secondary care.
      • Kim I.
      • Song H.
      • Kim H.J.
      • et al.
      Use of the National Early Warning Score for predicting in-hospital mortality in older adults admitted to the emergency department.
      • Spångfors M.
      • Arvidsson L.
      • Karlsson V.
      • et al.
      The National Early Warning Score: translation, testing and prediction in a Swedish setting.
      • Kivipuro M.
      • et al.
      National early warning score (NEWS) in a Finnish multidisciplinary emergency department and direct vs. late admission to intensive care.
      Both scores are advocated as screening tools for sepsis,
      • Royal College of Physicians
      National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS.
      • Corfield A.R.
      • Lees F.
      • Zealley I.
      • et al.
      Utility of a single early warning score in patients with sepsis in the emergency department.
      • Pullyblank A.
      • Tavaré A.
      • Little H.
      • et al.
      Implementation of the National Early Warning Score in patients with suspicion of sepsis: evaluation of a system-wide quality improvement project.
      • Lim W.T.
      • Fang A.H.
      • Loo C.M.
      • Wong K.S.
      • Balakrinan T.
      Use of the National Early Warning Score (NEWS) to identify acutely deteriorating patients with sepsis in acute medical ward.
      performing better than the quick Sequential (Sepsis-Related) Organ Failure Assessment (qSOFA) score
      • Redfern O.
      • Smith G.B.
      • Prytherch D.
      • Meredith P.
      • Inada-Kim M.
      • Schmidt P.
      A comparison of the quick sequential [Sepsis-related] organ failure assessment score and the National Early Warning Score in non-ICU patients with/without infection.
      • Mellhammar L.
      • Linder A.
      • Tverring J.
      • et al.
      NEWS2 is superior to qSOFA in detecting sepsis with organ dysfunction in the emergency department.
      • Silcock D.J.
      • Corfield A.R.
      • Staines H.
      • Rooney K.D.
      Superior performance of National Early Warning Score compared with quick Sepsis-related Organ Failure Assessment Score in predicting adverse outcomes: a retrospective observational study of patients in the prehospital setting.
      • Usman O.A.
      • Usman O.A.
      • Ward M.A.
      Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the emergency department.
      – a bedside prompt for identifying patients with suspected infection and a high risk of poor outcomes.
      • Singer M.
      • Deutschman C.S.
      • Seymour C.W.
      • et al.
      The third international consensus definitions for sepsis and septic shock (Sepsis-3).
      However, since the introduction of NEWS and NEWS2 in 2012 and 2017, respectively, the world has experienced an ongoing pandemic of a new infectious disease, coronavirus disease 2019 (COVID-19), caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). NEWS2 is currently recommended for use in patients with COVID-19 by the Royal College of Physicians of London (RCP) and the World Health Organisation (WHO),
      • Royal College of Physicians

      World Health Organisation. Clinical Management of Severe Acute Respiratory Infection when COVID-19 is Suspected. Interim Guidance 13 March 2020. https://www.who.int/publications-detail/clinical-management-of-severe-acute-respiratory-infection-when-novel-coronavirus-(ncov)-infection-is-suspected. (Accessed 21 May 2020).

      whilst the National Institute for Health Care and Excellence (NICE) states that it may be useful for predicting the risk of clinical deterioration in COVID-19 pneumonia in the community.

      National Institute for Health and Care Excellence (NICE). COVID-19 Rapid Guideline: Managing Suspected or Confirmed Pneumonia in Adults in the Community. 03 April 2020. https://www.nice.org.uk/guidance/ng165. (Accessed 9 July 2020).

      COVID-19 has a variable clinical presentation on a spectrum from asymptomatic carriage to life-threatening organ dysfunction.
      • Wiersinga W.J.
      • Rhodes A.
      • Cheng A.C.
      • Peacock S.J.
      • Prescott H.C.
      Pathophysiology, transmission, diagnosis, and treatment of coronavirus disease 2019 (COVID-19). A review.
      Many of its clinical features appear to be similar to those seen in other infections, e.g. increased respiratory rate, raised temperature and low systolic blood pressure,
      • Zhou F.
      • Yu T.
      • Du R.
      • et al.
      Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
      • Huang C.
      • Wang Y.
      • Li X.
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
      making the use of either NEWS or NEWS2 appear attractive for screening or monitoring in COVID-19.
      The aim of this study was to evaluate the ability of NEWS or NEWS2 to discriminate the combined outcome of either death or unanticipated intensive care unit (ICU) admission within 24 h of a vital sign set in five patient cohorts – a COVID-19 positive cohort and four control cohorts (one from 2018, one from 2019 and two from 2020).

      Methods

      The study took place in a large NHS acute general hospital that provides acute services to approximately 640,000 people. It has approximately 7900 staff and delivers all acute clinical services except for the management of burns, spinal injury, and neurosurgical and cardiothoracic surgery. The study was performed under existing Isle of Wight, Portsmouth and South East Hampshire Research Ethics Committee approval (ref 08/02/1394).
      In the study hospital, all routine patient vital signs measured at the bedside are documented and charted in real-time in commercially available, electronic software (Vitalpac, The Learning Clinic, London, United Kingdom)
      • Smith G.B.
      • Prytherch D.R.
      • Schmidt P.
      • et al.
      Hospital-wide physiological surveillance–a new approach to the early identification and management of the sick patient.
      running in handheld devices. (Note: The Learning Clinic was acquired by System C Healthcare Limited in 2015 and the Vitalpac product is now marketed within CareFlow Vitals). The software demands that each dataset contains the following: date/time of observation set (automatically recorded by Vitalpac); pulse rate (HR); breathing rate (RR); systolic blood pressure (sBP), body temperature (T); neurological status using either the Alert-Verbal-Painful-Unresponsive (AVPU) scale; and peripheral oxygen saturation (SpO2). A record of the inspired gas (i.e., air or oxygen) being breathed by the patient at the time of SpO2 measurement was also recorded in Vitalpac, as was the patient’s target SpO2 range, which established the correct NEWS2 SpO2 weighting scale for the patient.
      • Royal College of Physicians
      National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS.
      The Vitalpac software has embedded upper/lower limit rules to ensure that highly or extremely abnormal physiological values (out-of-range values) are only accepted after staff have been advised to check the validity of the submitted data and have been given the opportunity to modify the data originally entered. Vitalpac is used throughout the hospital’s general wards but not in the maternity unit or intensive care unit.
      Up until 05/02/2019, the original version of NEWS
      • Royal College of Physicians (London)
      National Early Warning Score (NEWS). Standardising the Assessment of Acute-Illness Severity in the NHS.
      was deployed within Vitalpac. Thereafter, the 2017 modification of NEWS, NEWS2,
      • Royal College of Physicians
      National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS.
      was implemented as handheld devices were updated. NEWS or NEWS2 values were calculated automatically by Vitalpac in accordance with the original guidance from the Royal College of Physicians (RCP) (Supplementary Figs. 1 and 2).
      • Royal College of Physicians (London)
      National Early Warning Score (NEWS). Standardising the Assessment of Acute-Illness Severity in the NHS.
      • Royal College of Physicians
      National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS.
      The structure of NEWS2 differs from that of NEWS by the inclusion of a specific scale for use in patients with hypercapnic respiratory failure (SpO2 Scale 2) and the addition of ‘new confusion’ (C) to the AVPU scale, which becomes ACVPU. However, for most patients, NEWS and NEWS2 are effectively identical. The NEWS or NEWS2 values used in our analyses were those calculated and displayed by the Vitalpac software.
      On 04/05/2020, we extracted a database of vital sign sets for consecutive adults (>16 years) admitted to Portsmouth Hospitals University NHS Trust (PHT) between 01/01/2018 and 04/05/2020, inclusive. The following were excluded: (a) data from patients discharged alive from hospital before midnight on the day of admission, (b) vital sign sets where measurements were outside the following limits: SpO2 50–100%; temperature 30–42 °C; pulse/heart rate 30−200 bpm; sBP 0−300 mmHg; respiratory rate 0–70 bpm, and (c) vital sign sets for which one or more parameter measurements were missing (because a full set of data is required to calculate NEWS and NEWS2 values). Additionally, in the current analysis, patients transferred directly at admission to a critical care area and those with no vital sign sets within the last 24 h before the primary outcome were also excluded.
      We studied five admission cohorts, based on the finding that the first reverse-transcriptase–polymerase-chain-reaction (RT-PCR) test that was positive for SARS-CoV-2 for an inpatient in the study hospital was sampled on 11/03/2020:
      • COVID-19 POSITIVE: Inpatients with vital sign sets between 11/03/2020 and 30/4/2020 who had a positive RT-PCR test. For the purposes of this study, only vital signs on or after the date of the patient’s first positive RT-PCR test were used.
      • COVID-19 NOT DETECTED: Inpatients with vital sign sets between 11/03/2020 and 30/4/2020 who had a negative RT-PCR test and never had a positive or indeterminate one.
      • COVID-19 NOT TESTED: Inpatients with vital sign sets between 11/03/2020 and 30/4/2020 who never had a RT-PCR test.
      • CONTROL 2018: Inpatients with vital sign sets measured between 11/03/2018 and 30/4/2018.
      • CONTROL 2019: Inpatients with vital sign sets measured between 11/03/2019 and 30/4/2019.
      We did not include those patients who only had an indeterminate test result, or where there were sampling or other technical failures (n = 53).
      The primary outcome studied was the combination of death or intensive care unit (ICU) admission within 24 h of a vital sign set. These were identified from the hospital’s patient administration system (PAS) and the ICU admission database, respectively.

      Data storage and analysis

      All data were stored in Microsoft SQL Server 2019, and analysed using R v3.6.0 statistical computing and graphics software.
      • Core Team R.
      R: A Language and Environment for Statistical Computing.
      Descriptive statistics were calculated including counts, means (±SD), medians (IQR, Q1-Q3) and proportions. Proportions were compared using the chi squared test with Bonferroni correction for multiple comparisons. Mean values were compared with one-way ANOVA and Scheffe's test, and median values were compared using Kruskal–Wallis and Dunn's test (Bonferroni correction).
      Analyses were undertaken using either NEWS or NEWS2 values, whichever was provided by Vitalpac. Using the area under the receiver-operating characteristics (AUROC) curve,
      • Hanley J.A.
      • McNeil B.J.
      The meaning and use of the area under a receiver operating characteristic (ROC) curve.
      we evaluated the ability of NEWS or NEWS2 to discriminate the combination of death or ICU admission within 24 h of a vital sign set in the each of the five cohorts previously described. We considered AUROC values of 0.700 to 0.800 to represent ‘reasonable’ discrimination; AUROC values exceeding 0.800 were considered to represent ‘good’ discrimination.

      Results

      After exclusions, the main data extract for the period 01/01/2018 to 04/05/2020 contained 2,867,313 vital sign sets from 97,669 admissions (60,436 discrete patients). Fig. 1 shows the development of the datasets for each of the five study cohorts (COVID-19 POSITIVE, COVID-19 NOT DETECTED, COVID-19 NOT TESTED, CONTROL 2018 and CONTROL 2019) and Table 1 presents the details of the admissions and vital sign sets. The categorisation of admissions to medical, surgical and other groups is shown in Supplementary Table 1. Admissions in the COVID-19 POSITIVE and COVID-19 NOT DETECTED cohorts were older than in the others (p<0.001). Those in the COVID-19 POSITIVE cohort were also more likely to be male (<0.001). There were also significant differences in some of the vital signs (notably, higher median RR and mean temperature, and lower median SpO2) and a higher median NEWS during the stay. More patients in the COVID-19 POSITIVE cohort received supplemental oxygen whilst in-hospital (<0.001).
      Fig. 1
      Fig. 1Development of the datasets for each of the five study cohorts.
      Table 1Demographics data for admissions in the five study cohorts.
      CONTROL 2018

      (a)
      CONTROL 2019

      (b)
      COVID-19 POSITIVE (c)COVID-19 NOT DETECTED (d)COVID-19 NOT TESTED

      (e)
      p-value all groupspost-hoc tests p-value
      (c) vs. (a)(c) vs. (b)(c) vs. (d)(c) vs. (e)
      Admissions
      Admissions, N6273652340517162686
      Patients, N5658588340016192505
      Admissions to Medical specialties, N (%)4088 (65.2%)4142 (63.5%)380 (93.8%)1435 (83.6%)1712 (63.7%)<0.001<0.001<0.001<0.001<0.001
      Admissions to Surgical specialties, N (%)1847 (29.4%)2000 (30.7%)22 (5.4%)260 (15.2%)877 (32.7%)<0.001<0.001<0.001<0.001<0.001
      Admissions to Other specialties, N (%)338 (5.4%)381 (5.8%)3 (0.7%)21 (1.2%)97 (3.6%)<0.001<0.001<0.0011.0000.008
      Emergency admissions, N (%)5290 (84.3%)5476 (83.9%)400 (98.8%)1681 (98%)2316 (86.2%)<0.001<0.001<0.0011.000<0.001
      Elective admissions, N (%)983 (15.7%)1047 (16.1%)5 (1.2%)35 (2%)370 (13.8%)<0.001<0.001<0.0011.000<0.001
      Age at admission [years], median (IQR)71 (55−82)72 (55−83)75 (62−85)74 (58−84)71 (55−82)<0.001<0.001<0.001<0.001<0.001
      Males, N (%)2945 (46.9%)3045 (46.7%)234 (57.8%)837 (48.8%)1277 (47.5%)0.001<0.001<0.0010.0110.001
      Patients receiving oxygen during stay, N (%)1928 (34.1%)1856 (31.5%)261 (65.3%)665 (41.1%)485 (19.4%)<0.001<0.001<0.001<0.001<0.001
      Patients scored using NEWS2, N (%)0 (0)%5814 (98.8%)400 (100%)1619 (100%)2505 (100%)<0.001<0.0010.2171.0001.000
      Patients scored using Scale 2 of NEWS2, N (%)n/a106 (1.8%)12 (3.0%)39 (2.4%)20 (0.8%)<0.001n/a0.7481.0000.004
      All observations
      No of observations, N150,581163,64114,70344,96747,042
      HR [bpm], mean (SD)79.9 (15.3)79.8 (15.1)81 (14.7)81.8 (15.5)78.1 (14.4)<0.001<0.001<0.001<0.001<0.001
      RR [1/min], median (IQR)17 (16−18)17 (16−18)18 (17−20)18 (16−19)17 (16−18)<0.001<0.001<0.001<0.001<0.001
      sBP [mmHg], mean (SD)126.2 (22.6)126.4 (22.1)127.2 (22.2)126.7 (22.1)126.8 (21.5)<0.001<0.0010.0060.3030.589
      Temperature [oC], mean (SD)36.8 (0.5)36.7 (0.4)36.9 (0.6)36.8 (0.5)36.7 (0.4)<0.001<0.001<0.001<0.001<0.001
      SpO2 [%], median (IQR)96 (95−98)96 (95−98)95 (94−97)96 (95−97)96 (95−98)<0.001<0.001<0.001<0.001<0.001
      Supplemental O2, N (%)25,313 (17%)25,018 (15%)6347 (43%)11,191 (25%)4889 (10%)<0.001<0.001<0.001<0.001<0.001
      AVPU - Alert, N (%)149,787 (99.5%)162,707 (99.4%)14,599 (99.3%)44,588 (99.2%)46,947 (99.8%)<0.0010.0670.4141.000<0.001
      AVPU - Responds to voice, N (%)541 (0.4%)496 (0.3%)80 (0.5%)219 (0.5%)64 (0.1%)<0.0010.009<0.0011.000<0.001
      AVPU - Responds to pain, N (%)132 (0.1%)148 (0.1%)10 (0.1%)73 (0.2%)9 (0%)<0.0011.0001.0000.0690.062
      AVPU - Unresponsive, N (%)92 (0.1%)82 (0.1%)4 (0%)43 (0.1%)3 (0%)<0.0011.0001.0000.0990.601
      NEWS/NEWS2, median (IQR)1 (0−2)1 (0−2)2 (1−4)1 (0−3)1 (0−2)<0.001<0.001<0.001<0.001<0.001
      Observation sets followed by combined outcome, N (%)1240 (0.8%)1194 (0.7%)492 (3.3%)724 (1.6%)279 (0.6%)<0.001<0.001<0.001<0.001<0.001
      HR = heart/pulse rate; RR = respiratory rate; sBP = systolic blood pressure SpO2 = peripheral oxygen saturation; NEWS = National Early Warning Score; NEWS2 = National Early Warning Score 2.
      Count data has been tested for significance using Fisher's exact test with Bonferroni correction for pairwise comparison.
      Percentages were compared using the chi squared test with Bonferroni correction.
      Mean values were compared with one-way ANOVA and Scheffe's test.
      Median values were compared using Kruskal–Wallis and Dunn's test (Bonferroni correction).
      IQR = Interquartile range (Q1–Q3).
      Fig. 2 shows the distribution of NEWS or NEWS2 values for patients in the five cohorts and Fig. 3 demonstrates an increasing risk of the combined outcome with increasing NEWS/NEWS2 value in all five cohorts. Confidence intervals are not shown on Fig. 3 to improve clarity; however, these are available in Supplementary Fig. 3 and Supplementary Table 2.
      Fig. 2
      Fig. 2Distribution of NEWS or NEWS2 values for the five admission cohorts.
      Fig. 3
      Fig. 3The relationship between NEWS or NEWS2 value and the risk of the combined outcome in all five cohorts.
      The AUROC values for NEWS or NEWS2 for the combined outcome were: COVID-19 POSITIVE, 0.882 (0.868−0.895); COVID-19 NOT DETECTED, 0.875 (0.861−0.89); COVID-19 NOT TESTED, 0.876 (0.85−0.902); CONTROL 2018, 0.894 (0.884−0.904); CONTROL 2019, 0.842 (0.829−0.855).
      Fig. 4 shows the AUROC values for NEWS or NEWS2 for the discrimination of the combined outcome plotted through time on a monthly basis from 01/01/2018 to 31/12/2019. Across the two years there was clear seasonal variation but all monthly AUROC values were within the range 0.830–0.895.
      Fig. 4
      Fig. 4Monthly area under the receiver operating characteristic (AUROC) curve values from 01/01/2018 to 31/12/2019 for NEWS or NEWS2 and the combined outcome.

      Discussion

      The study demonstrates that NEWS or NEWS2 are good discriminators of the combined outcome of either death or ICU admission within 24 h of a vital sign set in patients with a RT-PCR test result that is positive for SARS-CoV-2. Indeed, there was very little difference between the AUROC values for admissions in the COVID-19 POSITIVE cohort compared to any of our other study cohorts, with values ranging from 0.842 to 0.894.
      The results also demonstrate that admissions in the COVID-19 POSITIVE cohort were older than in the other groups and more likely to be male, which appear to be common findings in COVID-19.
      • Wiersinga W.J.
      • Rhodes A.
      • Cheng A.C.
      • Peacock S.J.
      • Prescott H.C.
      Pathophysiology, transmission, diagnosis, and treatment of coronavirus disease 2019 (COVID-19). A review.
      Vital sign values were also more abnormal in this cohort, the EWS quartiles were higher and more patients received oxygen therapy.
      In the CONTROL 2019 cohort we found an AUROC value of 0.842 (0.829−0.855), which is significantly lower than all other cohorts except the COVID-19 NOT TESTED cohort. Given the limited demographic data available to us and the pattern of the AUROC values across the period 01/01/2018–31/12/2019 (Fig. 4), we believe that this could reflect the influence of differing pressures on clinical services. However, it is also possible that the change from NEWS to NEWS2 from 06/02/2019 onwards may have contributed, as particular elements of implementing NEWS2, including the use of a specific scale for use in patients with hypercapnic respiratory failure, have proven challenging.
      • Royal College of Physicians
      National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS. Additional Implementation Guidance Update: March.
      Interestingly, the AUROC value for the COVID-19 NOT DETECTED cohort was the same as that for the COVID-19 POSITIVE group of admissions. However, the percentage of observation sets followed by the combined outcome in the COVID-19 NOT DETECTED cohort was half that in the COVID-19 POSITIVE cohort. In most cases in the COVID-19 NOT DETECTED cohort, the RT-PCR test was performed on the basis of symptoms and signs, and a presumed diagnosis of COVID-19 infection. However, the high false negative rate observed for RT-PCR tests for SARS-CoV-2
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.
      could mean that many of the patients in the COVID-19 NOT DETECTED cohort were actually positive for COVID-19 infection. Perhaps a lower viral load in the COVID-19 NOT DETECTED cohort leads to a lower chance of SARS-CoV-2 detection and a lower risk of the combined outcome.
      • Liu Y.
      • Yan L.-M.
      • Wan L.
      • et al.
      Viral dynamics in mild and severe cases of COVID-19.
      Major strengths of this study are that it considers all patient admissions to the study hospital during the first two months of the COVID-19 outbreak. It draws upon a large dataset of vital signs collected simultaneously in a standardized manner during clinical care using an electronic data collecting system. It also considers a repeatable, clinically and operationally useful, objective combined outcome, for which data were easily retrievable from the hospital’s electronic records.
      However, there are also weaknesses. The RT-PCR test for SARS-CoV-2 is not 100% accurate and negative results need to be interpreted with caution.
      • Wiersinga W.J.
      • Rhodes A.
      • Cheng A.C.
      • Peacock S.J.
      • Prescott H.C.
      Pathophysiology, transmission, diagnosis, and treatment of coronavirus disease 2019 (COVID-19). A review.
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.
      Therefore, as mentioned earlier, there is a risk that the COVID-19 NOT DETECTED cohort may have been contaminated by false-negative test results. In addition, there is also a chance, albeit much smaller, that the COVID-19 POSITIVE cohort includes false-positive results. On the basis that the first known UK case of COVID-19 occurred on 29/01/2020, we used control groups from 2018 and 2019 to permit comparisons with the COVID-19 era. However, uncertainty still exists about the location and date of the index case of COVID-19 and this may have relevance to our use of the control group from 2019 for comparisons with the COVID-19 POSITIVE cohort.
      We excluded admissions that were admitted directly to critical care areas of the hospital. Our logically necessary exclusion of all those with no vital sign sets in the last 24 h also had the effect of removing any patient on an End-of-Life (EoL) pathway where vital sign monitoring had ceased. We obtained the patient’s date and time of death and/or ICU admission from the hospital’s PAS and ICU admission database, respectively. These data are likely to be systematically late, implying that the number of observations followed by the combined outcome may be an underestimate.
      This is a single centre study such that the results are not necessarily transferrable and require external validation. We used repeated observation sets from the same patient episode in the analysis, making the assumption that the observation sets are independent based on previous work by our group.
      • Jarvis S.W.
      • Kovacs C.
      • Briggs J.S.
      • et al.
      Are vital signs sets independent? Effects of observation selection methods on early warning score performance.
      The conclusions of our work are also based on the assumption that all patients were treated optimally and equitably. A further weakness is that the study was conducted in a single site, where the precursor of NEWS, ViEWS,
      • Prytherch D.R.
      • Smith G.B.
      • Schmidt P.E.
      • Featherstone P.I.
      ViEWS–towards a national early warning score for detecting adult inpatient deterioration.
      was developed. Prediction models are almost always more accurate in the population in which they were derived. Finally, this is a statistical evaluation of NEWS or NEWS2 during the COVID-19 pandemic and there is no guarantee that similar results would be generated operationally.
      As SARS-CoV-2 emerged relatively recently, there are only currently a small number of publications regarding the performance of NEWS and NEWS2 in patients with COVID-19. Gidari et al. showed that the predictive value of admission NEWS2 values for ICU admission in 68 patients with severe COVID-19 was good (AUROC = 0.90 (CI, 0.82–0.97)).
      • Gidari A.
      • De Socio G.V.
      • Sabbatini S.
      • Francisci D.
      Predictive value of National Early Warning Score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection.
      Peng et al. showed similar AUROC values for predicting ‘serious events’ of 0.837 (0.748−0.943) for admission NEWS and 0.846 (0.735−0.939) for admission NEWS modified by age, where ‘serious events’ were defined as any of the following during hospitalisation: death, unplanned transfer to an ICU or initiation of non-invasive ventilation.
      • Peng X.
      • Subbe C.P.
      • Zhang L.
      • Luo Z.
      • Peng L.
      NEWS can predict deterioration of patients with COVID-19.
      Covino et al. reported that NEWS was the most accurate predictor of ICU admission within 48 h (AUROC = 0.802 (0.756−0.844)) and 7 days (AUROC 0.783 (0.735−0.826)) of emergency department arrival.
      • Covino M.
      • Sandroni C.
      • Santoro M.
      • et al.
      Predicting intensive care unit admission and death for COVID-19 patients in the emergency department using early warning scores.
      Liu et al. found that NEWS 0.882 (0.847–0.916) and NEWS2 0.880 (0.845–0.914) had the highest AUROC values for predicting in-hospital death.
      • Liu F.-Y.
      • Sun X.-L.
      • Zhang Y.
      • et al.
      Evaluation of the risk prediction tools for patients with coronavirus disease 2019 in Wuhan, China. A single-centered, retrospective, observational study.
      However, Knight et al. showed an AUROC of 0.654 (0.645–0.662) for NEWS with in-hospital mortality as an outcome.
      • Knight S.R.
      • Ho A.
      • Pius R.
      • et al.
      Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C Mortality Score.
      Our study provides data from a large cohort of unselected hospital admissions and focuses on an outcome measure that is both clinically relevant and has the potential to be averted with the appropriate and timely clinical intervention. Unlike other journal publications,
      • Gidari A.
      • De Socio G.V.
      • Sabbatini S.
      • Francisci D.
      Predictive value of National Early Warning Score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection.
      • Peng X.
      • Subbe C.P.
      • Zhang L.
      • Luo Z.
      • Peng L.
      NEWS can predict deterioration of patients with COVID-19.
      • Covino M.
      • Sandroni C.
      • Santoro M.
      • et al.
      Predicting intensive care unit admission and death for COVID-19 patients in the emergency department using early warning scores.
      • Liu F.-Y.
      • Sun X.-L.
      • Zhang Y.
      • et al.
      Evaluation of the risk prediction tools for patients with coronavirus disease 2019 in Wuhan, China. A single-centered, retrospective, observational study.
      • Knight S.R.
      • Ho A.
      • Pius R.
      • et al.
      Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C Mortality Score.
      it uses NEWS and NEWS2 values from the whole hospital admission and compares the performance of the scores in patients who had a positive RT-PCR test result for SARS-CoV-2 with several other patient groups in the same hospital that were, as far as can be determined, not suffering from a COVID-19 illness. This has permitted an assessment of the need for any amendment to NEWS or NEWS2, such as the addition of age
      • Peng X.
      • Subbe C.P.
      • Zhang L.
      • Luo Z.
      • Peng L.
      NEWS can predict deterioration of patients with COVID-19.
      • Liao X.
      • Wang B.
      • Kang Y.
      Novel coronavirus infection during the 2019-2020 epidemic: preparing intensive care units-the experience in Sichuan Province, China.
      or other covariates or a change in the weighting of existing parameters,
      • Lim N.T.
      • Pan D.
      • Barker J.
      NEWS2 system requires modification to identify deteriorating patients with COVID-19.
      when evaluating patients with COVID-19. The finding that NEWS or NEWS2 performance was good and similar in all five cohorts (range = 0.842−0.894) suggests that such changes are unnecessary and supports the continued use of NEWS or NEWS2 role as systems for the assessment of acute-illness severity in hospitalised patients irrespective of the underlying condition.
      • Smith G.B.
      • Prytherch D.R.
      • Meredith P.
      • Schmidt P.E.
      • Featherstone P.I.
      The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
      • Hydes T.J.
      • Aspinall R.J.
      • Prytherch D.R.
      • Meredith P.
      • Schmidt P.E.
      • Smith G.B.
      National Early Warning Score Accurately Discriminates Risk of serious adverse events in patients with liver disease.
      • Pimentel M.A.F.
      • Redfern O.C.
      • Collins G.S.
      • et al.
      A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: a multi-centre database study.
      • Kovacs C.
      • Jarvis S.W.
      • Prytherch D.R.
      • et al.
      Comparison of the National Early Warning Score in non-elective medical and surgical patients.
      • Scott L.J.
      • Redmond N.M.
      • Tavaré A.
      • Little H.
      • Srivastava S.
      • Pullyblank A.
      Association between National Early Warning Scores in primary care and clinical outcomes: an observational study in UK primary and secondary care.
      • Kim I.
      • Song H.
      • Kim H.J.
      • et al.
      Use of the National Early Warning Score for predicting in-hospital mortality in older adults admitted to the emergency department.
      • Spångfors M.
      • Arvidsson L.
      • Karlsson V.
      • et al.
      The National Early Warning Score: translation, testing and prediction in a Swedish setting.
      • Kivipuro M.
      • et al.
      National early warning score (NEWS) in a Finnish multidisciplinary emergency department and direct vs. late admission to intensive care.
      In addition, the results demonstrate that NEWS and NEWS2 remain highly relevant for illness severity assessment, even for patients suffering a completely novel disease that has emerged some years after the two scoring systems were developed. Previous research into the performance of NEWS or NEWS2 for 24 h outcomes shows similar AUROC values to those found in the current study.
      • Royal College of Physicians
      National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS.
      • Smith G.B.
      • Prytherch D.R.
      • Meredith P.
      • Schmidt P.E.
      • Featherstone P.I.
      The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
      • Hydes T.J.
      • Aspinall R.J.
      • Prytherch D.R.
      • Meredith P.
      • Schmidt P.E.
      • Smith G.B.
      National Early Warning Score Accurately Discriminates Risk of serious adverse events in patients with liver disease.
      • Pimentel M.A.F.
      • Redfern O.C.
      • Collins G.S.
      • et al.
      A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: a multi-centre database study.
      • Kovacs C.
      • Jarvis S.W.
      • Prytherch D.R.
      • et al.
      Comparison of the National Early Warning Score in non-elective medical and surgical patients.
      • Lim W.T.
      • Fang A.H.
      • Loo C.M.
      • Wong K.S.
      • Balakrinan T.
      Use of the National Early Warning Score (NEWS) to identify acutely deteriorating patients with sepsis in acute medical ward.
      • Mellhammar L.
      • Linder A.
      • Tverring J.
      • et al.
      NEWS2 is superior to qSOFA in detecting sepsis with organ dysfunction in the emergency department.
      Consequently, our results support the continued adherence to the national and international guidance on the use of the systems in patients with COVID-19.

      World Health Organisation. Clinical Management of Severe Acute Respiratory Infection when COVID-19 is Suspected. Interim Guidance 13 March 2020. https://www.who.int/publications-detail/clinical-management-of-severe-acute-respiratory-infection-when-novel-coronavirus-(ncov)-infection-is-suspected. (Accessed 21 May 2020).

      National Institute for Health and Care Excellence (NICE). COVID-19 Rapid Guideline: Managing Suspected or Confirmed Pneumonia in Adults in the Community. 03 April 2020. https://www.nice.org.uk/guidance/ng165. (Accessed 9 July 2020).

      • Wiersinga W.J.
      • Rhodes A.
      • Cheng A.C.
      • Peacock S.J.
      • Prescott H.C.
      Pathophysiology, transmission, diagnosis, and treatment of coronavirus disease 2019 (COVID-19). A review.
      Of course, demonstration that NEWS and NEWS2 discriminate adverse outcomes well does not imply that the important, but separate, issues of when and how care is escalated may not require modification.
      • Smith G.B.
      • Prytherch D.R.
      • Schmidt P.E.
      • Meredith P.
      Early warning scores: unravelling detection and escalation.
      The results of our study require validation, if possible, in a larger population. In addition, future research could investigate the hypothesis that, because both NEWS and NEWS2 use a binary weighting for oxygen supplementation [air = 0, oxygen = 2 points], they might not detect sudden and rapid respiratory decompensation if a patient who is already receiving supplemental oxygen develops a rapid increase in oxygen requirement without additional changes in other physiological parameters.
      • Royal College of Physicians

      Conclusions

      The finding that NEWS or NEWS2 performance was good and similar in all five cohorts (range = 0.842−0.894) suggests that amendments to NEWS or NEWS2, such as the addition of new covariates or the need to change the weighting of existing parameters, are unnecessary when evaluating patients with COVID-19. Our results support the national and international recommendations for the use of NEWS or NEWS2 for the assessment of acute-illness severity in patients with COVID-19.

      Funding

      None.

      Conflict of interest

      The following potential conflicts of interest are declared by the authors and the other members of the PACIFIC-19 team. Professor Chauhan, Dr Meredith, Dr Schmidt, Dr Spice, Dr Fox, Dr Mortlock, Dr Fleming, Dr Pilbeam, Dr Rowley and Dr Poole are employees of Portsmouth Hospitals University NHS Trust (PHT), which had a royalty agreement with The Learning Clinic (TLC) to pay for the use of PHT intellectual property within the Vitalpac product that expired before the commencement of this study. Professors Prytherch and Smith are former employees of PHT. Dr Schmidt, and the wives of Professors Prytherch and Smith, held shares in TLC until October 2015. Dr Schmidt and Professors Smith and Prytherch were unpaid research advisors to TLC and received reimbursement of travel expenses from TLC for attending symposia in the United Kingdom. Professor Briggs’ research has previously received funding from TLC through a Knowledge Transfer Partnership. Professor Smith was a topic expert for the National Institute for Clinical and Health Excellence’s clinical guideline surveillance process for Clinical Guideline 50 (Acutely ill patients in hospital. Recognition of and response to acute illness in adults in hospital) in 2007 and 2016, and a member of the Royal College of Physicians of London’s National Early Warning Score (NEWS) Development and Implementation Group, which developed NEWS and NEWS2. Dr Kostakis, Mr Price, Dr Scott, Dr Mortlock, Dr Spice, Dr Fox, Dr Fleming, Dr Pilbeam, Dr Rowley and Dr Poole declare no conflict of interests.

      THE PACIFIC-19 team: the PACIFIC-19 team comprises

      • Professor Anoop Chauhan, Executive Director of Research and Innovation and Consultant Respiratory Physician, Portsmouth Hospitals University NHS Trust, Portsmouth, UK, and Professor of Respiratory Medicine, University of Portsmouth, Portsmouth, UK.
      • Dr Paul Meredith, Data Analyst, Research & Innovation Department, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Alice Mortlock, Head of Research, Research & Innovation Department, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Paul Schmidt, Consultant in Acute Medicine, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Claire Spice, Consultant in Medicine for Older People, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Lauren Fox, Respiratory Research Fellow, Research & Innovation Department, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Daniel Fleming, Respiratory Foundation Year Doctor, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Lara Pilbeam, Respiratory Foundation Year Doctor, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Megan Rowley, Respiratory Core Medical Trainee, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Dr Hannah Poole, Respiratory Foundation Year Doctor, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Professor Jim Briggs, Professor of Informatics, Centre for Healthcare Modelling & Informatics, School of Computing, University of Portsmouth, Portsmouth, UK.
      • Professor David Prytherch, Professor of Health Informatics, Centre for Healthcare Modelling & Informatics, School of Computing, University of Portsmouth, Portsmouth, UK.
      • Dr Ina Kostakis, Research Fellow, Centre for Healthcare Modelling & Informatics, School of Computing, University of Portsmouth, Portsmouth, UK.
      • Mr Connor Price, PhD student, Centre for Healthcare Modelling & Informatics, School of Computing, University of Portsmouth, Portsmouth, UK.
      • Dr Philip Scott, Reader in Health Informatics, Centre for Healthcare Modelling & Informatics, School of Computing, University of Portsmouth, Portsmouth, UK and Digital Healthcare Architect, Portsmouth Hospitals University NHS Trust, Portsmouth, UK.
      • Professor Gary B Smith, Centre of Postgraduate Medical Research & Education (CoPMRE), Faculty of Health and Social Sciences, Bournemouth University, Bournemouth, UK.

      CRediT authorship contribution statement

      Ina Kostakis: Methodology, Software, Validation, Formal analysis, Investigation, Writing - review & editing, Visualization. Gary B. Smith: Conceptualization, Methodology, Validation, Investigation, Writing - original draft, Validation, Formal analysis, Investigation, Visualization, Project administration. David Prytherch: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - review & editing, Supervision. Paul Meredith: Conceptualization, Methodology, Software, Validation, Formal analysis, Data curation, Writing - review & editing. Connor Price: Software, Formal analysis, Validation, Writing - review & editing. Anoop Chauhan: Conceptualization, Methodology, Writing - review & editing, Supervision. Anoop Chauhan: . Paul Meredith: . Alice Mortlock: . Paul Schmidt: . Claire Spice: . Lauren Fox: . Daniel Fleming: . Lara Pilbeam: . Megan Rowley: . Hannah Poole: . Jim Briggs: . David Prytherch: . Ina Kostakis: . Connor Price: . Philip Scott: . Gary B. Smith: .

      Acknowledgements

      All authors have made substantial contributions to all of the following: (1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.

      Appendix A. Supplementary data

      The following is Supplementary data to this article:

      References

        • Royal College of Physicians (London)
        National Early Warning Score (NEWS). Standardising the Assessment of Acute-Illness Severity in the NHS.
        Report of a working party, 2012 (July)
        • Royal College of Physicians
        National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS.
        Updated report of a working party, 2017 (December)
        • Smith G.B.
        • Prytherch D.R.
        • Meredith P.
        • Schmidt P.E.
        • Featherstone P.I.
        The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
        Resuscitation. 2013; 84: 465-470
        • Hydes T.J.
        • Aspinall R.J.
        • Prytherch D.R.
        • Meredith P.
        • Schmidt P.E.
        • Smith G.B.
        National Early Warning Score Accurately Discriminates Risk of serious adverse events in patients with liver disease.
        Clin Gastroenterol Hepatol. 2018; 16: 1657-1666
        • Pimentel M.A.F.
        • Redfern O.C.
        • Collins G.S.
        • et al.
        A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: a multi-centre database study.
        Resuscitation. 2019; 134: 147-156
        • Kovacs C.
        • Jarvis S.W.
        • Prytherch D.R.
        • et al.
        Comparison of the National Early Warning Score in non-elective medical and surgical patients.
        BJS. 2016; 103: 1385-1393
        • Scott L.J.
        • Redmond N.M.
        • Tavaré A.
        • Little H.
        • Srivastava S.
        • Pullyblank A.
        Association between National Early Warning Scores in primary care and clinical outcomes: an observational study in UK primary and secondary care.
        Br J Gen Pract. 2020; (Accessed 6 May 2020)https://doi.org/10.3399/bjgp20X709337
        • Kim I.
        • Song H.
        • Kim H.J.
        • et al.
        Use of the National Early Warning Score for predicting in-hospital mortality in older adults admitted to the emergency department.
        Clin Exp Emerg Med. 2020; 7: 61-66
        • Spångfors M.
        • Arvidsson L.
        • Karlsson V.
        • et al.
        The National Early Warning Score: translation, testing and prediction in a Swedish setting.
        Intensive Crit Care Nurs. 2016; 37: 62-67
        • Kivipuro M.
        • et al.
        National early warning score (NEWS) in a Finnish multidisciplinary emergency department and direct vs. late admission to intensive care.
        Resuscitation. 2018; 128: 164-169
        • Corfield A.R.
        • Lees F.
        • Zealley I.
        • et al.
        Utility of a single early warning score in patients with sepsis in the emergency department.
        EMJ. 2014; 31: 482-487
        • Pullyblank A.
        • Tavaré A.
        • Little H.
        • et al.
        Implementation of the National Early Warning Score in patients with suspicion of sepsis: evaluation of a system-wide quality improvement project.
        Br J Gen Pract. 2020; https://doi.org/10.3399/bjgp20X709349
        • Lim W.T.
        • Fang A.H.
        • Loo C.M.
        • Wong K.S.
        • Balakrinan T.
        Use of the National Early Warning Score (NEWS) to identify acutely deteriorating patients with sepsis in acute medical ward.
        Ann Acad Med Singapore. 2019; 48: 145-149
        • Redfern O.
        • Smith G.B.
        • Prytherch D.
        • Meredith P.
        • Inada-Kim M.
        • Schmidt P.
        A comparison of the quick sequential [Sepsis-related] organ failure assessment score and the National Early Warning Score in non-ICU patients with/without infection.
        Crit Care Med. 2018; 46: 1923-1933
        • Mellhammar L.
        • Linder A.
        • Tverring J.
        • et al.
        NEWS2 is superior to qSOFA in detecting sepsis with organ dysfunction in the emergency department.
        J Clin Med. 2019; 8: 1128
        • Silcock D.J.
        • Corfield A.R.
        • Staines H.
        • Rooney K.D.
        Superior performance of National Early Warning Score compared with quick Sepsis-related Organ Failure Assessment Score in predicting adverse outcomes: a retrospective observational study of patients in the prehospital setting.
        Eur J Emerg Med. 2019; 26: 433-439https://doi.org/10.1097/MEJ.0000000000000589
        • Usman O.A.
        • Usman O.A.
        • Ward M.A.
        Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the emergency department.
        Am J Emerg Med. 2019; 37: 1490-1497
        • Singer M.
        • Deutschman C.S.
        • Seymour C.W.
        • et al.
        The third international consensus definitions for sepsis and septic shock (Sepsis-3).
        JAMA. 2016; 315: 801-810
        • Royal College of Physicians
        NEWS2 and Deterioration in COVID-19.
        (Accessed 21 May 2020)
      1. World Health Organisation. Clinical Management of Severe Acute Respiratory Infection when COVID-19 is Suspected. Interim Guidance 13 March 2020. https://www.who.int/publications-detail/clinical-management-of-severe-acute-respiratory-infection-when-novel-coronavirus-(ncov)-infection-is-suspected. (Accessed 21 May 2020).

      2. National Institute for Health and Care Excellence (NICE). COVID-19 Rapid Guideline: Managing Suspected or Confirmed Pneumonia in Adults in the Community. 03 April 2020. https://www.nice.org.uk/guidance/ng165. (Accessed 9 July 2020).

        • Wiersinga W.J.
        • Rhodes A.
        • Cheng A.C.
        • Peacock S.J.
        • Prescott H.C.
        Pathophysiology, transmission, diagnosis, and treatment of coronavirus disease 2019 (COVID-19). A review.
        JAMA. 2020; 324: 782-793https://doi.org/10.1001/jama.2020.12839
        • Zhou F.
        • Yu T.
        • Du R.
        • et al.
        Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
        Lancet. 2020; (published online March 9)https://doi.org/10.1016/S0140-6736(20)30566-3
        • Huang C.
        • Wang Y.
        • Li X.
        • et al.
        Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
        Lancet. 2020; 395: 497-506
        • Smith G.B.
        • Prytherch D.R.
        • Schmidt P.
        • et al.
        Hospital-wide physiological surveillance–a new approach to the early identification and management of the sick patient.
        Resuscitation. 2006; 71: 19-29
        • Core Team R.
        R: A Language and Environment for Statistical Computing.
        (Available online at) R Foundation for Statistical Computing, Vienna, Austria2019
        • Hanley J.A.
        • McNeil B.J.
        The meaning and use of the area under a receiver operating characteristic (ROC) curve.
        Radiology. 1982; 143: 29-36
        • Royal College of Physicians
        National Early Warning Score (NEWS) 2. Standardising the Assessment of Acute-Illness Severity in the NHS. Additional Implementation Guidance Update: March.
        2020
        • Woloshin S.
        • Patel N.
        • Kesselheim A.S.
        False negative tests for SARS-CoV-2 infection—challenges and implications.
        NEJM. 2020; https://doi.org/10.1056/nejmp2015897
        • Liu Y.
        • Yan L.-M.
        • Wan L.
        • et al.
        Viral dynamics in mild and severe cases of COVID-19.
        Lancet Infect Dis. 2020; 20: 656-657
        • Jarvis S.W.
        • Kovacs C.
        • Briggs J.S.
        • et al.
        Are vital signs sets independent? Effects of observation selection methods on early warning score performance.
        Resuscitation. 2015; 90: 1-6
        • Prytherch D.R.
        • Smith G.B.
        • Schmidt P.E.
        • Featherstone P.I.
        ViEWS–towards a national early warning score for detecting adult inpatient deterioration.
        Resuscitation. 2010; 81: 932-937
        • Gidari A.
        • De Socio G.V.
        • Sabbatini S.
        • Francisci D.
        Predictive value of National Early Warning Score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection.
        Infect Dis. 2020; 52 (2020): 698-704
        • Peng X.
        • Subbe C.P.
        • Zhang L.
        • Luo Z.
        • Peng L.
        NEWS can predict deterioration of patients with COVID-19.
        Resuscitation. 2020; 152: 26-27
        • Covino M.
        • Sandroni C.
        • Santoro M.
        • et al.
        Predicting intensive care unit admission and death for COVID-19 patients in the emergency department using early warning scores.
        Resuscitation. 2020; 156: 84-91
        • Liu F.-Y.
        • Sun X.-L.
        • Zhang Y.
        • et al.
        Evaluation of the risk prediction tools for patients with coronavirus disease 2019 in Wuhan, China. A single-centered, retrospective, observational study.
        Crit Care Med. 2020; (Online First)https://doi.org/10.1097/CCM.0000000000004549
        • Knight S.R.
        • Ho A.
        • Pius R.
        • et al.
        Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C Mortality Score.
        BMJ. 2020; 370: m3339
        • Liao X.
        • Wang B.
        • Kang Y.
        Novel coronavirus infection during the 2019-2020 epidemic: preparing intensive care units-the experience in Sichuan Province, China.
        Intensive Care Med. 2020; 46: 357-360
        • Lim N.T.
        • Pan D.
        • Barker J.
        NEWS2 system requires modification to identify deteriorating patients with COVID-19.
        Crit Care Med. 2020; 48: e1004-e1011https://doi.org/10.7861/clinmed.Let.20.4.6
        • Smith G.B.
        • Prytherch D.R.
        • Schmidt P.E.
        • Meredith P.
        Early warning scores: unravelling detection and escalation.
        Int J Health Care Qual Assur. 2015; 28: 872-875

      Linked Article