Authors' objectives: Guidelines for sepsis recommend treating those at highest risk within 1 hour. The emergency care system can only achieve this if sepsis is recognised and prioritised. Ambulance services can use prehospital early warning scores alongside paramedic diagnostic impression to prioritise patients for treatment or early assessment in the emergency department. To determine the accuracy, impact and cost-effectiveness of using early warning scores alongside paramedic diagnostic impression to identify sepsis requiring urgent treatment. Sepsis is a life-threatening reaction to an infection in which the immune system overreacts to infection and causes organ damage. Early recognition and treatment of sepsis has the potential to reduce mortality. Guidelines for sepsis highlight the importance of early recognition and treatment, with treatment recommended within 1 hour of presentation for those at highest risk. The emergency care system can only achieve this if sepsis is recognised and prioritised. Ambulance services can use prehospital early warning scores to identify people with a high risk of sepsis and then pre-alert the emergency department (ED) or provide the patient with prehospital treatment. However, they need to determine which score to use, the threshold of positivity for the score, and whether to apply the early warning score to all medical cases or just those where the paramedic diagnostic impression suggests sepsis, infection or a non-specific presentation. This requires estimates of the diagnostic accuracy of early warning scores and consideration of the balance between sensitivity (avoiding missing sepsis) and specificity (prioritising too many patients who do not have sepsis). We aimed to determine the accuracy, impact and cost-effectiveness of prehospital early warning scores for adults with suspected sepsis. Our specific objectives were: to estimate the accuracy of prehospital early warning scores for identifying sepsis requiring time-critical treatment in adults with possible sepsis who are attended by emergency ambulance to estimate the impact of using prehospital early warning scores to guide key prehospital decisions, in terms of the operational consequences, and the cost-effectiveness of alternative strategies.

Authors' results and conclusions: Data from 95,022 episodes involving 71,204 patients across four hospitals showed that most early warning scores operating at their pre-specified thresholds would prioritise more than 10% of cases when applied to non-specific attendances or all attendances. Data from 12,870 episodes at one hospital identified 348 (2.7%) with the primary reference standard. The National Early Warning Score, version 2 (NEWS2), had the highest area under the receiver operating characteristic curve when applied only to patients with a paramedic diagnostic impression of sepsis or infection (0.756, 95% confidence interval 0.729 to 0.783) or sepsis alone (0.655, 95% confidence interval 0.63 to 0.68). None of the strategies provided high sensitivity (> 0.8) with acceptable positive predictive value (> 0.15). NEWS2 provided combinations of sensitivity and specificity that were similar or superior to all other early warning scores. Applying NEWS2 to paramedic diagnostic impression of sepsis or infection with thresholds of > 4, > 6 and > 8 respectively provided sensitivities and positive predictive values (95% confidence interval) of 0.522 (0.469 to 0.574) and 0.216 (0.189 to 0.245), 0.447 (0.395 to 0.499) and 0.274 (0.239 to 0.313), and 0.314 (0.268 to 0.365) and 0.333 (confidence interval 0.284 to 0.386). The mortality relative risk reduction from prioritisation at which each strategy would be cost-effective exceeded 0.975 for all strategies analysed. No strategy is ideal but using NEWS2, in patients with a paramedic diagnostic impression of infection or sepsis could identify one-third to half of sepsis cases without prioritising unmanageable numbers. No other score provided clearly superior accuracy to NEWS2. Research is needed to develop better definition, diagnosis and treatments for sepsis. Retrospective cohort study We collected data from 95,022 ambulance episodes involving 71,204 patients with median age 66 years, and included 37,588 (53.0%) women, and 40,045 (94.9%) with white ethnicity. The mean (standard deviation) number of daily attendances meeting the study inclusion criteria was 93.5 (14.7) at Sheffield Northern General Hospital, 59.5 (10.8) at Doncaster Royal Infirmary, 51.3 (8.9) at Rotherham General Hospital and 74 (11) at University Hospitals Coventry and Warwickshire. Most early warning scores operating at their pre-specified thresholds would prioritise fewer than 10% of attendances when applied only to those with a diagnostic impression of sepsis or infection, but would prioritise more than 10% when applied to non-specific attendances or all attendances. The exceptions were qSOFA (threshold > 1), the Screening to Enhance PrehoSpital Identification of Sepsis (SEPSIS) score, the Critical Illness Score (CIS; threshold > 4), the Paramedic Initiated Treatment of Sepsis Targeting Out-of-hospital Patients clinical trial rule, the PRESS score and the sepsis alert criteria. Yorkshire Ambulance Service recorded only one diagnostic impression, whereas West Midlands Ambulance Service recorded multiple unranked impressions, so strategies prioritised a greater proportion of patients transported to University Hospitals Coventry and Warwickshire. Consequently, in the West Midlands most strategies prioritised more than 10% of cases when applied to those with a diagnostic impression of infection or sepsis. We linked 12,870 out of 24,955 (51.6%) cases to the Sheffield Northern General Hospital records at Sheffield and identified 348 (2.7%) with the primary reference standard. The sensitivity and positive predictive value of paramedic diagnostic impression were 0.328 (95% CI 0.28 to 0.379) and 0.285 (95% CI 0.243 to 0.331) for sepsis, 0.572 (95% CI 0.519 to 0.623) and 0.156 (95% CI 0.137 to 0.176) for infection or sepsis, and 0.897 (95% CI 0.86 to 0.924) and 0.053 (95% CI 0.048 to 0.059) for non-specific presentation, infection or sepsis. The early warning scores had a greater area under the ROC curve when applied to all cases rather than alongside diagnostic impression, but the low prevalence of the reference standard meant that thresholds with sensitivity above 0.7 generally had positive predictive value below 0.15, which would prioritise an unmanageable number of cases. When higher thresholds were used to provide acceptable positive predictive value and a manageable number of cases, strategies that applied the early warning score only to those with a diagnostic impression of sepsis or infection tended to have better overall accuracy. NEWS2 had the highest area under the ROC curve when applied only to those with a paramedic diagnostic impression of sepsis or infection (0.756, 95% CI 0.729 to 0.783) or sepsis alone (0.655, 95% CI 0.63 to 0.68). Only the SEPSIS score had a higher area under the ROC curve than NEWS2 when applied to non-specific presentation, infection or sepsis (0.862 vs. 0.858) and all cases (0.882 vs. 0.877). None of the strategies provided high sensitivity (e.g. > 0.8) with acceptable positive predictive value (e.g. > 0.15). NEWS2, using varying thresholds and combinations with diagnostic impression, provided combinations of sensitivity and specificity that were similar or superior to all other early warning scores. We identified strategies reflecting published recommendations for prioritisation that could offer options with varying trade-offs between sensitivity and positive predictive value. Applying NEWS2 only to those with a paramedic diagnostic impression of sepsis or infection respectively provided sensitivities and positive predictive values of 0.522 (95% CI 0.469 to 0.574) and 0.216 (95% CI 0.189 to 0.245) with a threshold > 4, 0.447 (95% CI 0.395 to 0.499) and 0.274 (95% CI 0.239 to 0.313) with a threshold > 6, and 0.314 (95% CI 0.268 to 0.365) and 0.333 (95% CI 0.284 to 0.386) with a threshold > 8. Applying qSOFA > 1 only to those with a paramedic diagnostic impression of sepsis or infection provided sensitivity of 0.305 (95% CI 0.259 to 0.355) and positive predictive value of 0.356 (95% CI 0.304 to 0.412). We were unable to identify a strategy that would prioritise a substantial majority of patients with sepsis without prioritising a potentially unmanageable number of patients for the ED. Most early warning scores, used at a recommended threshold, are likely to prioritise an unmanageable number of cases unless they are only used to prioritise cases with a paramedic diagnostic impression of infection or sepsis. However, paramedic diagnostic impression of infection or sepsis only identified 57% of cases with a reference standard diagnosis of sepsis requiring urgent treatment. The NEWS2 provides sensitivity and specificity for identifying sepsis that is generally similar or superior to other scores operating at a comparable threshold. We therefore found no evidence to justify the support and training required to implement an alternative strategy to NEWS2, which is already widely used in NHS ambulance services. National Early Warning Score, version 2, could be used at thresholds of > 4 or > 6 in presentations with a diagnostic impression of infection or sepsis, reflecting the Academy of Medical Royal Colleges clinical decision support framework, or > 8 to provide similar sensitivity and specificity to the use of qSOFA > 1 recommended in the sepsis-3 guidelines. These strategies provide a range of options that ambulance services and hospitals could use, depending upon capacity to manage prioritised cases and what prioritisation involves. Health economic modelling suggests that sensitive strategies for identifying patients with possible sepsis for prioritisation could be cost-effective, if we are convinced that reducing treatment delay reduces mortality and the emergency care system has the capacity to deliver meaningful prioritisation to substantial numbers of cases.

Authors' methods: Retrospective diagnostic cohort study and decision-analytic modelling of operational consequences and cost-effectiveness. Two ambulance services and four acute hospitals in England. Adults transported to hospital by emergency ambulance, excluding episodes with injury, mental health problems, cardiac arrest, direct transfer to specialist services, or no vital signs recorded. Twenty-one early warning scores used alongside paramedic diagnostic impression, categorised as sepsis, infection, non-specific presentation, or other specific presentation. Proportion of cases prioritised at the four hospitals; diagnostic accuracy for the sepsis-3 definition of sepsis and receiving urgent treatment (primary reference standard); daily number of cases with and without sepsis prioritised at a large and a small hospital; the minimum treatment effect associated with prioritisation at which each strategy would be cost-effective, compared to no prioritisation, assuming willingness to pay £20,000 per quality-adjusted life-year gained. We estimated accuracy using a sample of older patients at one hospital. Reliable evidence was not available to estimate the effectiveness of prioritisation in the decision-analytic modelling. We undertook (1) a retrospective cohort study to estimate the accuracy of prehospital early warning scores alongside paramedic diagnostic impression and (2) decision-analytic modelling of the operational consequences and cost-effectiveness of using prioritisation strategies based on early warning score and diagnostic impression. Retrospective cohort study We used a literature review and expert opinion to identify 21 early warning scores for evaluation. We used routine ambulance service data to identify all episodes in 2019 in which two ambulance services (Yorkshire and West Midlands) transported patients with medical presentations to four acute hospitals (Sheffield Northern General Hospital, Doncaster Royal Infirmary, Rotherham General Hospital, University Hospitals Coventry and Warwickshire). We excluded episodes with injury, mental health problems, cardiac arrest or direct transfer to specialist services, and cases with no vital signs recorded. We calculated early warning scores from the first recorded vital signs on the ambulance service electronic patient-report form and categorised the paramedic diagnostic impression as sepsis, infection, non-specific presentation or other specific presentation. We then determined the number of cases that ambulance services would prioritise at each hospital using each early warning score alongside the categorised paramedic diagnostic impression. We planned to use the National Health Service (NHS) Digital Data Access Request Service to link ambulance service to hospital data but NHS Digital were unable to provide this service. We therefore instituted a rescue plan to link ambulance service to hospital data at one participating hospital (Sheffield) to determine whether patients had a reference standard diagnosis of sepsis, adjudicated by two independent clinicians following hospital record review. The primary reference standard consisted of meeting the sepsis-3 definition [evidence of infection with a change of two or more points in the Sequential (sepsis-related) Organ Failure Assessment (SOFA) score] and receiving treatment for sepsis. The secondary reference standard consisted of meeting the sepsis-3 definition alone. We analysed the ambulance service data descriptively to report the mean daily number of cases that the ambulance service would pre-alert to each hospital for each combination of early warning score and diagnostic impression. For the accuracy analysis, we constructed receiver operating characteristic (ROC) curves to evaluate sensitivity and specificity over the range of each score. We calculated the area under the ROC curve, sensitivities, specificities, and positive and negative predictive values at key cut-points, each with a 95% confidence interval (CI). Reporting of the results highlights sensitivity and positive predictive value as these best indicate under-triage (sensitivity, the proportion of sepsis cases prioritised) and overtriage (positive predictive value, the proportion of prioritised cases with sepsis). To select strategies for the decision-analytic modelling, we calculated the proportion of mean daily ambulance arrivals that would be prioritised at each hospital and excluded strategies that would prioritise a potentially unmanageable proportion (> 10%). We then compared the accuracy of strategies and excluded those with sensitivity and specificity both inferior to another strategy. We also excluded strategies that were not clearly superior to a comparable strategy involving the National Early Warning Score, version 2 (NEWS2) on the basis that NEWS2 is already widely used by NHS ambulance services, whereas other strategies would require additional training and support to implement. Inability of NHS Digital to link ambulance service to hospital data meant that we were only able to estimate the accuracy of early warning scores at one hospital using data from 51.6% of the eligible population for whom the ambulance service had NHS numbers. The included patients were markedly older than the excluded patients. We were unable to identify reliable evidence to estimate the effectiveness of early treatment for sepsis, so were unable to identify the most cost-effective strategy.

Authors' identified further research: Research into prehospital early warning scores for sepsis is limited by our current inability to clinically measure the dysregulated host response that characterises sepsis and uncertain estimates of the benefits of early treatment. We therefore need to prioritise research to develop better ways of defining and diagnosing sepsis, and to develop and evaluate effective early treatment for sepsis. Future research involving routine ambulance service and hospital data requires a system for NHS data management that supports health data science.

Project Status: Completed

URL for project: https://www.journalslibrary.nihr.ac.uk/programmes/hta/17/136/10

Year Published: 2024

URL for published report: https://www.journalslibrary.nihr.ac.uk/hta/NDTY2403

URL for additional information: English

English language abstract: An English language summary is available

Publication Type: Full HTA

Country: England, United Kingdom

DOI: 10.3310/NDTY2403

Organisation Name: NIHR Health Technology Assessment programme

Contact Address: NIHR Journals Library, National Institute for Health and Care Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

Contact Name: journals.library@nihr.ac.uk

Contact Email: journals.library@nihr.ac.uk