Implementing an artificial intelligence command centre in the NHS: a mixed-methods study
Johnson OA, McCrorie C, McInerney C, Mebrahtu TF, Granger J, Sheikh N, Lawton T, Habli I, Randell R, Benn J
Record ID 32018013542
English
Authors' objectives:
Hospital ‘command centres’ use digital technologies to collect, analyse and present real-time information that may improve patient flow and patient safety. Bradford Royal Infirmary has trialled this approach and presents an opportunity to evaluate effectiveness to inform future adoption in the United Kingdom. To evaluate the impact of the Bradford Command Centre on patient care and organisational processes. A hospital command centre (CC) is a new approach to the management of hospital operations based on the colocation of decision-making staff and supported by digital technology to provide these staff with close to real-time information. Recent adoption of hospital CCs in the USA predominantly has demonstrated that the approach can be applied to manage hospital operations, despite their complexity. The supporting technology often includes decision-support algorithms that trigger digital notifications and alerts that identify potential safety or flow issues. These algorithms may be based on simple rules or more complex rules generated by machine learning from historic data, and the software technology has therefore been described as AI, meaning either artificial intelligence or, more accurately, augmented intelligence. In the UK, the implementation of a CC and associated technology by Bradford Teaching Hospitals NHS Trust represents a first of type for the UK NHS. To date, there has been limited evidence of the effectiveness of the CC approach and this study aims to address that evidence gap. Bradford Teaching Hospitals NHS Trust manages Bradford Royal Infirmary, an 800-bed NHS hospital located in Bradford in northern England. The Bradford Hospital Command Centre was implemented at Bradford Royal Infirmary through a phased approach in late 2019, was operational through the COVID-19 pandemic, and remains operational to date. It is designed to integrate and centralise operational decision-making to improve patient flow and patient safety across the whole hospital. The Comand Centre (CC) is implemented in a dedicated room in which up to 30 trained non-clinical management and support staff from different operational functions can sit together in teams facing a wall on which 8 large digital display screens are mounted. Each staff member answers telephone calls and performs their operational role using Information Technology (IT) systems on their desktop computer, information on the wall of display screens and communicating with team members and other teams within the room. A senior clinician or manager supports decision-making. The CC was implemented in November 2019, several months before the COVID-19 pandemic began to have a major impact on hospitals in the UK and globally. It was operational throughout the pandemic and, to date, it is firmly established as the centre for operational management of the hospital. The implementation of the CC at Bradford Royal Infirmary presents an opportunity to evaluate the potential strengths and weaknesses of the approach and to generate learning that can inform other hospitals considering adopting the approach. We aimed to evaluate the impact of the Bradford Command Centre on patient care and organisational processes. We hypothesised that the CC would improve patient flow, reduce bottlenecks and delays, enhance situational awareness to support operational decision-making, and facilitate identification and timely mitigation of threats to patient safety. This study had four research objectives: to evaluate the impact of the CC on patient safety, hospital operational efficiency and related organisational processes to understand the process of implementation of the CC and its integration into hospital management to contextualise the findings using cross-sector and cross-industry perspectives to synthesise the research findings to inform future investment and practice.
Authors' results and conclusions:
The Command Centre was implemented successfully and has improved staff confidence of better operational control. Unintended consequences included tensions between localised and centralised decision-making and variable confidence in the quality of data available. The Command Centre supported the hospital through the COVID-19 pandemic, but the direct impact of the Command Centre was difficult to measure as the pandemic forced all hospitals, including the study and control sites, to innovate rapidly. Late in the study we learnt that the control site had visited the study site and replicated some aspects of the command centre themselves; we were unable to explore this in detail. There was no significant difference between pre- and post-intervention periods for the quantitative outcome measures and no conclusive impact on patient flow and data quality. Staff and patients supported the command-centre approaches but patients expressed concern that individual needs might get lost to ‘the system’. Qualitative evidence suggests the Command Centre implementation was successful, but it proved challenging to link quantitative evidence to specific technology interventions. Staff were positive about the benefits and emphasised that these came from the way they adapted to and used the new technology rather than the technology per se. We were unable to evaluate the impact of the CC as fully as we had planned because the study was impacted by the COVID-19 pandemic. Hospital staff were extremely busy and access on site was challenging. The additional work negotiating access and the direct impact of the pandemic on our own team meant that we had fewer resources and were unable to complete all our objectives. We were able to observe how the CC helped support the hospital manage its operations through the pandemic and received strong positive evidence of its success. Complex pandemic challenges and rapid innovation to meet these challenges made it difficult to attribute outcomes to the specific intervention of the CC. Evaluation of the impact of the Command Centre (Objective 1) Our ethnographic observations and interviews with 15 study-site staff provided documentary evidence of successful use in a complex environment. The CC made a significant impact on the management of the hospital through the pandemic including through the introduction of a COVID-19 ‘tile’ which was used to managed COVID-19-specific processes. The CC and its staff worked with the new technology to change the way that the hospital operated. We identified unintended consequences that included front-line staff developing a sense of being monitored and a fear of interventions from the CC team that were perceived as unwelcome. Linked to this were challenges keeping electronic records up to date and acting on evidence of operational issues that were seen as being under local autonomy or ownership. Data quality was a constant concern for staff working in and around the CC and there were limitations in how up to date and accurate (or complete) records were, often necessitating triangulation and verification from multiple sources and systems and discrepancy between the data in the systems and what front-line staff reported. Compromise, goodwill and a shared sense of purpose were necessary to ensure the CC was effective. We were able to extract time-series data on patient safety, patient flow and data quality from operational systems by selecting representative indicators and plotting these over time. We were able to measure changes in these indicators over time and evaluate statistically the long-term impact of the CC on these indicators. We were not able to isolate improvements in these indicators that could directly be attributed to the introduction of the CC. Similarly, we were unable to isolate noticeable improvements in these indicators between study and control site. We found improvements in mortality and reduced rates of re-admission at the study site but caution against drawing conclusions from this at a time when the pandemic was raging. Some indicators, notably data quality, worsened rather than improved. We reason that the pandemic had such a profound impact on all aspects of operation that it is not possible to separate out and measure the impact of the CC. Similarly, the later adoption of a CC approach by the control site means we cannot use it to draw strong comparisons. This study has presented a case study of a successful implementation of a CC in the UK NHS. The Bradford Command Centre demonstrates that systems are available but it also reveals challenges in the reliability, timeliness and quality of these data that reduce transparency and limit confidence. Management and staff have managed to overcome many of these challenges through determination, negotiation and gradual improvements while also dealing with the pandemic. There was a strong sense that the CC had been invaluable during the pandemic but we were not able to empirically validate this. Our case study was affected by the pandemic and does not provide sufficient evidence to demonstrate major benefits on its own. We therefore recommend: Command centres are a viable approach that should be considered. Reliable, modern hospital-wide information systems are an essential foundation for command-centre technology and poor data quality will undermine implementation if not addressed. Further work should follow the evolving use of the Bradford Command Centre and disseminate learning to other hospitals considering adoption. Further studies that use our time-series approach for performance metrics would allow comparison across more hospitals and support the evaluation of other implementations. Studies should consider mixed methods rather than relying solely on qualitative or quantitative approaches to draw conclusions.
Authors' methods:
A comparative mixed-methods study. Operational data from a study and control site were collected and analysed. The intervention was observed, and staff at both sites were interviewed. Analysis was grounded in a literature review and the results were synthesised to form conclusions about the intervention. The study site was Bradford Royal Infirmary, a large teaching hospital in the city of Bradford, United Kingdom. The control site was Huddersfield Royal Infirmary in the nearby city of Huddersfield. Thirty-six staff members were interviewed and/or observed. Qualitative perspectives on hospital management. Quantitative metrics on patient flow, patient safety, data quality. Anonymised electronic health record data. Ethnographic observations including interviews with hospital staff. Cross-industry review including relevant literature and expert panel interviews. The COVID-19 pandemic disrupted care patterns and forced rapid innovation which reduced our ability to compare study and control sites and data before, during and after the intervention. We conducted a comparative mixed-method case study at two sites: Study site – the CC at Bradford Royal Infirmary. Control site – Huddersfield Royal Infirmary, a hospital in the city of Huddersfield. The control site was selected as being geographically close (15 miles) and part of a similar-sized NHS hospital trust with similar challenges serving areas of high deprivation. At the start of this study, the control site had no plans to implement a CC but, during the study, they learnt from the study site and replicated some aspects of the Bradford Command Centre as part of their own pandemic response. The study combined ethnographic observation and interviews with data analytics of time-series operational data. We accessed and analysed data in the form of anonymised electronic health records from the study and control sites between January 2018 and August 2021 to cover the period before and after implementation of the Bradford Command Centre in November 2019. We conducted 72 hours of ethnographic observations of the CC operation over a period of 9 months after implementation. We conducted 15 interviews with hospital staff at the study site and 4 interviews with staff at the control site. To evaluate the impact of the CC (Objective 1), we described (qualitatively) and evaluated (statistically) the effect on hospital operations and outcomes. We used Interrupted Time Series Analysis to analyse variation in key output indicators in patient safety, patient flow and data quality. We qualitatively investigated situational awareness, operational decision-making, risk and co-ordination/communication across organisational units, from multiple stakeholder perspectives. To understand the process of implementation (Objective 2) our interviews explored staff recall of the implementation, including critical implementation factors and exploring unintended consequences. We used the interviews with staff at the control site to understand how a similar organisation adapted and changed over the same study period. To contextualise the findings (Objective 3), we reviewed the literature on command and control processes in non-healthcare safety-critical operations to extract key principles and contextual factors that may influence transferability of these models into a hospital setting. We also searched for evidence of other hospital CC implementations in the UK and worldwide. To synthesise the research findings (Objective 4), we developed a logic model to map system preconditions, processes, technology and outcomes.
Details
Project Status:
Completed
URL for project:
https://www.journalslibrary.nihr.ac.uk/programmes/hsdr/NIHR129483
Year Published:
2024
URL for published report:
https://www.journalslibrary.nihr.ac.uk/hsdr/TATM3277
URL for additional information:
English
English language abstract:
An English language summary is available
Publication Type:
Full HTA
Country:
United Kingdom
DOI:
10.3310/TATM3277
MeSH Terms
- Artificial Intelligence
- State Medicine
- Hospitals
- Hospital Administration
- Patient Safety
- Hospital Information Systems
Contact
Organisation Name:
NIHR Health and Social Care Delivery Program
Contact Name:
Rhiannon Miller
Contact Email:
rhiannon.m@prepress-projects.co.uk
This is a bibliographic record of a published health technology assessment from a member of INAHTA or other HTA producer. No evaluation of the quality of this assessment has been made for the HTA database.