Authors' objectives: Health-care organisations in England that are rated as inadequate for leadership and one other domain enter the Special Measures for Quality regime to receive support and oversight. A ‘watch list’ of challenged providers that are at risk of entering Special Measures for Quality also receive support. Knowledge is limited about whether or not the support interventions drive improvements in quality, the costs of the support interventions and whether or not the support interventions strike the right balance between support and scrutiny. To analyse the responses of trusts to the implementation of (1) interventions for Special Measures for Quality trusts and (2) interventions for challenged provider trusts to determine their impact on these organisations’ capacity to achieve and sustain quality improvements. Health-care organisations in England that are rated as inadequate in terms of leadership and one other domain enter the Special Measures for Quality (SMQ) regime to receive increased support and oversight. Challenged providers (CPs) that are at risk of entering SMQ are placed on a ‘watch list’ and also receive support. Interventions for trusts in SMQ/CP regimes typically vary between trusts, and may include appointment of an improvement director (ID), a review of the trust’s leadership capability, access to financial resources for quality improvement (QI), an improvement plan, buddying with other trusts and commissioning external expertise. These interventions may be delivered in conjunction with other interventions and within a context of significant senior leadership changes. There is limited knowledge about whether or not these interventions drive improvements in quality, the costs of the interventions and whether or not the interventions strike the right balance between support and scrutiny. Our evaluation sought to understand how trusts respond to being placed in SMQ/CP regimes and whether or not and how the interventions provided impact the trusts’ capacity to achieve sustainable QI. The objective was to analyse the responses of trusts to the implementation of (1) interventions for trusts in SMQ and (2) interventions for CP trusts to determine the impact of these interventions on the trusts’ capacity to sustain and achieve QIs. The study focused on the main interventions that NHS Improvement has identified as forming part of the SMQ/CP regimes: appointment and use of an ID buddying with other trusts the opportunity to bid for central funding to spend on QI. We also remained open to any other interventions that participating trusts identified as being part of the SMQ/CP regimes and considered these interventions within a wider context of any leadership changes. Our research questions (RQs) were: What are the programme theories (central and local) guiding the interventions delivered to trusts in SMQ/CP regimes? How and why do trusts respond to SMQ/CP regimes and the interventions within these regimes? Which features of trusts in SMQ/CP regimes, and their wider context, contribute to their differing performance trajectories? What are the relative costs of the interventions and how do these compare with their benefits? How are data used by trusts in SMQ/CP regimes, and how do data contribute to their understanding of improvements in quality and service delivery, especially in areas where performance concerns have been raised by the Care Quality Commission? Do trusts in SMQ/CP regimes find it more difficult to recruit and retain staff?

Authors' results and conclusions: The Special Measures for Quality/challenged provider regimes were intended to be ‘support’ programmes. Special Measures for Quality/challenged provider regimes had an emotional impact on staff. Perceptions of NHS Improvement interventions were mixed overall. Senior leadership teams were a key driver of change, with strong clinical input being vital. Local systems have a role in improvement. Trusts focus efforts to improve across multiple domains. Internal and external factors contribute to positive performance trajectories. Nationally, only 15.8% of Special Measures for Quality trusts exited the regime in 24 months. Entry into Special Measures for Quality/challenged provider regimes resulted in changes in quality indicators (such the number of patients waiting in emergency departments for more than 4 hours, mortality and the number of delayed transfers of care) that were more positive than national trends. The trends in staff sickness and absence improved after trusts left Special Measures for Quality/challenged provider regimes. There was some evidence that staff survey results improved. No association was found between Special Measures for Quality/challenged provider regimes and referral to treatment times or cancer treatment waiting times. NHS Improvement spending in case study trusts was mostly directed at interventions addressing ‘training on cultural change’ (33.6%), ‘workforce quality and safety’ (21.7%) and ‘governance and assurance’ (18.4%). The impact of Special Measures for Quality on financial stability was equivocal; most trusts exiting Special Measures for Quality experienced the same financial stability before and after exiting. Empirical evidence of positive impacts of Special Measures for Quality/challenged provider regimes were identified; however, perceptions were mixed. Key lessons were that (1) time is needed to implement and embed changes; (2) ways to mitigate emotional costs and stigma are needed; (3) support strategies should be more trust specific; (4) poor organisational performance needs to be addressed within local systems; (5) senior leadership teams with stability, strong clinical input and previous Special Measures for Quality experience helped to enact change; (6) organisation-wide quality improvement strategies and capabilities are needed; (7) staff engagement and an open-listening culture promote continuous learning and a quality improvement ‘mindset’, which is critical for sustainable improvement; and (8) consideration of the level of sustainable funds required to improve patients’ outcomes is needed. The rapid review found dominant definitions of success/failure and turnaround, which affected the design and implementation of improvement interventions. Successful interventions included restructuring senior leadership teams, inspections (in schools) and internal reorganisation by external organisations. The review also found that most interventions were designed and implemented at an organisational level, without considering system context, and very little attention was paid to the potential negative consequences of the interventions and their costs. National perspectives and the programme theories underpinning the SMQ/CP regimes (RQ1) were explored through national-level interviews and documentary analysis. National stakeholders perceive the SMQ/CP regimes as ‘support’ programmes that aim to enable organisations to bring about improvements. QI plans are a central element of SMQ/CP regimes, and an essential role of IDs is to proactively engage organisational leaders and support the development of the improvement strategy. Through our case study analysis, we identified stakeholder perceptions of SMQ/CP regimes and the NHS Improvement interventions (RQ2). Although SMQ/CP could be viewed positively, with some trusts feeling that they received the right support or were allowed space to make changes, others saw SMQ as heavy-handed scrutiny or punishment. Over time, and in hindsight, as a trust went on to improve and if in receipt of support, there could be a shift to a more positive view of SMQ/CP as a needed catalyst for positive change. We also found that there was an emotional impact on staff of the trust being labelled as failing and being placed in SMQ/CP. The perceptions of NHS Improvement interventions such as IDs, buddy trusts, funding and deep dives were mixed overall, and it was highlighted that trusts will have individual issues and needs for support that mean specific tailoring of the interventions is required. Leadership teams were found to be a key driver of change and, in terms of senior-level oversight, the medical director and chief nursing roles appear to be vital for communication between divisional and executive leadership tiers, promoting trust-wide clinical engagement and overseeing improvement planning. It was noted at a national and case study level that local system-wide issues may need to be addressed for a trust to exit SMQ/CP regimes. We examined how trusts responded to the SMQ/CP regimes (RQ2) and found that the case study trusts focused their efforts to improve across eight domains: governance, accountability and leadership – review of governance and accountability; increased ‘board to ward’ interactions; development of sustainable strategies for QI and patient safety; and stronger clinical leadership at senior, divisional and ward levels service delivery – prioritising improvements that ensure patient and staff safety, focus on compliance with national standards and improved ED performance and referral to treatment (RTT) times data monitoring and use of data – improving the use of data by addressing how it is being collected and analysed, and how findings are shared organisational culture and staff engagement – addressing problems with organisational culture (e.g. bullying); recognising and celebrating staff; and improving lines of communication between senior team and staff workforce – addressing staffing levels, skill mix and retention; ensuring safe staff levels; and introducing strategies to reduce staff turnover and improve staff retention QI plan or strategy – working with ID to develop a plan/strategy; and setting a vision for culture change and continuous improvement QI interventions, methods or techniques – a range of QI methods and tools (e.g. Plan-Do-Study-Act (PDSA), WHO checklists) and broader interventions used to drive improvement; and leadership and resource commitments to embed these trust wide estates and equipment – improvements in working and patient environments to ensure safety, improve capacity and modernise services. Our analysis of trusts’ performance trajectories (RQ3) using national-level data found that, relative to national trends, entry into SMQ/CP regimes corresponded to positive changes in 4-hour waits in EDs, mortality and delayed transfers of care. Trends in sickness and absence improve after trusts leave the regime. There was also some evidence that staff survey results improve over the period that trusts are in the regime (significant improvements in five of nine domains). SMQ/CP regimes do not, however, influence RTT times or cancer waiting times. From the case study analysis, we identified several key internal and external factors that contributed to positive performance trajectories (RQ3): internal factors – characteristics of trusts that exit SMQ, including trusts that have sustained QIs over time; systematic use of data for QI; use of QI method(s) and dedicated resources; safe workforce levels; focus on staff engagement and recognition; integrated quality, financial and risk management with clear lines of accountability; and an embedded open and listening improvement culture. external factors – established good working relationships with the regulators; collaborating with external partners and peers; and have had time to embed change. We identified several key internal and external factors that contributed to our case study trusts not yet exiting SMQ/CP (RQ3): internal factors – instability and churn at senior leadership level; absence of an organisation-wide QI methodology and culture; poor governance and risk management at all levels; poor staff engagement and issues with harassment and bullying; outdated equipment and/or deteriorating estates; and problems with staff recruitment and retention. external factors – financial pressures in the regional health economy; recent entry into SMQ or CP recently; or improvements made but not yet embedded. The cost–consequence analysis (RQ4) was based on case study and national-level data. Mean funds spent on trusts in the SMQ regime at the national level during 2018–19 were more than twice as high as the mean funds spent during the same period for trusts under the CP regime. The largest components of NHS Improvement spending for our case studies were identified as interventions directed at ‘training on cultural change’ (33.6%), ‘workforce quality and safety’ (21.7%) and ‘governance and assurance’ (18.4%). CP trusts were four times more likely to exit within the time limits (12 months) than SMQ trusts (24 months). The interventions delivered to trusts as part of the SMQ/CP regimes showed a positive effect on staff-based measures, although there were fewer improvements in the context of ‘promotion of staff’s health & wellbeing’, ‘staff’s satisfaction with quality of care’ or ‘organisation’s actions on quality, diversity & inclusion’. The impact of SMQ on financial stability was equivocal, as we found that most of the trusts that exited SMQ experienced the same financial stability before and after exiting, while this share was lower for the group remaining in the regime. Our case study analyses found that trusts recognised the importance of use of data in QI processes (RQ5). Trusts focused on a standard set of nationally agreed metrics for high-level reporting. The limitations, such as inadequate monitoring of the impact of QI activities, were acknowledged. Trusts increasingly recognised the importance of triangulating different indicators and information sources, including ‘soft data’ from staff and patients, to obtain a more holistic view of quality. Our analysis of the impact of SMQ/CP on the recruitment and retention of staff (RQ6) was based on national-level and case study data. National-level analysis found that sickness absence, staff vacancy rates, proportions of consultants and nurse-to-doctor ratio were not significantly different from national means at SMQ/CP trusts when they enter the regime. Workforce issues, such as staff turnover, recruitment and retention, and sickness and agency spend, were identified as underlying reasons for why case study trusts enter SMQ/CP. In turn, workforce investment was a key component of case study trusts’ response to being in SMQ/CP, with trusts striving to address gaps in staffing levels, particularly in ED, and skill mixes, reduce staff turnover and improve staff retention. Stigma from the SMQ label was perceived as having a negative impact on recruitment and retention of staff. Supporting poor-performing health-care organisations to improve is essential and we have added to the limited knowledge base on the implementation and impact of improvement interventions. Through our evaluation, we have delivered a greater understanding of the programme theory, impact and staff views and experiences of the SMQ/CP regimes, with formative feedback shared with key stakeholders. We have demonstrated the value of mixed-methods approaches that combine quantitative and qualitative data from local case studies alongside quantitative indicators derived from nationally available routine data. The key overarching lessons for regulators, policy-makers and trusts are: Regulatory bodies Time is needed to implement and embed sustainable changes, 2–3 years not 1 year, and staff should be given ‘slack’ to develop and implement changes. Strategies to support improvement need to be more trust specific. Duplication of reporting requirements to different regulatory bodies should be reduced. Consideration should be given to the provision of sustainable funds required to improve patients’ outcomes. Trust leadership Stable leadership is needed once the new team is established because of the time that it takes to make improvements – otherwise problems are perpetuated. Inclusion of people with previous experience with SMQ in senior leadership teams can help to manage regulatory requirements and bring knowledge and confidence to enacting change. Staff and culture Staff engagement and an organisational culture that supports learning are key to sustainable improvement. Emotional costs and stigma Ways to mitigate the emotional cost and stigma of SMQ are needed. QI strategies and capabilities Development of organisation-wide QI strategies and capabilities is important. Local systems Poor organisational performance needs to be considered at both organisational and system levels. Patients and the public Engagement with patients and the public should be emphasised as an important part of the process of making improvements. Key areas and considerations for future research include: prospective evaluation of the impact of the new NHS Improvement/NHS England operating framework focus on SMQ/CP at the local system level and expand the range of stakeholders external to the trust giving viewpoints on SMQ/CP prospective studies could use using sequential monitoring techniques to allow ‘real-time’ assessments of the impact of interventions study of trusts in Special Measures for Finance (SMF) and link between SMQ and SMF further research to understand the impact of SMQ on financial stability prospectively link financial stability to changes in direct/indirect costs and additional opportunity costs using indicators that are part of routinely reported data longitudinal studies to look at the sustainability of improvement and where trusts re-enter SMQ tailoring data collection to trust-specific concerns and areas highlighted for improvement that can feed into the overall evaluation framework.

Authors' methods: This was rapid research comprising five interrelated workstreams: (1) a literature review using systematic methods; (2) an analysis of policy documents and interviews at the national level; (3) eight multisite, mixed-methods trust case studies; (4) an analysis of national performance and workforce indicators; and (5) an economic analysis. The rapid research design and 1-year time frame precludes longitudinal observations of trusts and local systems. The small number of indicators limited the quantitative analysis of impact. Measurement of workforce effects was limited by data availability. We conducted a multisite, mixed-methods study combining qualitative and quantitative approaches. Data collection and analysis followed a rapid research design involving teams of field researchers, and iterative data collection and analysis. The protocol was developed with input from relevant Department of Health and Social Care (DHSC) and NHS Improvement teams, by scoping relevant documents and with feedback from academic peers and patient representatives. The evaluation was formative, with findings shared and discussed with key stakeholders during the study. The study comprised five interrelated work streams: A review of the literature, using systematic methods, on the implementation of improvement interventions in education, local government and health care. Analysis of policy documents and interviews at a national level to understand the origins, evolution and intended purpose of the SMQ/CP regimes. Eight multisite, mixed-method case studies (four ‘high level’, four ‘in-depth’) purposively sampled from 59 trusts that entered SMQ or CP between July 2013 and September 2018. Interview, observational, documentary and quantitative data from the case studies were triangulated and analysed thematically and comparatively. Quantitative analysis at a national level to explore relationships between being in SMQ/CP and performance [4-hour waiting time target for emergency departments (EDs)/62-day cancer target/staff survey] and workforce (mix of staff employed at each trust/sickness absence/staff vacancy rates) indicators. Economic analysis to quantify the costs and benefits of the SMQ/CP interventions; direct costs (national and case studies) were determined and a cost–consequences analysis was conducted for primary consequences (entry to and exit from SMQ/CP) and secondary consequences (staff experiences/cultural changes from the NHS staff survey 2014–18 and trust financial stability).

Project Status: Completed

URL for project: https://www.journalslibrary.nihr.ac.uk/programmes/hsdr/NIHR129663

Year Published: 2023

URL for published report: https://www.journalslibrary.nihr.ac.uk/hsdr/GQQV3512

URL for additional information: English

English language abstract: An English language summary is available

Publication Type: Full HTA

Country: England, United Kingdom

DOI: 10.3310/GQQV3512

Organisation Name: NIHR Health Services and Delivery Research programme

Contact Address: NIHR Journals Library, National Institute for Health and Care Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

Contact Name: journals.library@nihr.ac.uk

Contact Email: journals.library@nihr.ac.uk