Authors' objectives: Artificial intelligence-derived software technologies have been developed that are intended to facilitate the review of computed tomography brain scans in patients with suspected stroke. To evaluate the clinical and cost-effectiveness of using artificial intelligence-derived software to support review of computed tomography brain scans in acute stroke in the National Health Service setting. The primary population for this assessment was people presenting or attending secondary care with a suspected acute stroke who were last known to be well within the previous 24 hours. Stroke is a serious life-threatening medical condition defined by the World Health Organization (WHO) as a clinical syndrome consisting of ‘rapidly developing clinical signs of focal (at times global) disturbance of cerebral function, lasting more than 24 hours or leading to death with no apparent cause other than that of vascular origin’. Timely and effective management of the patients with suspected stroke substantially impacts patients’ outcomes. A number of software products with artificial intelligence (AI)-derived software technologies have been developed, which are intended to facilitate the review of computed tomography (CT) images of the brain in patients with suspected stroke. These products are not intended to provide a diagnosis but to support review and reporting healthcare professionals. This assessment aimed to evaluate the clinical and cost-effectiveness of using AI-derived software to support the review of CT brain scans in acute stroke, in the NHSs setting. Three research questions were considered. (1) Does AI-derived software-assisted review of non-enhanced CT brain scans for guiding thrombolysis treatment decisions for people with suspected acute stroke represent a clinically and cost-effective use of NHS resources? (2a) Does AI-derived software-assisted review of CT angiography (CTA) brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke represent a clinically and cost-effective use of NHS resources? (2b) Does AI-derived software-assisted review of CT perfusion brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke after a CTA brain scan represent a clinically and cost-effective use of NHS resources?

Authors' results and conclusions: A total of 22 studies (30 publications) were included in the review; 18/22 studies concerned artificial intelligence-derived software for the interpretation of computed tomography angiography to detect large-vessel occlusion. No study evaluated an artificial intelligence-derived software technology used as specified in the inclusion criteria for this assessment. For artificial intelligence-derived software technology alone, sensitivity and specificity estimates for proximal anterior circulation large-vessel occlusion were 95.4% (95% confidence interval 92.7% to 97.1%) and 79.4% (95% confidence interval 75.8% to 82.6%) for Rapid (iSchemaView, Menlo Park, CA, USA) computed tomography angiography, 91.2% (95% confidence interval 77.0% to 97.0%) and 85.0 (95% confidence interval 64.0% to 94.8%) for Viz LVO (Viz.ai, Inc., San Fransisco, VA, USA) large-vessel occlusion, 83.8% (95% confidence interval 77.3% to 88.7%) and 95.7% (95% confidence interval 91.0% to 98.0%) for Brainomix (Brainomix Ltd, Oxford, UK) e-computed tomography angiography and 98.1% (95% confidence interval 94.5% to 99.3%) and 98.2% (95% confidence interval 95.5% to 99.3%) for Avicenna CINA (Avicenna AI, La Ciotat, France) large-vessel occlusion, based on one study each. These studies were not considered appropriate to inform cost-effectiveness modelling but formed the basis by which the accuracy of artificial intelligence plus human reader could be elicited by expert opinion. Probabilistic analyses based on the expert elicitation to inform the sensitivity of the diagnostic pathway indicated that the addition of artificial intelligence to detect large-vessel occlusion is potentially more effective (quality-adjusted life-year gain of 0.003), more costly (increased costs of £8.61) and cost-effective for willingness-to-pay thresholds of £3380 per quality-adjusted life-year and higher. Assessment of clinical effectiveness A total of 22 studies (30 publications) were included in the review; for 9 of the 13 manufacturers of AI-derived software included in the scope, no studies were identified. All included studies concerned AI-derived software produced by Avicenna, Brainomix, iSchemaView or Viz. The majority (18/22 studies) reported data concerning research question 2a (i.e. evaluated AI-derived software for the interpretation of CTA). All included studies either assessed the diagnostic accuracy of AI-derived software alone (i.e. not as it would be used in clinical practice, as recommended by the manufacturers and as specified in the inclusion criteria for this assessment) or were ‘before and after’ observational studies reporting information about the effects of implementing AI-derived software in treated patients. Eleven studies provided information about the accuracy of various AI-derived software technologies for the detection of LVO on CTA scans in patients with acute ischaemic stroke. Where the target condition included occlusions of internal carotid artery, carotid terminus or the M1 or M2 segments of the middle cerebral artery (MCA), the sensitivity and specificity estimates were 95.4% (95% CI 92.7% to 97.1%) and 79.4% (95% CI 75.8% to 82.6%) for Rapid CTA (iSchemaView, Menlo Park, CA, USA), 91.2% (95% CI 77.0% to 97.0%) and 85.0 (95% CI 64.0% to 94.8%) for Viz LVO, 83.8% (95% CI 77.3% to 88.7%) and 95.7% (95% CI 91.0% to 98.0%) for Brainomix e-CTA, and 98.1% (95% CI 94.5% to 99.3%) and 98.2% (95% CI 95.5% to 99.3%) for Avicenna CINA LVO, based on one study each. There was some evidence to indicate that, where studies included more distal (e.g. M3 segment of the MCA) elements of the anterior circulation or included posterior circulation in their definition of the target condition, sensitivity was reduced. All four studies that provided information about the effects of implementing Viz LVO and one study that provided information about the effects of implementing Rapid CTA reported that implementation was associated with reductions in time to treatment for thrombectomy patients and, where reported, with no significant change in clinical outcomes (mRS). However, it should be noted that two of the studies of Viz LVO and the study of Rapid CTA evaluated implementation in the context of providing an automated alert system (i.e. not as specified in the scope for this assessment); it is plausible that reductions in time to intervention, observed in these studies, may be driven by this ‘early alert’ step. The information provided by studies of this type is also limited in that it concerns only treated (i.e. test positive) patients; no information is provided about test negative patients and hence there is no information about the extent to which AI-derived software, as implemented, may miss patients with LVO. There is no evidence about the accuracy of AI-derived software when used as an aid to human interpretation; all evidence concerns only stand-alone AI. This might imply that a CEA is not feasible for any of the three research questions. However, we conducted a CEA in relation to the research question 2a, where there is most evidence about the performance of AI-derived software technologies alone and one study comparing an AI-derived software technology alone with human reader alone. These studies were not considered appropriate to inform cost-effectiveness modelling but formed the basis by which the accuracy of AI plus human reader could be elicited by expert opinion. The available evidence is not suitable to determine the clinical effectiveness of using AI-derived software to support the review of CT brain scans in acute stroke. The economic analyses did not provide evidence to prefer the AI-derived software strategy over current clinical practice. However, results indicated that if the addition of AI-derived software-assisted review for guiding mechanical thrombectomy treatment decisions increased the sensitivity of the diagnostic pathway (i.e. reduced the proportion of undetected LVOs) this may be considered cost-effective. Nevertheless, the sensitivity of AI-derived software-assisted review when added to current clinical practice is largely uncertain and probably depends on the implementation of AI-derived software-assisted review.

Authors' methods: Twenty-five databases were searched to July 2021. The review process included measures to minimise error and bias. Results were summarised by research question, artificial intelligence-derived software technology and study type. The health economic analysis focused on the addition of artificial intelligence-derived software-assisted review of computed tomography angiography brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke. The de novo model (developed in R Shiny, R Foundation for Statistical Computing, Vienna, Austria) consisted of a decision tree (short-term) and a state transition model (long-term) to calculate the mean expected costs and quality-adjusted life-years for people with ischaemic stroke and suspected large-vessel occlusion comparing artificial intelligence-derived software-assisted review to usual care. The available evidence is not suitable to determine the clinical effectiveness of using artificial intelligence-derived software to support the review of computed tomography brain scans in acute stroke. The economic analyses did not provide evidence to prefer the artificial intelligence-derived software strategy over current clinical practice. However, results indicated that if the addition of artificial intelligence-derived software-assisted review for guiding mechanical thrombectomy treatment decisions increased the sensitivity of the diagnostic pathway (i.e. reduced the proportion of undetected large-vessel occlusions), this may be considered cost-effective. Assessment of clinical effectiveness Twenty-five databases, including MEDLINE and Embase, research registers, conference proceedings and a preprint resource, were searched for relevant studies from inception to July 2021; update searches were conducted in October 2021. Search results were screened for relevance independently by two reviewers. Full-text inclusion assessment, data extraction and quality assessment were conducted by one reviewer and checked by a second. The methodological quality of included diagnostic test accuracy studies was assessed using QUADAS-2 (Bristol Medical School, University of Bristol, Bristol, UK). The methodological quality of observational ‘before and after’ studies was assessed using a checklist, devised by the authors, for this review. The hierarchical summary receiver operating characteristic (HSROC) model was used to estimate summary sensitivity and specificity with 95% confidence intervals (CIs) and prediction regions around the summary points, and to derive HSROC curves for meta-analyses of diagnostic test accuracy, where four or more studies evaluated the same intervention for a given research question. All other results, including those of ‘before and after’ studies, were summarised in a narrative synthesis, grouped by research question addressed, AI-derived software evaluated and study type.

Project Status: Completed

URL for project: https://www.journalslibrary.nihr.ac.uk/programmes/hta/NIHR133836

Year Published: 2024

URL for published report: https://www.journalslibrary.nihr.ac.uk/hta/RDPA1487

URL for additional information: English

English language abstract: An English language summary is available

Publication Type: Full HTA

Country: England, United Kingdom

DOI: 10.3310/RDPA1487

Organisation Name: NIHR Health Technology Assessment programme

Contact Address: NIHR Journals Library, National Institute for Health and Care Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

Contact Name: journals.library@nihr.ac.uk

Contact Email: journals.library@nihr.ac.uk