Original Title: Efectividad, seguridad y coste-efectividad de los sistemas automatizados de detección de reacciones adversas a medicamentos

Authors' objectives: • To evaluate the effectiveness and safety of automated systems for the detection of ADR. • To evaluate the cost-effectiveness of automated systems for the detection of ADR from the perspective of the National Health System (SNS). • To estimate the costs of the incorporation of automated systems in electronic medical records would entail to the common portfolio of SNS services in Spain. • To identify ethical, legal, organizational social and environmental considerations related to technology. • To identify research needs and standard outcome measures from the perspective of patients, family members/caregivers, healthcare professionals and research staff on the use of ADR detection systems.

Authors' results and conclusions: RESULTS: Effectiveness/safety Only one effectiveness study was identified. This was a prospective OS (3 months) with historical control (previous 3 months) that evaluated changes in the prescription of antihypertensive medication by doctors, after the implementation of a system with an NLP component. A significantly lower rate of prescription of medication suspected of causing ADR (MSADR) was observed compared to the control group, during the first 24 hours (28% vs. 39% of the MSADR detected, p < 0.001) and throughout the hospital stay (47% vs. 58%, p < 0.001). Of the MSADR prescribed during admission, the intervention group showed a lower rate in the first 24 hours (77% vs. 84%, p = 0.003). There were no significant differences in the rate of MSADR discontinuation before discharge (16% vs 14%, p = 0.40). In the overall sample, there were no differences in the rate of suspension/discontinuation of the prescribed MSADR throughout the hospitalization (p = 0.28), but there were differences when only the professionals in the intervention group who responded to the post-alert questionnaire were analyzed, that is, those who definitely saw the alerts (25% vs. 15%, p = 0.003). Regarding the MSADR mentioned in the discharge report, the difference in suspension/discontinuation was not significant in the overall sample (20% vs. 23%, p = 0.053), but again it was significant when including only those who answered the questionnaire (38% vs. 23%, p < 0.001). We included 20 diagnostic performance studies (17 with PLN), plus another 29 PLN studies included in the identified review. Only 7 of them verified the actual occurrence of the adverse events included in the databases used. Of the total NLP studies, only 6 performed an external validation of their models (i.e., with data not used in training). The results generally show a good overall performance of the models (detection of all relevant terms, medication, symptoms, diseases, adverse events), but the performance is considerably lower when it comes to detecting the specific terms (or relationships) in the texts that define the possible ADRs. Overall performance varies widely between different specific adverse events evaluated within the same study. Only one study used data from symptom questionnaires prospectively, obtaining high performance, although it was a small sample of patients without external validation. Cost-effectiveness No evidence was identified on the cost-effectiveness of automated systems for ADR detection. A complete economic evaluation could not be carried out due to lack of the minimum necessary information. Cost study The cost study from the perspective of the SNS estimated an initial cost of €25,000 for implementation and support to carry out the integration of the system, to which an annual recurring cost of €45,000 would have to be added for maintenance, updating the database and the costs of cloud infrastructure. Ethical, legal, organizational, social and environmental considerations Regarding ethical and legal aspects, it should be noted that privacy is of utmost importance in automated ADR detection systems. The use of large amounts of data (Big Data) has played a significant role in ADR identification and health information sharing. New data sources should be explored, data mining tools and privacy protection technologies should be developed. Regarding cybersecurity, telemedicine services in general often require wireless network connections that collect and transmit patient information to telemedicine systems, allowing third parties to access these data, so the telemedicine service system may be exposed to security threats. The existence of potential biases in the datasets used to train AI system algorithms may generate fairness issues. Regarding organizational aspects, the evidence suggests thar some practitioners consider that ADR reporting technology systems integrated into the workflow can be facilitators for ADR reporting. Some factors affecting the feasibility of automated systems implementation were identified, such as lack of standardization between EHR systems, specificity in data entry options, incomplete and inaccurate documentation, and alert fatigue. It is important that automated systems for ADR reporting be integrated with medication management or hospital electronic records, as it can help save time and resources. Education and a positive culture of ADR reporting can be invaluable in increasing ADR reporting rates, so nursing, medical, pharmacy and other members of the healthcare team need to be trained to take responsibility for ADR reporting. CONCLUSIONS: • No study has been identified that evaluates the effectiveness and safety of automated ADR detection systems developed in the last 10 years in reducing ADR and its consequences on final health outcomes. Only one study evaluated the hospital implementation of a NLP system and its effect on the prescription of antihypertensive medication for 3 months, showing a reduction in the prescription of medication evaluated by the system as suspicious of producing ADRs prior to admission. • A good number of studies have been carried out evaluating the diagnostic performance of AI methods applied to EHR data, especially NLP. These studies generally show good overall performance of the models, but that is mainly due to the detection of medication, while the results are still below acceptable performance in the specific detection of ADRs in clinical texts. On the other hand, overall performance shows wide variability when specific RAMs are studied within the same study. Most of these studies only carried out internal validation of their models and did not verify the occurrence of actual ADRs beyond corroborating their mention in the texts to train the models. • The implementation of an automated ADR detection system should be carried out only after meticulous planning, in conjunction with the clinical professionals involved, taking into account the clinical area where its effectiveness and efficiency are maximized (for example, in populations with high risk or in the case of severe ADRs), the expectations and concerns of professionals and the effect on their workflow, as well as the technical issues regarding the feasibility of its implementation, its updating in the future, as well as its scalability and generalization to other clinical areas and other hospitals in the health system. • No evidence was identified on the cost-effectiveness of automated systems for ADR detection. • The estimated cost of an automated system to intercept the prescription process and be able to support pharmaceutical prescriptions was €25,000, for implementation and support to integrate it into the Hospital Information System. For maintenance, updating of the database and the costs of the cloud infrastructure, an annual recurring cost of €45,000 would have to be added. • Proper planning of the implementation of an early ADR detection system is a crucial aspect for its introduction into the healthcare system. Within the context of the NHS, it is highly recommended to incorporate automated ADR detection systems through the interoperability office, which guarantees the quality and safety of the system, as well as facilitating the implementation of the same system in different healthcare centers. Additionally, the use of methodologies such as co-design when developing these automated systems can respond to the needs expressed by all stakeholders.

Authors' methods: SYSTEMATIC REVIEW ON EFFECTIVENESS AND SAFETY: A literature search was carried out in the MEDLINE, Embase, CDRS/CENTRAL and Web of Science Core Collection databases until June 2023. The search was not restricted by start date, but studies published after 2013 were excluded in the selection, given the rapid development of this technology. We included systematic reviews (SR), randomized controlled trials (RCTs) and observational studies (OS) that evaluated the effectiveness/safety or diagnostic performance of automated systems for the detection of ADRs from data included in the EMR (including patient-reported outcomes), in primary or hospital care. Classical computing systems with structured data (e.g., ICD-10 codes, laboratory data, medication, etc.) and AI systems with structured and unstructured data (narrative texts) are included. We excluded studies of signal detection in pharmacovigilance, studies on prediction of future ADRs, studies that evaluate alert systems exclusively at the time of drug prescription, studies focused exclusively on medication errors, and studies of diagnostic performance included in the identified reviews. A narrative synthesis of the results was carried out. SYSTEMATIC REVIEW ON COST-EFFECTIVENESS: An SR of economic evaluations (EE) was carried out to evaluate the cost-effectiveness of automated systems for the detection of ADRs following the same search strategy as for the SR of effectiveness and safety. The assessment of the methodological quality of the EE was carried out, by two independent reviewers, using the list of criteria of Drummond et al. and/or the Critical Reading Sheet FLC 3.0 of OSTEBA. It was planned to extract data related to the identification of the article, the design and methodology, and the results of the study (perspective, time horizon, costs, effectiveness, ICER, cost-effectiveness threshold and sensitivity analysis), with special attention to the quality of the information sources used to develop the economic evaluation and the source of funding. A narrative synthesis was planned with tabulation of the characteristics and results of the included studies. COST STUDY: A partial economic evaluation (cost study) was carried out from the perspective of the National Health System (SNS) of an automated system for the detection of ADRs in people with any pathology or clinical condition in which there is an indication for pharmacological treatment. Based on a system currently in the pilot phase, the use of resources and costs of the implementation and start-up of such system were taken into account to intercept the prescription process and be able to support pharmaceutical prescriptions. ETHICAL, LEGAL, ORGANIZATIONAL, SOCIAL AND ENVIRONMENTAL ASPECTS: A definition of the scope was made following the algorithm for the evaluation of ethical, legal, organizational, social and environmental aspects specific to the technology. To this end, a series of manual searches were carried out and stakeholders (authors and contributors to the report) as well as industry were consulted. Based on the results of this exploratory process, specific research questions were defined, and an SR of the scientific literature was carried out in the electronic databases MEDLINE, Embase and Web of Science Core Collection. Systematic reviews and scoping reviews; narrative reviews, qualitative studies, observational studies and mixed methods studies covering ethical, legal, organizational and social aspects were included. A narrative synthesis of the results was made and supplemented by further consultation with stakeholders to clarify those aspects that were considered relevant.

Project Status: Completed

Year Published: 2024

URL for published report: https://sescs.es/informe-sistemas-deteccion-reacciones-adversas-medicamentos/

English language abstract: An English language summary is available

Publication Type: Full HTA

Country: Spain

Organisation Name: Canary Health Service

Contact Address: Dirección del Servicio. Servicio Canario de la Salud, Camino Candelaria 44, 1ª planta, 38109 El Rosario, Santa Cruz de Tenerife

Contact Name: sescs@sescs.es

Contact Email: sescs@sescs.es