Authors' objectives:

"This project aimed to address two sets of questions: How should the impact of the NHS Health Technology Assessment (HTA) Programme be measured? What models are available and what are their strengths and weaknesses? What has been the impact of the first 10 years of the NHS HTA programme from its inception in 1993 to June 2003? What factors seem to be associated with HTA research making an impact?" (from executive summary)

Authors' results and conclusions: The literature review identified a highly diverse literature, but confirmed that the 'payback' framework pioneered by Buxton and Hanney was the most widely used and most appropriate model available. It encompassed key elements of many of the alternatives. The review confirmed that impact on knowledge generation was more easily quantified than that on policy, behaviour or especially health gain. The review of the included studies indicated a higher level of impact on policy than is often assumed to occur. The diverse literature suggested that two different sets of studies might provide the most appropriate comparators for the two main parts of the NHS HTA Programme. Studies of the impact of 'HTA Programmes' for policy-making bodies can best be compared with the TARs for NICE. The group 'Other Health Research Programmes' provides comparisons for the primary and secondary research projects of the NHS HTA Programme. The survey showed that data pertinent to payback exist and can be collected. However over one-third of projects did not respond, despite repeated reminders. Against this, a 100% response was not feasible as over the 10-year period several lead researchers had died, retired, moved or were otherwise not reachable. The completed questionnaires confirmed, corrected and extended the data collated by NCCHTA on publications and other indicators. They showed that the HTA Programme had considerable impact in terms of publications, dissemination, policy and behaviour. It also showed, as expected, that different parts of the Programme had different impacts. The TARs for NICE had the clearest impact on policy in the form of NICE guidance. Other policy 'customers' included the National Screening Committee (NSC) and National Service Frameworks. Overall impacts measured in the survey were consistent with or somewhat better than those for other programmes identified in the literature review. Mean publications per project were 2.93 (1.98 excluding the monographs), above the level reported for other programmes. The proportion of NICE TARs reporting an impact on past policy at 96% was among the highest for the 'HTA Programmes'. The 60% of primary and secondary studies reporting an impact on policy was above the other programmes in its group (although some of the latter were responsive mode programmes which would not have been expected to make much impact on policy). The percentage of primary and secondary projects reporting an impact on behaviour was somewhere in the middle of the range for the 'Other Health Research Programmes'. Comparisons with other programmes must be treated with considerable caution due to differences in programme objectives, topics researched and methods of assessing impact. The NCCHTA's reliance on researchers to inform it of publications was shown to lead to incomplete data. Around one-quarter of publications in peer-reviewed journals were missed. These data could probably be better collected using the Internet and then asking the researchers to correct and amend the resulting list. Other data such as those on presentations and further research could only be collected from the researchers. The case studies revealed the large diversity in the levels and forms of impacts and the ways in which they arise. All the NICE TARs and more than half of the other case studies had some impact on policy making at the national level, whether through NICE, NSC, National Service Frameworks, professional bodies or the Department of Health. This underlines the importance of having a customer or 'receptor' body. A few case studies had very considerable impact in terms of knowledge production and in informing national and international policies. In some of these the Principal Investigator had prior expertise and/or a research record in the topic. The case studies confirmed the questionnaire responses, but also provided more details, including information on how some projects led to further research. All but one of the case studies with high impact had successful peer-reviewed publications and engaged in active dissemination. Although researchers were generally satisfied with NCCHTA, some complained about lengthy procedures and one about changes in study design. The pre- and post-interview scoring showed reasonable correlations and high inter-rater reliability. This indicated that most researchers were not making exaggerated claims for impact in their questionnaire responses.

Authors' recommendations: Conclusions This study concluded that the HTA Programme has had considerable impact in terms of knowledge generation and perceived impact on policy and to some extent on practice. This high impact may have resulted partly from the HTA Programme's objectives, in that topics tend to be of relevance to the NHS and have policy customers. The required use of scientific methods, notably systematic reviews and trials, coupled with strict peer reviewing, may have helped projects publish in high-quality peer-reviewed journals. Implications for healthcare It could be argued on the basis of the review that the NHS would benefit from an expansion of the HTA Programme, and that more should be done to encourage NHS customers to seek research relevant to their own work with a view to changing practice. Implications for the HTA Programme Recommendations were made on how the HTA Programme could improve, including: more rapid commissioning and publication improved collection of data on publications better archiving of documents relating to the origination, development and commissioning of research projects minimising late changes in research project design increased feedback to researchers on the dissemination of their reports, particularly on the large numbers of web views and downloads. Recommendations for research Three main areas for further research were identified: More detailed, comprehensive case studies, based on the payback approach, should be undertaken on selective projects within research programmes. Further enhancement of the 'payback framework' would be useful. The impact assessment provided lessons on how the payback framework might be improved, specifically how the questionnaire might be amended in the light of the case studies, and how factors associated with high impact can be further refined. A project that collated health research impact studies in an ongoing manner and analysed them in a consistent fashion would be valuable.

Project Status: Completed

URL for project: http://www.hta.ac.uk/1440

Year Published: 2007

English language abstract: An English language summary is available

Publication Type: Not Assigned

Country: England, United Kingdom

Organisation Name: NIHR Health Technology Assessment programme

Contact Address: NIHR Journals Library, National Institute for Health and Care Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

Contact Name: journals.library@nihr.ac.uk

Contact Email: journals.library@nihr.ac.uk

Copyright: 2009 Queen's Printer and Controller of HMSO