Authors' objectives: This project focused on Agency for Healthcare Research and Quality (AHRQ) methods guidance to its Evidence-based Practice Center (EPC) program on grading the strength of evidence (SOE) related to therapeutic interventions. Our project focused on inter-rater reliability testing of the two main components of the AHRQ approach to grading SOE for specific outcomes: (1) scoring evidence on the four required domains (risk of bias, consistency, directness, and precision), separately for randomized controlled trials (RCTs) and observational studies, and (2) developing an overall SOE grade, given the scores for the individual domains.

Authors' recommendations: Our findings demonstrate that the conclusions reached by experienced reviewers based on the same evidence can differ greatly, particularly when they are faced with bodies of evidence that do not lend themselves to meta-analysis and they need to rely more heavily on their own judgment. Of particular concern is how to deal with (a) outcomes that are evaluated through a combination of RCTs and observational studies, (b) outcomes that are evaluated through more than one measure and (c) grading evidence that appears to show no difference. We conclude that additional methodological guidance is needed, including more details and examples, supported by more training, particularly on how best to evaluate the "thornier" bodies of evidence as discussed above. However, some potential will always exist for disagreement even among experienced reviewers. EPC reviewer teams need to be transparent in how they have conducted this task. This will help to ensure that stakeholders can be confident of their interpretation of the evidence. Our study provided only a first approximation of reviewers' rationales for differences in SOE decisions. Additional research is needed to understand gaps in guidance that should be filled, areas of insufficient understanding of the guidance itself and how best to overcome that deficit, and complex decisions that may still need to be left to the review team's substantive expertise.

Project Status: Completed

Year Published: 2012

URL for published report: http://www.effectivehealthcare.ahrq.gov/ehc/products/339/1099/Methods_ReliabilityTesting_FinalReport_20120522.pdf

English language abstract: An English language summary is available

Publication Type: Not Assigned

Country: United States

Organisation Name: Agency for Healthcare Research and Quality

Contact Address: Center for Outcomes and Evidence Technology Assessment Program, 540 Gaither Road, Rockville, MD 20850, USA. Tel: +1 301 427 1610; Fax: +1 301 427 1639;

Contact Name: martin.erlichman@ahrq.hhs.gov

Contact Email: martin.erlichman@ahrq.hhs.gov

Copyright: <p>Agency for Healthcare Research and Quality (AHRQ)</p>