Authors' recommendations: The key results are summarised below:1) The majority of G-I-N member organisations used the AGREE tool (or the adapted version of the AGREE tool) as the quality system for guideline development programmes. Systematicreviews on the use of evaluation tools for assessment of guidelines also found that AGREE was an optimum tool (supplementary document available upon request). 2) Several international organisations based in the United States, Canada, as well as the World Health Organisation use GRADE as a preferred system for evaluation of evidence. GRADE provides a systematic process to identify, analyse, and present a large body of evidence and a transparent methodological approach for the development of evidence-based optimal therapy recommendations (Shukla et al., 2008).23 According to an evaluation made by the US Agency for Health Care Research Quality (AHRQ), GRADE and SIGN were among the best evaluated by a group of experts. This finding was also supported by a very recent systematic review by the CADTH (Canada) which adopted the same methodological approach to the AHRQ systematic review of rating systems for grading evidence supported the results above. In personal communication with the authors of this review, they indicated that the CADTH is adopting the GRADE system for new projects. A similar finding was found in a previous work on this topic by WHO on preliminary results from a non-systematic review of the literature on grading evidence and recommendations in guidelines. They have found a large body of work on the development and evaluation of various grading systems, and that GRADE was grading system to grade the quality of evidence and strength of recommendations that is sensible and is being widely used.3) Personal communication with experts on the evidence-based health electronic discussion group showed that GRADE and SIGN were among the best evaluated, followed by NICE and CEBM (information from the email is available upon request).4) The AHRQ conducted a comprehensive review on the majority of the grading systems being used (West, 2002). Information from the discussion section of the report that might be of importance when considering analysing results using a grading system is annotated here for reference. For further detailed information, it is advised that the readers consult the original document.The report by the AHRQ identified factors that are important when developing and using rating systems:Distinctions among types of studies, evaluation criteria, and systemsNumbers of quality rating systemsChallenges of rating observational studiesInstrument lengthReporting guidelinesConflicting findings when bodies of evidence contain different types of studiesOverall, many systems covered most of the domains that are considered generally informative for assessing study quality. From this set, the authors identified 19 generic systems that fully address the key quality domains (with the exception of funding or sponsorship for several systems). Three systems were used for both randomised controlled trials (RCTs) and observational studies.In the authors’ judgment, those who plan to incorporate study quality into a systematic review, evidence report, or technology assessment can use one or more of these 19 systems as a starting point, being sure to take into account the types of study designs occurring in the articles under review.The authors identified seven systems that fully addressed all three domains for grading the strength of a body of evidence. The earliest system was published in 1994 (Gyorkos et al., 1994). The remaining systems were published in 1999 (Clarke and Oxman 1999) and 2000 (Briss et al., 2000; Greer et al., 2000; Guyatt et al., 2000; NHS 2001; Harris et al., 2001) indicating that this is a rapidly evolving field.Systems for grading the strength of a body of evidence are much less uniform than those for rating study quality. This variability complicates the job of selecting one or more systems that might be put into use today.In conclusion, this brief survey of the existing literature has identified several desirable attributes of a grading system, including ease of use, perceived quality or validity of the grading system, and clarity of the output or time taken. Based on these considerations, HSAC identified the five tools that are more frequently used and highly rated worldwide. In alphabetical order, these are AGREE, GRADE, NICE, OCEBM, and SIGN. There is significant heterogeneity among different ‘interest groups’. There is, therefore, a need for a uniform system of grading the rapidly generated evidence so that it can be effectively utilized in clinical practice.

Project Status: Completed

URL for project: http://www.healthsac.net/downloads/publications/HSAC26%20Grading%20Evidence%20030909%20FINAL.pdf

Year Published: 2009

English language abstract: An English language summary is available

Publication Type: Not Assigned

Country: New Zealand

Organisation Name: Health Services Assessment Collaboration

Contact Address: University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand

Contact Name: hsac@canterbury.ac.nz

Contact Email: hsac@canterbury.ac.nz

Copyright: Health Services Assessment Collaboration (HSAC)