Authors' objectives:

We set out to assemble a large dataset of language restricted and language inclusive systematic reviews, including both conventional medicinal (CM) and complementary and alternative medicine (CAM) interventions. We also assessed the quality of these different types of systematic reviews and their associated RCTs and compared the quality of systematic reviews investigating a CM intervention with those reviews examining CAM interventions. We also examined whether language restrictions compared with language inclusions exaggerate the estimates of an interventions effectiveness. Finally we evaluated whether language restrictions of conventional interventions are similar to those for CAM interventions, and whether these results are influenced by other issues, including statistical heterogeneity and publication bias, in the systematic review process.

Authors' results and conclusions: We included 130 systematic reviews: 50 language restricted, 32 language inclusive/EL and 48 language inclusive/languages other than English (LOE) systematic reviews. Approximately 20% of the reviews were investigating CAM. The language inclusive/LOE reviews included the largest number of RCTs and participants. The LOE RCTs were published predominantly in French and German. Language inclusive/LOE systematic reviews were of the highest quality compared with the other types of reviews, scoring 57% of the maximum possible score. The CAM reviews were of higher quality, averaging 71% of their maximum possible score, compared with the CM reviews. There were only minor differences in the quality of reports of RCTs published in English compared with the eight other languages included in this analysis. However, there are inconsistent differences in the quality of LOE reports depending upon the type of intervention. The present results, and those reported previously, suggest that excluding reports of RCTs in LOE from the analytical part of a systematic review is a reasonable way to conduct a review [random effects model (RE) ROR = 1.02; 95% confidence interval (CI): = 0.83 to 1.26]. Because the present research and previous efforts have not included every type of CM RCT and the resulting possibility of the uncertainty as to when bias will be present by excluding LOE, it is always prudent to perform a comprehensive search for all evidence. This result only applies to reviews investigating the benefits of CM interventions. This does not imply that systematic reviewers should neglect reports in LOE. We recommend that systematic reviewers search for reports regardless of the language of their publication. There may be merit in including them in some aspects of the review process although this decision is likely to depend on several factors, including fiscal and other resources being available. However, language restrictions significantly shift the estimates of an interventions effectiveness when the intervention is CAM. Here, excluding trials reported in LOE, compared with their inclusion, resulted in a reduced intervention effect, 63% on average (RE ROR = 1.63; 95% CI: 1.03 to 2.60). The present results do not appear to be influenced by statistical heterogeneity and publication bias.

Authors' recommendations: With the exception of CAM systematic reviews, the quality of recently published systematic reviews is less than optimal. Language inclusive/LOE systematic reviews appear to be a marker for a better quality systematic review. Language restrictions do not appear to bias the estimates of a conventional intervention's effectiveness. However, there is substantial bias in the results of a CAM systematic review if LOE reports are excluded from it.

Project Status: Completed

URL for project: http://www.hta.ac.uk/1092

Year Published: 2003

English language abstract: An English language summary is available

Publication Type: Not Assigned

Country: England, United Kingdom

Organisation Name: NIHR Health Technology Assessment programme

Contact Address: NIHR Journals Library, National Institute for Health and Care Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

Contact Name: journals.library@nihr.ac.uk

Contact Email: journals.library@nihr.ac.uk

Copyright: 2009 Queen's Printer and Controller of HMSO