Authors' objectives:

The aims of this study were:

- To examine the characteristics of clinical trials that are difficult to locate (unpublished trials, trials published in languages other than English, trials published in journals not indexed in the MEDLINE database) and of trials of lower quality (inadequate/unclear concealment of treatment allocation, not double-blind).

- To compare within meta-analyses the treatment effects reported in trials that are difficult to locate with trials that are more accessible, and of trials of lower with trials of higher quality.

- To assess the impact of excluding trials that are difficult to locate and of trials of lower quality on pooled effect estimates, p-values and the shape of funnel plots.

Authors' results and conclusions: A total of 159 systematic reviews met the inclusion criteria but not all included trials that are difficult to locate. Comparisons of treatment effects were based on the following: - unpublished versus published (60 meta-analyses) - other languages versus English (50 meta-analyses) - non-indexed versus MEDLINE-indexed (66 meta-analyses). Analyses of trial quality were based on: - inadequately concealed/unclear versus adequately concealed (39 meta-analyses) - not double-blind versus double-blind (45 meta-analyses). The importance of trials that are difficult to locate appears to vary across medical specialities. For example, unpublished trials are particularly prevalent in oncology whereas trials published in languages other than English and trials published in sources not indexed in MEDLINE are important in psychiatry, rheumatology and orthopaedics. A large proportion of trials of complementary medicine are difficult to locate. Unpublished trials show less beneficial effects than published trials whereas non-English language trials and non-indexed trials tend to show larger treatment effects. Trials that are difficult to locate tend to be smaller and of lower methodological quality than trials that are easily accessible and published in English. Trials with inadequate or unclear concealment of allocation show more beneficial effects than adequately concealed trials. Similarly, open trials tend to be more beneficial than double-blind trials. In the majority of meta-analyses exclusion of trials with inadequate or unclear concealment and trials without double-blinding led to a change towards less beneficial treatment effects, which was often substantial. Including unpublished trials reduces funnel plot asymmetry whereas the inclusion of trials published in languages other than English and of non-indexed trials increases the degree of asymmetry in the funnel plot. The impact of trials of lower methodoloqical quality on the funnel plot is substantial for trials with inadequate or unclear concealment of allocation.

Authors' recommendations: Systematic reviews that are based on a search of English language literature that is accessible in the major bibliographic databases will often produce results that are close to those obtained from reviews based on more comprehensive searches that are free of language restrictions. We recommend that when planning a review, investigators should consider the type of literature search and the degree of comprehensiveness that are appropriate for the review in question, taking into account budgetary and time constraints. The finding that trials which are difficult to locate are often of lower quality raises the worrying possibility that rather than preventing bias through extensive literature searches, bias could be introduced by including trials of low methodological quality. We believe that in situations where resources are limited, thorough quality assessments should take precedence over extensive literature searches and translations of articles. Our results confirm that the funnel plot and the regression method to assess funnel plot asymmetry are useful to detect small-study effects, the tendency for smaller studies in a meta-analysis to show larger treatment effects.

Authors' methods: Empirical study

Project Status: Completed

URL for project: http://www.hta.ac.uk/1099

Year Published: 2003

English language abstract: An English language summary is available

Publication Type: Not Assigned

Country: England, United Kingdom

Organisation Name: NIHR Health Technology Assessment programme

Contact Address: NIHR Journals Library, National Institute for Health and Care Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

Contact Name: journals.library@nihr.ac.uk

Contact Email: journals.library@nihr.ac.uk

Copyright: 2009 Queen's Printer and Controller of HMSO