Authors' objectives:

This review explores those issues related to the process of randomisation that may affect the validity of conclusions drawn from the results of RCTs and non-randomised studies.

Authors' results and conclusions: Previous comparisons of RCTs and non-randomised studies Eighteen papers that directly compared the results of RCTs and prospective non-randomised studies were found and analysed. No obvious patterns emerged; neither the RCTs nor the non-randomised studies consistently gave larger or smaller estimates of the treatment effect. The type of intervention did not appear to be influential, though more comparisons need to be conducted before definite conclusions can be drawn. Several reasons emerged as to why RCTs might produce a greater or lesser estimate of treatment effect than non-randomised studies. A greater effect may occur in RCTs if patients receive higher quality care or are selected in a way that gives greater capacity to benefit. A lower estimate of treatment effect may occur if: 1. patient selection produces a study population with less capacity to benefit than would be the case in non-randomised studies 2. strong patient preference exists against a particular treatment in an unblind RCT, thus reducing the treatment effect 3. non-randomised studies of preventive interventions include a disproportionate number of people with greater capacity to benefit 4. publication bias exists; negative results are less likely to be published from non-randomised trials than from RCTs. Exclusions The number of eligible subjects included in the RCTs ranged from 1% to 100%. Reasons for exclusions may be medical (e.g. high risk of adverse events in certain groups) or scientific (selecting only small homogeneous groups in order to increase the precision of estimated treatment effects). Blanket exclusions (e.g. the elderly, women of childbearing potential) are also common in RCTs. Large clinical databases containing detailed information on patient severity and prognosis have been used instead of RCTs, and where database subjects are selected according to the same inclusion criteria as RCTs, the treatment effects of the two methods are similar. Participation Most RCTs failed to document adequately the characteristics of eligible individuals who did not participate in trials. However, RCTs were more likely than non-randomised trials to include university and teaching centres and this may have exaggerated the treatment effect measured in the RCTs. Participation in RCTs differed between studies of treatment interventions (subjects tended to be less affluent, less educated and more severely ill and therefore had greater capacity to benefit from treatment) and those evaluating preventive interventions (more affluent, better educated and generally healthier and therefore had less potential to benefit than eligible subjects who declined to participate). Adjusting for baseline differences Adjustment for differences in baseline prognostic factors in non-randomised studies often changed the treatment effect size but not significantly; importantly, the direction of change was inconsistent. Most of the case studies were too small to draw conclusions but where this was possible, the superiority of one treatment over another was probably a function of the patients' clinical characteristics. Patient preference Only four papers directly addressed the role of patient preference on trial results. However, preference could account for some of the observed differences between RCTs and non-randomised studies.

Authors' recommendations: Results of RCTs and non-randomised studies do not inevitably differ, and the available evidence suffers from many limitations. It does, however, suggest that it may be possible to minimise any differences by ensuring that subjects included in each type of study are comparable. The effect of adjustment for baseline differences between groups in non-randomised studies is inconsistent but, where it is done, it should involve rigorously developed formulae. Existing studies have generally been too small to assess the impact of such adjustment.

Authors' methods: Systematic review

Project Status: Completed

URL for project: http://www.hta.ac.uk/931

Year Published: 1998

English language abstract: An English language summary is available

Publication Type: Not Assigned

Country: England, United Kingdom

Organisation Name: NIHR Health Technology Assessment programme

Contact Address: NIHR Journals Library, National Institute for Health and Care Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

Contact Name: journals.library@nihr.ac.uk

Contact Email: journals.library@nihr.ac.uk

Copyright: 2009 Queen's Printer and Controller of HMSO