815 844
Table 3 – Checklist of points to consider when the findings from a systematic review and meta-analysis differ with those from a large
randomized controlled trial
Criteria to consider
Questions to ask
Rationale
Selection bias
Were the sequence generation and allocation
concealment adequate in both the studies included
in the SR/MA and the subsequent
[4_TD$DIFF]
RCT?
If the sequence generation was not truly random or the
allocation was not effectively concealed, this can lead to
exaggerated estimates in individual studies, and these may be
amplified in MAs.
Confounding bias
Were the groups balanced for known prognostic
factors at baseline and were any imbalances
controlled for in the analysis?
Imbalances in known and unknown prognostic factors are
possible even in well-designed RCTs. Baseline imbalances may
explain differences in estimates of effect if not controlled for in
the analysis.
Performance and
detection bias
Where possible, in all the studies included in the SR/
MA and for the new
[4_TD$DIFF]
RCT, was there blinding of study
participants, clinicians administering the treatment,
ancillary care-givers, and outcomes
[5_TD$DIFF]
assessors?
When blinding is not possible, could knowledge of
the treatment received affect interpretation of any
of the outcomes?
Some objective outcomes are unlikely to be affected by
knowledge of the intervention arm, but failure to blind
(particularly for subjective outcomes) may lead to an
exaggeration of effect sizes in individual studies, and these may
be amplified in MAs.
Attrition bias
Were all dropouts documented and unlikely to be
related to the treatment outcome in the studies
included in the SR/MA and in the new
[4_TD$DIFF]
RCT?
If dropout rates differ between the treatment arms, then the
reasons may be related to the outcome of interest and may hide
important outcome effects.
Reporting bias
Were all outcomes that were stated in the methods
and/or protocol for all the studies included in the
SR/MA and in the new
[4_TD$DIFF]
RCT documented in the trial
report?
Were all the outcomes measured appropriately (as
defined in the protocol) or were deviations
reasonably explained?
Selective reporting of outcomes, or selective methods of
reporting, may lead to exaggerated estimates of effect.
Publication bias
Were funnel plots used to investigate publication
bias in the SR/MA? Is the funnel plot symmetric or is
there reason to believe there is a systematic
difference between published and unpublished
studies?
Note: This is difficult to assess when there are
<
10 RCTs contributing to an MA.
Asymmetric funnel plots raise suspicion that there are
systematic differences between published and unpublished
studies and that some positive or negative trials may be
unpublished. This may lead to exaggerated effect sizes in an
MA.
Consistency and
heterogeneity
of outcome
Did the studies included in the SR/MA have
overlapping 95% CIs for the outcome?
Was variation more than would be expected by
chance alone?
Was the
I
2
statistic
<
40%? (Cochrane/GRADE rule of
thumb)
Were subgroups used to explain any observed
heterogeneity?
Were event rates in the control group similar in the
different studies?
Note: Subgroups of the population, the intervention/
control types, or the outcome measurement may
explain heterogeneity.
If it can be shown that the outcomes are more effective in
certain subgroups, or with variations of an intervention (eg, a
higher dose), then this explained heterogeneity may indicate a
key difference that may justify the results in the new RCT.
Where unexplained heterogeneity exists, then the estimate of
effect is likely to be uncertain, even if precise.
Directness
Do the studies included in the SR/MA and the new
[4_TD$DIFF]
RCT both directly assess the research question about
the population, interventions, and outcomes?
Indirect populations, interventions, surrogate outcome
measures, or indirect comparisons may conceal or exaggerate
important differences within and between studies, and may
impact the estimate of effect.
Precision
Were the sample sizes for the studies included in
the SR/MA and the new
[4_TD$DIFF]
RCT powered to address the
outcomes of interest?
Does the 95% CI in the MA include clinically judged
appreciable benefit and harm?
If any of the trials in the SR/MA or the new
[4_TD$DIFF]
RCT were not
powered to detect a clinically meaningful difference in the
effect estimate, this may reduce confidence in the estimate of
effect.
If the lower and upper 95% CI thresholds indicate that the
intervention may be beneficial at one end, but harmful at the
other, this will probably reduce confidence in the estimate of
effect.
Sensitivity
analyses
When some studies included in an SR/MA are
judged to be at high RoB and others at low RoB, or
extreme variations in the populations or
interventions in the studies are apparent, did the
authors conduct a sensitivity analysis to ascertain
the estimates of effect for only studies judged to be
at low RoB?
Sensitivity analyses are different from subgroup analyses. Some
studies are actively omitted as we are only interested in the
results when the biased or ‘‘different’’ studies are omitted.
SR = systematic review; MA = meta-analysis; RCT = randomized controlled trial; CI = confidence interval; RoB = risk of bias.
E U R O P E A N U R O L O G Y 7 1 ( 2 0 1 7 ) 8 1 1 – 8 1 9
815