Author Reply

Editor – Hackshaw et al¹ raise some methodologic concerns regarding our re-analysis and suggest on the basis of these that their original analysis² leads to a more valid conclusion than our re-analysis.³ The first point raised was that our re-analysis of the intervention studies should have included only randomized controlled trials, implying that because the randomized controlled trial is a more valid study design for causal inference compared with the observational study design, the inclusion of the latter in our re-analysis is likely to be affected by inherent biases and confounding.¹ The problem with this argument is that confounding can occur by chance or by indication.⁴ When confounding occurs by chance, it will occur with the same probability in randomized controlled trials and observational studies because it is, by definition, due to chance. In both randomized controlled trials and observational studies, the P-value automatically incorporates the uncertainty due to confounding by chance.⁴ Furthermore, in the context of a meta-analysis, confounding by chance in one direction in one study is expected to be matched by confounding by chance in the other direction in another study.⁴

The more important problem with observational studies is confounding by indication. If radioiodine is allocated in an observational study, we can envisage three scenarios.⁴ First, the allocator is not knowledgeable about important confounders, in which case it is highly unlikely that they will be causally related to exposure to treatment, and any unequal distribution must have occurred solely by chance. Second, radioiodine could have been allocated by someone who does not know the risk status of the patient, and thus, the situation is similar to the first scenario. Third, if radioiodine was allocated by someone who knew there was a high versus low chance for ablation (for example, had surgical status information and understood what this means), and thus could influence treatment allocation, there can be a bias. In the case of radioiodine, this sort of influence, however, is likely to be a higher dose bias towards high-risk patients and a lower dose bias towards low-risk patients leading to non-differential bias.⁵ This again suggests, under the constraints of our re-analysis, that the risk of confounding in an observational study of this sort should be minimal, and including observational studies in this meta-analysis would only lead to increased precision of the estimate.⁴ Therefore, the statement by Hackshaw et al¹ that our “meta-analyses of observational studies need to be interpreted with great care because the effect of potential confounding in each study could be magnified when the studies are combined, thus producing a spuriously precise effect” may be true for confounding by indication in other observational studies, but not in this series of studies on radioiodine.

The second point Hackshaw et al¹ make is that when our re-analysis was restricted to the six randomized trials (in which they correctly indicate that no comparison was based on more than 150 patients in total), the pooled relative risk was 0.68 (95% confidence interval [CI] 0.43–1.07), which was not statistically significant (P = 0.093). They claim that the correct interpretation of this is that while there is some evidence of a lower ablation rate when using a low dose, this effect could be due to chance, and they question why we did not comment on this. We did not think this important because this line of reasoning is based on the assumption that randomization removes the chance of confounding; however, we have just pointed out that including information from observational studies may actually improve the inference based on only randomized controlled trials. Furthermore, a review of empirical studies suggests that meta-analyses based on observational studies can also produce estimates of effect similar to those from meta-analyses based on randomized controlled trials.^6,7 Indeed, these and other authors^4,7,8 have found that the discrepancies between observational studies and randomized controlled trials are highly impacted upon by the quality of the studies. What is interesting to note is that Hackshaw et al² appear to have made no attempt to assess the methodologic quality of studies in their meta-analysis, even though there is now empiric evidence that would mandate some form of quality assessment of primary studies in meta-analysis.^9,10 Indeed, a minimum requirement of heterogeneity is to look up the list of studies to find out why they differ.¹¹ We did assess the quality of every study, and more importantly, we did not simply link it to the interpretation of results or use it to limit the scope of the review.¹² Instead, we created a new and robust method of meta-analysis that did justice to the quality information in this synthesis,^12,13 and our conclusions remained unchanged.

Third, Hackshaw et al suggest that the “dose level” (activity of I-131) that defines ‘low’ and ‘high’ differs greatly between these trials. For example, a low dose could be 30 or 50 mCi (a difference of two-thirds), and a high dose could be 50 or 100 mCi (a difference of two-fold). The fundamental error here is that radiation dosimetry is not an exact science and involves huge approximations and thus a difference of two-thirds or two-fold does not translate into an effect change of an exact magnitude.¹⁴ Also, the use of a meta-analysis in this regard is only problematic (more so for observational studies) if we need to know the estimate of the effect of a particular dose of radioiodine, as opposed to if we need to show that there is an effect on average of a higher versus lower dosage. The same argument also stands for the combination of observational and randomized controlled trial data because here we need only to show that the observational studies agree with randomized controlled trials on average.¹⁵

Finally, we have to disagree with Hackshaw et al’s conclusions for a second time. Once these considerations have been taken into account, it is misleading to rule out the potentially greater effectiveness of a higher (as opposed to high) dose based on current evidence. Based on the majority of studies, this would be in the range of 2775 3700 MBq. We look forward to the results of future randomized controlled trials that will help to determine whether a low dose should be avoided or could be used instead of a high dose.

• Received January 22, 2009.
• Revision received February 16, 2009.
• Accepted March 4, 2009.