What is new?
Key findings- •
Conventional funnel plots appear to be an inaccurate way of assessing publication bias (PB) in meta-analyses of proportion studies with extreme proportional outcomes.
- •
Funnel plots of study size against log odds may be a more accurate method of assessing PB and are robust across all proportional outcomes.
What this adds to what is known?- •
Funnel plots are the commonest statistical method of assessing PB but the utility and accuracy of such plots in meta-analyses of proportion studies have not been addressed.
What is the implication, what should change now?- •
Our study brings into question the accuracy of using traditional funnel plots as a method of assessing publication in meta-analyses of proportion studies and introduces an alternative funnel plot that can be used for analysis.
Evidence-based medicine is dependent on an adequate understanding of the literature. Meta-analyses are frequently put forward as the highest level of evidence on any given topic. However, as with all statistical techniques, the interpretation of data presented in meta-analyses is subject to error. Proportion data from observational studies and other sources are frequently used by clinicians to counsel patients and to compare practices and are frequently combined in meta-analyses. Meta-analyses should contain an assessment of the potential for publication bias (PB) to have influenced the results of the analysis. PB can occur when studies with statistically significant results or clinically favorable results are preferentially published. Indeed, studies that have statistically significant results are twice as likely to be published as null studies [1]. Furthermore, studies that have positive findings are both more likely to be published and published more quickly than those with negative findings [2]. The exact prevalence of PB is impossible to ascertain but it is estimated that about 50% of the literature on any given topic is unpublished [3], [4].
As a group of academic vascular surgeons, we have developed an interest in the use of meta-analysis to estimate outcomes after vascular surgery [5], [6], [7]. In this work, we have used funnel plots [8] to assess our datasets for the presence of potential PB (which we suspect we have seen on a number of occasions). In our field, we frequently assess procedures with low proportional outcomes such as elective abdominal aortic aneurysm (AAA) repair (peri-operative mortality rate ≈ 4%) and have developed concerns that conventional methods for constructing funnel plots for these low proportional outcomes may be over-estimating the degree of funnel plot asymmetry, concerns shared by other authors [9], [10], [11]. Previous work has addressed this issue in meta-analyses of randomised control trials (RCTs) and in the assessment of diagnostic tests but not in meta-analyses of proportion data. These studies focused on the accuracy of funnel plots in detecting PB and investigated alternative methods of plotting the vertical axis including using standard error (SE), precision (1/SE), variance, and sample size [9], [12], [13], [14]. Furthermore, Tang [15] has previously demonstrated, using meta-analyses of RCTs, that the inclusion of a single study with a much lower proportional outcome than the rest of the included studies disproportionately distorts the results. The author suggested methods to avoid introducing bias into such a scenario including the assessment of the effect of study size on the weighting of the results. However these findings have yet to be assessed in meta-analyses of proportion studies [15]. Low proportional outcomes are not isolated to proportion and non-comparative data and also occur in interventional research. For the purpose of this study, ‘events’ in the simulation dataset detailed in the following relate to mortality but could also be an expression of an alternative outcome such as those observed in interventional research. In this study, we have focused on proportion data.
Many authors construct funnel plots with outcome (x-axis) against the SE (or its reciprocal) as a measure of individual study size and variability (y-axis) [12]. However, this approach may not be suitable for all analyses. Meta-analyses of observational or non-comparative studies are commonly performed where the outcome estimates are simply the proportions of concerned outcomes observed in individual studies, and the outputs from these studies are frequently used for decision models and economic evaluations and also for their descriptive value. These types of analyses often assess extreme proportional outcomes. For example, meta-analyses of mortality in endovascular AAA repair will have outcome values as low as 3%. Some time ago [9], the observation was made that for relative effect measures, such as odds ratios and relative risks, the SE of the outcome is actually correlated with the size of the effect. Similarly, for a proportion, which is commonly transformed to the log odds scale due to better statistical properties for meta-analysis, its SE is dependent on the value of the log odds (and the underlying proportion). Consider a study in which r out of n patients were observed to have an event, leading to a proportion of r/n. The associated log odds is ln(r/(n−r)) with SE sqrt(1/r + 1/(n−r)). It can be seen by substituting different r's into the formulae for a fixed n that the SE naturally increases as r approaches 0 or n (and thus P approaches 0 or 1). In Fig. 1, for a fixed study size of 100, we plot the curve generated by plotting the SE of the log odds against the log odds. Observe the “U” shaped relationship, compared with plotting the log odds against sample size, which–of course–results in a straight horizontal line. This relationship will distort the appearance of a funnel plot using SE (or a function of it) as the vertical axis, particularly when the underlying proportion is quite extreme and there may be a risk of attributing funnel plot asymmetry to PB for such outcomes when, in fact, the funnel plot asymmetry is nothing more than an artifact of the method of funnel plot construction. Previous work has focused on effect measures including relative risk and weighting bias to assess whether funnel plot asymmetry is incorrectly attributed to PB [15]. We hypothesized that study size, which is not influenced by extreme outcomes measured on a log odds scale (Fig. 1) is a better value for the vertical axis of funnel plots when assessing meta-analyses of non-comparative studies.
The aim of this study therefore, was to test this hypothesis using both simulation and real data. Additionally, we evaluate the utility of conventional and alternative funnel plots in this context.