Original Article
In meta-analyses of proportion studies, funnel plots were found to be an inaccurate method of assessing publication bias

https://doi.org/10.1016/j.jclinepi.2014.03.003Get rights and content

Abstract

Objective

To assess the utility of funnel plots in assessing publication bias (PB) in meta-analyses of proportion studies.

Study Design and Setting

Meta-analysis simulation study and meta-analysis of published literature reporting peri-operative mortality after abdominal aortic aneurysm (AAA) repair. Data for the simulation study were stochastically generated. A literature search of Medline and Embase was performed to identify studies for inclusion in the published literature meta-analyses.

Results

The simulation study demonstrated that conventionally constructed funnel plots (log odds vs. 1/standard error [1/SE]) for extreme proportional outcomes were asymmetric despite no PB. Alternative funnel plots constructed using study size rather than 1/SE showed no asymmetry for extreme proportional outcomes. When used in meta-analyses of the mortality of AAA repair, these alternative funnel plots highlighted the possibility for conventional funnel plots to demonstrate asymmetry when there was no evidence of PB.

Conclusion

Conventional funnel plots used to assess for potential PB in meta-analyses are inaccurate for meta-analyses of proportion studies with low proportion outcomes. Funnel plots of study size against log odds may be a more accurate way of assessing for PB in these studies.

Introduction

What is new?

Key findings

  1. Conventional funnel plots appear to be an inaccurate way of assessing publication bias (PB) in meta-analyses of proportion studies with extreme proportional outcomes.

  2. Funnel plots of study size against log odds may be a more accurate method of assessing PB and are robust across all proportional outcomes.

What this adds to what is known?
  1. Funnel plots are the commonest statistical method of assessing PB but the utility and accuracy of such plots in meta-analyses of proportion studies have not been addressed.

What is the implication, what should change now?
  1. Our study brings into question the accuracy of using traditional funnel plots as a method of assessing publication in meta-analyses of proportion studies and introduces an alternative funnel plot that can be used for analysis.

Evidence-based medicine is dependent on an adequate understanding of the literature. Meta-analyses are frequently put forward as the highest level of evidence on any given topic. However, as with all statistical techniques, the interpretation of data presented in meta-analyses is subject to error. Proportion data from observational studies and other sources are frequently used by clinicians to counsel patients and to compare practices and are frequently combined in meta-analyses. Meta-analyses should contain an assessment of the potential for publication bias (PB) to have influenced the results of the analysis. PB can occur when studies with statistically significant results or clinically favorable results are preferentially published. Indeed, studies that have statistically significant results are twice as likely to be published as null studies [1]. Furthermore, studies that have positive findings are both more likely to be published and published more quickly than those with negative findings [2]. The exact prevalence of PB is impossible to ascertain but it is estimated that about 50% of the literature on any given topic is unpublished [3], [4].

As a group of academic vascular surgeons, we have developed an interest in the use of meta-analysis to estimate outcomes after vascular surgery [5], [6], [7]. In this work, we have used funnel plots [8] to assess our datasets for the presence of potential PB (which we suspect we have seen on a number of occasions). In our field, we frequently assess procedures with low proportional outcomes such as elective abdominal aortic aneurysm (AAA) repair (peri-operative mortality rate ≈ 4%) and have developed concerns that conventional methods for constructing funnel plots for these low proportional outcomes may be over-estimating the degree of funnel plot asymmetry, concerns shared by other authors [9], [10], [11]. Previous work has addressed this issue in meta-analyses of randomised control trials (RCTs) and in the assessment of diagnostic tests but not in meta-analyses of proportion data. These studies focused on the accuracy of funnel plots in detecting PB and investigated alternative methods of plotting the vertical axis including using standard error (SE), precision (1/SE), variance, and sample size [9], [12], [13], [14]. Furthermore, Tang [15] has previously demonstrated, using meta-analyses of RCTs, that the inclusion of a single study with a much lower proportional outcome than the rest of the included studies disproportionately distorts the results. The author suggested methods to avoid introducing bias into such a scenario including the assessment of the effect of study size on the weighting of the results. However these findings have yet to be assessed in meta-analyses of proportion studies [15]. Low proportional outcomes are not isolated to proportion and non-comparative data and also occur in interventional research. For the purpose of this study, ‘events’ in the simulation dataset detailed in the following relate to mortality but could also be an expression of an alternative outcome such as those observed in interventional research. In this study, we have focused on proportion data.

Many authors construct funnel plots with outcome (x-axis) against the SE (or its reciprocal) as a measure of individual study size and variability (y-axis) [12]. However, this approach may not be suitable for all analyses. Meta-analyses of observational or non-comparative studies are commonly performed where the outcome estimates are simply the proportions of concerned outcomes observed in individual studies, and the outputs from these studies are frequently used for decision models and economic evaluations and also for their descriptive value. These types of analyses often assess extreme proportional outcomes. For example, meta-analyses of mortality in endovascular AAA repair will have outcome values as low as 3%. Some time ago [9], the observation was made that for relative effect measures, such as odds ratios and relative risks, the SE of the outcome is actually correlated with the size of the effect. Similarly, for a proportion, which is commonly transformed to the log odds scale due to better statistical properties for meta-analysis, its SE is dependent on the value of the log odds (and the underlying proportion). Consider a study in which r out of n patients were observed to have an event, leading to a proportion of r/n. The associated log odds is ln(r/(nr)) with SE sqrt(1/r + 1/(nr)). It can be seen by substituting different r's into the formulae for a fixed n that the SE naturally increases as r approaches 0 or n (and thus P approaches 0 or 1). In Fig. 1, for a fixed study size of 100, we plot the curve generated by plotting the SE of the log odds against the log odds. Observe the “U” shaped relationship, compared with plotting the log odds against sample size, which–of course–results in a straight horizontal line. This relationship will distort the appearance of a funnel plot using SE (or a function of it) as the vertical axis, particularly when the underlying proportion is quite extreme and there may be a risk of attributing funnel plot asymmetry to PB for such outcomes when, in fact, the funnel plot asymmetry is nothing more than an artifact of the method of funnel plot construction. Previous work has focused on effect measures including relative risk and weighting bias to assess whether funnel plot asymmetry is incorrectly attributed to PB [15]. We hypothesized that study size, which is not influenced by extreme outcomes measured on a log odds scale (Fig. 1) is a better value for the vertical axis of funnel plots when assessing meta-analyses of non-comparative studies.

The aim of this study therefore, was to test this hypothesis using both simulation and real data. Additionally, we evaluate the utility of conventional and alternative funnel plots in this context.

Section snippets

Methods

We simulated meta-analysis datasets in which no PB existed. Specifically, we generated a meta-analytic dataset comprising 100 simulated (single arm) studies. The sample size of each of the studies was determined stochastically by sampling from an exponential distribution (λ = 1) Studies with sample sizes less than 10 were removed from this dataset. For each simulated study, the number of patients experiencing events was generated using a binomial distribution. Five separate outcomes for each

Methods

Studies reporting the peri-operative mortality of AAA were identified to assess the utility of the alternative (study size–based) funnel plots used in the simulation study. This subject was chosen because it is an area well known to the authors, and the recent evolution of surgical techniques from open to endovascular techniques has yielded a large volume of data. Furthermore, AAA repair can be considered as four distinct categories (elective open AAA repair [EAAA], elective endovascular AAA

Results

A total of 15 meta-analyses reporting the outcomes of AAA repair were identified [5], [6], [7], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], and data from the individual studies within each of these analyses were used to construct funnel plots for the peri-operative mortality of RAAA, EAAA, EVAR, and REVAR. We first constructed conventional funnel plots of log odds against the inverse of the SE log odds for each contributing study and then compared these with funnel

Discussion

This is the first study to formally assess the utility of funnel plots for the detection of PB in meta-analyses of non-comparative, proportion studies. In this study, we have demonstrated that funnel plots, as traditionally constructed, may be a potentially misleading method for the assessment of PB in such analyses. This is particularly true for low or high proportional outcomes, and there is the potential that funnel plot asymmetry in these circumstances could be mistakenly interpreted as

Acknowledgments

Author contributions: J.P.H. performed the literature search, analyzed the data, produced the figures, and alongside M.J.B. co-wrote the manuscript. A.S. performed a literature search and assisted in production of the figures. A.J.S. was involved in study design, statistical analysis, and was available for statistical advice and edited the manuscript. R.H.B. assisted with statistical analysis and production of figures. R.D.S., alongside M.J.B., conceived the study and edited the manuscript.

References (32)

  • M.S. Sajid et al.

    Endovascular aortic aneurysm repair (EVAR) has significantly lower perioperative mortality in comparison to open repair: a systematic review

    Asian J Surg

    (2008)
  • I.M. Nordon et al.

    Secondary interventions following endovascular aneurysm repair (EVAR) and the enduring value of graft surveillance

    Eur J Vasc Endovasc Surg

    (2010)
  • R.W. Scherer et al.

    Full publication of results initially presented in abstracts

    Cochrane Database Syst Rev

    (2007)
  • N.A. Constantine

    Publication bias

  • N. Siddiqi

    Publication bias in epidemiological studies

    Cent Eur J Public Health

    (2011)
  • F. Song et al.

    Dissemination and publication of research findings: an updated review on related biases

    Health Technol Assess

    (2010)
  • Cited by (0)

    Competing interests: None.

    The study was an analysis of previously published data and as such no ethical approval was required.

    Funding: No specific funding was sought and costs were covered by departmental funds.

    View full text