Original ArticleStatistically significant papers in psychiatry were cited more often than others
Introduction
An article can be regarded as having been cited if it is mentioned in the list of references of another publication. Citation bias occurs when the chances of a paper being cited by others are associated in some way with its results. For example, authors of published articles may tend to cite studies that support their position. This criterion may operate as a filter that favors some research findings and thereby introduces a preferential citation bias into their interpretation by readers [1].
Preferential citations can be based on statistical significance of primary outcome [1] or, for example, on whether an article supports the effectiveness of a new intervention, or some other fashionable hypothesis in the field [2], [3]. Some authors define citation bias as preferential citations based on nationality or discipline of the publishing journal [4]. Citation bias is a specific type of dissemination bias. The wider term “dissemination bias” includes bias due to selective publication and other publication-related biases caused by the time, type and language of publication, selective citation of references, database index bias, and biased media attention [1].
Medical research is beset by dissemination bias. Selective publication, based on statistical significance, is well documented [1], [5]. More recently, evidence of selective reporting of statistically significant outcomes in randomized trials has emerged [6]. Here, we investigate selective citation bias based on statistical significance of the primary outcome in the psychiatric literature.
It has been suggested that authors are less likely to cite studies that were nonsignificant, negative, or neutral in outcome [1], [7], [8]. Arguably, more frequent citation of positive and statistically significant findings is likely to be an intrinsic consequence of the mechanical (mis)use of “hypothesis testing” methodology [9] carrying over to favor the inclusion of studies that support the proposed hypothesis more often than those that do not. If authors covertly select papers for citation in this way, they present a distorted view of the evidence to their readers, often leaving them with an exaggerated view of the medical evidence.
There is already some evidence of citation bias in clinical trials. Kjaergard and Gluud [10] found preferential citation of hepatobiliary randomized clinical trials reporting significant differences on the primary outcome. Comparison of frequency of citation of all controlled cholesterol lowering trials using coronary heart disease or death as the end point showed that supportive trials were cited almost six times more often than others, even though meta-analysis showed lowering cholesterol concentration was not beneficial [7]. A survey identified that even a database (ACP Journal Club) summarizing findings from original studies has a preference for summarizing therapeutic trials that have a positive outcome [11]. Koren and Klein [12] demonstrate that there is bias against negative studies in newspaper reports of medical research. However, Gluud [13] notes in her review that empirical evidence concerning selective citation bias of statistically significant articles is still scarce and not so convincing.
Analysis of the relationship between the outcomes of statistical tests and the citations received are likely to face several potential confounding factors. The prestige and visibility of journals is a potential confounding factor, which should be adjusted for. Further, different subfields of psychiatry emphasize different research methods differently, and this is reflected in the fact that some psychiatric journals publish more epidemiological reports than clinical trials or biological studies. Some authors have found the number of citations received to be associated with the study design [14] and the direction of the outcome [7], or that the poor quality of methodological reporting has led to exaggerated positive findings [10], [15], [16]. A larger number of coauthors has also been shown to increase the number of received citations [17], [18], [19]. Exclusion of authors' self-citations in the citation count can, and should in our view, be used to control this effect.
Previous work in this area has been limited to intervention trials. Further it has either been based on a relatively small number of studies [7] or has adjusted for a limited set of confounders [10]. Further, self-citation has not been excluded, nor sample size or journal taken into account. Here, we investigate preferential citation in the psychiatric literature, adjusting for these and other factors, in both noninterventional and intervention studies. We focus on psychiatric research, an area in which the first author has worked for many years.
Section snippets
Set of articles
We selected four general English-language psychiatric journals: The American Journal of Psychiatry (AJP), Archives of General Psychiatry (AGP), the British Journal of Psychiatry (BJP), and Nordic Journal of Psychiatry (NJP). AJP and AGP have consistently been the top two psychiatric research journals (Garfield's impact factor [IF] 7.61 and 11.21, respectively), whereas BJP is the most visible and cited psychiatric journal outside the United States (IF 4.18) [20]. NJP (IF 0.89) represents the
Results
The statistical significance of the primary outcome of articles in the four psychiatric journals is summarized in Table 1. All the eligible articles were classified according to whether or not the authors had reported or decided that the primary outcome was statistically significant or nonsignificant. If the results of formal statistical significance testing or confidence interval estimation were not included in the reporting of the primary finding, the article was classified as not evaluating
Discussion
The purpose of this work was to investigate whether papers with statistically significant findings on the primary outcome had an increased probability of being cited by other authors. We found articles containing significant primary findings were cited more often, an association that could not solely be explained by the journal of publication or the study design. The influence of the outcome of statistical testing on the decision to refer to a particular published finding is consistent with a
Acknowledgments
This study was supported by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft, FOR534, AN 365/2-1).
References (37)
- et al.
Citation bias of hepato-biliary randomized clinical trials
J Clin Epidemiol
(2002) - et al.
A survey identified publication bias in the secondary literature
J Clin Epidemiol
(2006) - et al.
Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials
J Clin Epidemiol
(2005) Genetic associations: false or true?
Trends Mol Med
(2003)- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Publication and related biases
Health Technol Assess
(2000) - et al.
The citation bias: fad and fashion in the judgment and decision literature
Am Psychol
(1984) - et al.
Comprehensiveness and bias in reporting clinical trials. Study of reviews of pneumococcal vaccine effectiveness
Can Fam Physician
(1995) - et al.
Citation bias in medical journals
Scientometrics
(1999) - et al.
Publication decision revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa
Am Stat
(1995)
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles
JAMA
Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome
BMJ
Reference bias in reports of drug trials
BMJ
Statistics in medical journals: developments in the 1980s
Stat Med
Bias against negative studies in newspaper reports of medical research
JAMA
Bias in clinical intervention research
Am J Epidemiol
The effect of statistical methods and study reporting characteristics on the number of citations: a study of four general psychiatric journals
Scientometrics
Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials
JAMA
Cited by (65)
Citation bias in the literature on dietary trans fatty acids and serum cholesterol
2019, Journal of Clinical EpidemiologyScientific citations favor positive results: a systematic review and meta-analysis
2017, Journal of Clinical EpidemiologyCitation Excerpt :The direction of citation bias was fairly consistent: with some exceptions [6,7], most studies reported that positive articles were cited more often than negative articles (Table 1). Most of the studies are biomedical [7–42], but some also concern the social [43–49] and natural sciences [6,50,51], or a combination of these [52]. The biomedical studies ranged from highly specific fields—such as the relationship between job strain and cardiovascular disease [16], or the treatment of chronic nonspecific lower-back pain [8]—to broader categories such as cardiovascular medicine [10].
Brief alcohol intervention trials conducted by higher prestige authors and published in higher impact factor journals are cited more frequently
2016, Journal of Clinical EpidemiologyCitation Excerpt :The prevailing hypothesis is that methodologically sound publications will be cited more frequently than lower quality publications. This hypothesis is unsubstantiated, however, given that citation rates have been linked to other factors unassociated with study quality, including the direction and statistical significance of effects [1−5]. Two other factors that may contribute to citation rates above and beyond study quality, however, are author prestige and journal impact factors.
Biased and inadequate citation of prior research in reports of cardiovascular trials is a continuing source of waste in research
2016, Journal of Clinical EpidemiologyCitation Excerpt :Trials seen as supportive, those with results consistent with subsequent trials, were 45% more likely to be cited by those subsequent trials than nonsupportive trials (95% CI, 30–61%; P < 0.001). Although the most recent systematic review to date found no significant association between support and citation [9], this positive association is consistent with previous studies [6,19]. Omission of previous studies threatens the validity of the research question and dismisses results of completed trials, the conduct of which incurred risk on the part of trial participants.
Predictors of citation rate
2016, Annals of Epidemiology