Comparing traditional bibliometrics and Altmetric assessments of research impact in respiratory disease

To the Editor:

Respiratory diseases such as asthma, COPD and coronavirus disease 2019 (COVID-19) are a significant global health concern. A large volume of original research is generated, which has the most potential for impact when disseminated widely. Traditional bibliometrics such as citations and impact factor (IF) of the publishing journal do not capture online dissemination of information, including through social media. Thus, traditional bibliometrics may not provide a comprehensive assessment of the reach and impact of research. Moreover, it is known to take between 2 and 3 years from first publication for an article to reach its peak rate of citations, resulting in a delay in impact [1].

Altmetric.com is a web-based platform that quantifies and evaluates scholarly digital attention and impact beyond conventional citation-based metrics. The Altmetric Attention Score (AAS) is a numerical indicator generated by Altmetric.com reflecting the extent of online engagement with a particular research output. The database has over four million research sources [2]. It includes various measures of online mention and interaction, such as social media discussions, news coverage, blog posts and policy document references. The AAS offers researchers, institutions and publishers a comprehensive assessment of the visibility and influence of research output in the digital landscape.

We have investigated relationships between time from publication, traditional bibliometrics and the AAS in Asthma, COPD and COVID-19. We hypothesised that there would not be a correlation between AAS and traditional bibliometrics (number of citations and journal IF) because what is widely shared in the media is not necessarily impactful in science, and vice versa, and that this effect would be exaggerated for COVID-19 given the intense media attention around COVID-19 research.

We selected studies based on the keywords “Asthma”, “COPD” and “COVID-19” on 15 July 2023. We used the Altmetric database and AAS to determine total mentions, outputs with attention and total outputs tracked for each condition. We selected the top 100 papers for Asthma, COPD and COVID-19 based on the AAS. We reviewed the titles and discarded irrelevant publications, with further papers added to ensure we had the top 100 publications for each of the three diseases. For each paper, we calculated the time from publication, identified the journal IF for the year of publication and recorded the number of citations by cross-referencing the DOI to the Web of Science (WOS) database. We also selected the top 100 papers for Asthma, COPD and COVID-19 based on the number of citations in WOS, only selecting those with a publication date falling between 1 January 2012 and 31 December 2022 (as earlier papers were published before the widespread use of social media). For these papers, we also calculated the time from publication, identified the journal IF for the year of publication and recorded the AAS. Data analysis was conducted using SPSS version 29.0. Data were tested for normality using the Kolmogorov–Smirnov test. Nonparametric data are presented as median and interquartile range. Comparisons across groups used the Kruskal–Wallis test. Correlations between variables used Spearman rank. We considered p<0.05 to be statistically significant.

Table 1 presents the traditional bibliometrics and altmetrics for each disease. Despite the first publications only appearing in 2020, considering the 100 top papers by AAS, COVID-19 had already generated more total mentions than Asthma (17.9 million versus 414 858), both of which were higher than COPD (147 615). This was reflected in the total tracked outputs (244 236 versus 61 543 versus 18 439, respectively).

Despite a shorter time from publication, the top 100 COVID-19 papers had a greater number of citations and AAS, and were published in journals with higher IF than Asthma and COPD (table 1, all p<0.001).

Considering the top 100 papers by AAS, we found significant correlations for all diseases between time from publication and citations (Asthma ρ=0.40, p=0.001; COPD ρ=0.39, p=0.001; COVID-19 ρ=0.66, p=0.001) and IF and citations (Asthma ρ=0.27, p=0.008; COPD ρ=0.23, p=0.019; COVID-19 ρ=0.20, p=0.05). AAS did not correlate significantly with the traditional bibliometrics for any of the three diseases (data not shown).

For Asthma and COPD, considering the top 100 papers by citations, this pattern was reversed with no relationships between time from publication and citations, or IF and citations, but significant correlations between AAS with citations (Asthma ρ=0.36, p<0.001; COPD ρ=0.35, p<0.001) and AAS with IF (Asthma ρ=0.28, p=0.05; COPD ρ=0.48, p<0.001). Whilst there were also significant relationships between AAS and time from publication, this was positive in Asthma (ρ=0.52 p<0.001) but negative in COPD (ρ=−0.50, p<0.001). For COVID-19, there was a significant correlation between citations and time from publication (ρ=0.22, p=0.031) but not citations and IF, and a significant relationship between AAS and IF (ρ=0.57, p<0.001) but not AAS and time from publication or citations. There were 30 papers across the three disease areas that appeared in both top-100 lists. There was no correlation between their respective positions on the two lists (ρ=−0.14, p=0.45).

We found that COVID-19 research has generated more research impact than Asthma and COPD when considering both traditional and novel bibliometrics. The higher media interest around COVID-19 is perhaps understandable, with reporting of scientific developments in the mainstream media seen more frequently than for chronic airway diseases. Affecting daily life for all citizens, COVID-19 was newsworthy in a way that Asthma and COPD are not – despite the high global prevalence of both Asthma and COPD.

Relationships between traditional and novel measures of research impact were complex. Considering papers with the greatest AAS impact, there were no correlations between AAS and traditional bibliometrics for any of the three conditions, suggesting that the papers generating the most media interest are not initially rated most highly by the research community. However, citations did correlate with AAS for papers with the greatest impact as assessed by citations. We suggest that a complete understanding of research impact on respiratory disease, therefore, requires consideration of both new and traditional bibliometrics, which were seen to vary across the three diseases studied.

Our finding, in general, of few correlations between traditional bibliometrics and the AAS suggests that early online attention does not necessarily predict later academic impact. Variables contributing to the influence assessed by traditional bibliometrics may not necessarily coincide with those that influence AAS. Our results are consistent with previous myopia, dermatology, paediatric surgery and cardiology research [3–6]. AAS is not a measure of scientific quality. It may be influenced by factors such as authors’ and the journal's social media presence, or general interest in the topic, particularly if the subject is contentious or sensational as has been seen with COVID-19.

Medical research is conducted to have an impact, and an assessment of research impact is important across many domains, from individual career progression through to decisions about the distribution of research funding. Our demonstration that there are complex relationships between novel and traditional assessments of impact implies that a complete evaluation of impact requires consideration of the complementary insights gained by considering both assessment methods.

In conclusion, for articles generating the most attention as measured by AAS, there were no correlations between novel and traditional assessments of impact in Asthma, COPD and COVID-19 research, and a comprehensive review of research impact must consider both methods of assessment.