Abstract
Randomised controlled trials (RCTs) evaluating the management of acute exacerbations of chronic obstructive pulmonary disease (COPD) report heterogeneous outcome measures, thus rendering their results incomparable, complicating their translation into clinical practice. As a first step in the development of a core outcome set that will aim to homogenise outcome measures in future RCTs, we assessed the outcomes reported in recent relevant RCTs and systematic reviews.
We conducted a methodological systematic review (https://www.crd.york.ac.uk/prospero/ registration number CRD42016052437) of RCTs and systematic reviews on COPD exacerbation management indexed on Medline and PubMed during the last decade. We evaluated their methodology, specifically focusing on the reported outcome measures.
Based on 123 RCTs and 38 systematic reviews, we found significant variability in the outcomes reported and in their definition. Mortality, which was assessed in 82% of the included trials, was the most frequently assessed outcome, followed by the rate of treatment success or failure (63%), adverse events (59%), health status, symptoms and quality of life (59%), lung function (47%), and duration of exacerbations (42%).
The significant heterogeneity in the selection and definition of outcome measures in RCTs and systematic reviews limits the interpretability and comparability of their results, and warrants the development of a core outcome set for COPD exacerbations management.
Abstract
Comparability of RCTs evaluating the management of COPD exacerbations is limited by heterogeneity in their design. Standardisation of outcome measures would help researchers to compare, contrast and synthesise them. http://bit.ly/2v5wwH4
Introduction
Chronic obstructive pulmonary disease (COPD), affecting >10% of people aged >40 years, is a leading cause of death and disability globally [1]. Acute exacerbations punctuate the natural history of COPD, representing a major determinant of disease morbidity, mortality and progression, and health care utilisation and costs [1, 2]. Their management is currently based on treatments that are only partially effective and almost unchanged for over a decade [1, 3]. Conversely, our understanding of the exacerbations pathogenesis and underlying mechanisms is growing rapidly [3, 4]; therefore, we anticipate that novel, targeted treatments will be introduced and trialled in the near future.
An important challenge that researchers conducting randomised controlled trials (RCTs) of the management of acute exacerbations of chronic obstructive pulmonary disease (AECOPD) face is the selection of outcome measures. The definition and consistent use of relevant and comparable outcomes, including patient important outcomes, is currently lacking. Characteristically, there is no universal measure for treatment success or failure in AECOPD, and simple to measure outcomes such as symptom burden and duration are not easily agreed (figure 1). In the absence of consensus, the outcomes reported in published RCTs vary significantly. This complicates comparing, synthesising and interpreting trial results, leading several recent Cochrane reviews on AECOPD to report limited confidence on the body of evidence due to this variability [5, 6].
Hypothetical exacerbations with different outcomes. Important outcomes include the duration of symptoms from presentation, treatment failure rate, mortality and adverse effects of the interventions. Standardisation of their measurement is required.
The development of a core set of outcomes, representing the minimum that should be measured and reported in all relevant clinical trials, could remedy this issue. Following methodology developed by the Core Outcome Measures in Effectiveness Trials Initiative [7], we conducted a methodological systematic review to evaluate the outcomes reported in trials on AECOPD, as a first step in the development of a core outcome set. In addition, we specifically assessed the extent to which these trials report patient-important outcomes.
Methods
This systematic review is based on a prospectively registered protocol (PROSPERO register identifier CRD42016052437) [8].
Study selection and data extraction
Eligible studies were RCTs and systematic reviews of RCTs studying interventions aimed to treat AECOPD. Medline and PubMed were searched in October 2018 for eligible studies published between 2006 and 2018. The detailed search strategy is available in the supplementary material.
Search results were screened for eligibility by two reviewers independently (A.G. Mathioudakis, J. Janner or M. Moberg) in two steps, which included an initial screening of the titles and abstracts, followed by detailed assessment of the full text reports of potentially relevant articles. The selection process is described in a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart in the supplementary material. Details of the included studies and the studies that were deemed potentially eligible during the screening of titles and abstracts but were excluded at a later stage, as well as the reasons for their exclusion, are also presented in the supplementary material. Relevant study characteristics and all the outcomes evaluated in each included study were independently extracted in a standardised form by two reviewers (A.G. Mathioudakis, J. Janner or M. Moberg). When it was deemed appropriate, further data were sought from published trial protocols. Disagreement in each of these stages was resolved by consensus among the three reviewers. Results are presented narratively.
Outcome classification
All outcomes reported in the included studies were classified into the following three categories by the three reviewers, who were working independently and in a blinded fashion. Disagreement was resolved by consensus among all authors.
Patient-important outcomes
Patient-important outcomes reflect how patients feel, function or survive [9, 10]. These include mortality, morbidity, measures of health status and quality of life.
Surrogate outcomes
Surrogate outcomes are early outcomes that may indicate disease progression and increased risk of patient-important outcomes, such as oxygen saturation, which could predict treatment success or duration of hospitalisation in patients requiring supplemental oxygen [11, 12].
Physiological and laboratory outcomes
Physiological and laboratory outcomes assess the response of physiological or laboratory measures, without direct, tangible effects on patients [11].
Results
Description of the included studies
Details of the search results and study selection process are provided in a PRISMA flowchart in the supplementary material. Briefly, our search strategy yielded 1796 results, among which we selected 173 eligible manuscripts reporting on 38 systematic reviews and 123 RCTs. References of all included studies are available in the supplementary material.
Most of the included studies focused on the use of antibiotics (14 systematic reviews and 34 RCTs), corticosteroids (eight and 13, respectively), physiotherapy and rehabilitation (two and 18), oxygen therapy and ventilation (six and 15), bronchodilators (one and nine), self-management and telehealth (two and five), and complementary medicine (three and eight). Study populations of the included trials ranged between nine and 980 participants, and follow-up period between 45 min and 2 years. With regards to blinding, 45, 21 and 57 trials were double-blind, single-blind and open label, respectively. Investigators were blinded to the intervention in 15 of the single-blind trials, with patients blinded in the remaining single-blind studies. 55 (45%) of the included trials were multicentre and 23 (19%) were sponsored by a pharmaceutical company.
A prospectively registered protocol was only available for 67 out of 123 (55%) RCTs and nine out of 38 (24%) systematic reviews. There was an upward trend for trials, as 60%, 54%, 92%, 86% and 100% of the trials published in 2014, 2015, 2016, 2017 and 2018, respectively, reported a prospective protocol. Unfortunately, systematic reviews did not follow a similar pattern and almost exclusively, systematic reviews published by the Cochrane Collaboration reported a prospective protocol.
Primary and secondary outcomes were clearly and separately defined in 84 (68%) of the trials and 19 (50%) of the included systematic reviews.
Outcome measures of RCTs
Patient-important outcomes
Mortality
Mortality at longest follow-up was evaluated in 101 of included RCTs (82%). It was the primary outcome of six RCTs (5%).
Clinical treatment success or failure
Treatment success or failure rates were evaluated in 77 trials (63%). More specifically, 21 studies (17%) reported data on both treatment success and failure rates, while 27 (22%) and 29 (24%) studies only reported on treatment failure or treatment success, respectively. Treatment success or failure were evaluated at variable time-points, between 3 and 90 days from presentation (table 1).
Treatment success or failure evaluation time-points
Treatment failure was defined in the majority of studies (n=24) as a composite outcome including several of the following components: death, intensive care unit admission, requirement of additional treatment (usually systemic corticosteroids or antibiotics for respiratory reasons), need for hospitalisation or re-admission, significant symptom deterioration or, infrequently, diagnosis of a new AECOPD after complete symptom resolution. 14 studies defined treatment failure as lack of complete symptom resolution or lack of improvement in the symptoms at a specific time point, and 10 studies as the need for treatment intensification.
Most trials (n=37) defined treatment success as complete resolution or significant improvement of the clinical symptoms associated with the AECOPD. Some evaluated both “cure”, defined as complete resolution of the signs and symptoms of an exacerbation, and “improvement”, usually defined as complete resolution of the fever, with incomplete resolution of other signs and symptoms, and without clinical need for additional treatments. Four trials considered treatment success as discharge from hospital, two as discharge from the intensive care unit and seven as successful withdrawal of ventilation or supplementary oxygen administration (depending on the intervention and study setting).
Adverse effects
73 RCTs (59%) reported on the adverse effects of the study interventions. Most evaluated all adverse effects, severe adverse effects and mortality, or presented the most frequent adverse effects. Some trials focused on specific known side-effects of the study drugs or interventions, and others also evaluated the impact of study drugs on vital signs, electrolytes, acid–base balance or specific biochemical tests.
Health status, quality of life and symptoms
Changes in symptoms, quality of life and/or health status of the participants was evaluated in 73 out of 123 trials (59%). 41 (33%) assessed symptom progression using simple symptom scores, such as visual analogue scales or Likert scales. Breathlessness was the most frequently evaluated symptom, followed by phlegm volume. Other symptoms included cough, phlegm colour and fatigue.
34 other studies (28%) utilised more comprehensive health status and quality of life questionnaires, mostly the COPD Assessment Test and the Clinical COPD Questionnaire. Other questionnaires included St George's Respiratory Questionnaire, Chronic Respiratory Questionnaire, Exacerbation of Chronic Pulmonary Disease Tool (EXACT), EuroQol-5D, Short Form 36 Health Survey and the BODE (body mass index, airflow obstruction, dyspnoea, exercise capacity) index.
Two of the trials evaluating health status did not report adequate details on the methodology utilised.
Duration of the exacerbation
Duration of the index exacerbation was reported in 42 RCTs (34%).
1) Time to treatment success: all three studies (3%) reporting time to treatment success were based on symptom questionnaires. Two studies used the same, well-defined criteria, based on daily symptom diaries evaluating the Anthonisen symptoms. The first day of the AECOPD was defined as the day the patient sought medical advice. The last day was defined as the first of three consecutive days when the patient had returned to their normal health state or the first of seven consecutive days in which the patient only reported a minor increase in symptoms compared to baseline, without fever or altered sputum colour.
2) Length of hospital or intensive care stay: length of hospital or intensive care stay was reported in 39 of the included RCTs (32%). 33 studies reported on the length of hospitalisation, 10 on the length of stay in the intensive care unit and 10 on the length of ventilation.
Re-exacerbation, re-hospitalisation and healthcare utilisation
Time to next exacerbation or hospitalisation, re-exacerbation, re-hospitalisation or number of AECOPD during follow up were reported in 33 trials (27%). More specifically, 28 (23%) reported on further exacerbations and 14 (11%) on further hospitalisations.
14 studies (11%) assessed healthcare utilisation during follow-up. From this outcome we excluded those studies that only evaluated length of hospital or intensive care stay or ventilation during the initial presentation, as this was described separately. These 14 studies evaluated hospital admissions (number and duration), emergency department visits, emergency outpatient visits, telephone calls with the physicians and/or consultations with primary care physicians.
Exercise capacity
Only 14 studies (11%) reported on the rate of improvement in exercise capacity. Most (n=11) utilised the 6-min walking test, while two used the 3-min walking test, one the incremental shuttle walk test and one used 2-min step in place.
Anxiety and depression
Change in the levels of anxiety and depression was evaluated in six studies (5%). Four studies utilised the Hospital Anxiety and Depression Scale, one used a visual analogue scale and one used the Geriatric Depression Scale.
Surrogate outcomes
Arterial blood gases and oxygen saturation
Arterial blood gases and oxygen saturation were mostly reported in trials evaluating oxygen therapy, ventilation or chest physiotherapy. More specifically, 40 RCTs (33%) reported on arterial blood gases (pH, oxygen tension, carbon dioxide tension and oxygen saturation measured by pulse oximetry) and five additional studies reported on oxygen saturation.
Microbiological response
Microbiological response (which was assessed based on serial sputum cultures) at various time-points was assessed in 16 RCTs (13%). All studies used similar definitions for success or failure. For patients with a positive sputum culture at presentation, bacteriological eradication was defined as the absence of the original causative organism in subsequent sputum cultures; presumed bacteriological eradication was defined as the absence of appropriate culture material, because the patient had clinically improved and was unable to produce phlegm. The presence of the original causative organism in repeat sputum cultures was defined as persistence and the isolation of a new organism, as superinfection. Patients with persistent clinical symptoms (clinical treatment failure) who were unable to produce phlegm were categorised as having presumed bacteriological persistence. Bacteriological relapse and eradication with re-infection were also defined in some of the RCTs.
Medication use
18 RCTs (145%) reported on the use of medications during follow-up, including use of reliever therapy, duration of antibiotic and/or systemic steroid courses, or the administration of additional courses of the these.
Physiological and laboratory outcomes
Pulmonary function
Rate of pulmonary function improvement over time was the most frequently evaluated physiological outcome (58 out of 123 RCTs, 47%). Forced expiratory volume in 1 s was the most frequently assessed parameter, followed by forced vital capacity and peak expiratory flow rate.
Biomarkers
32 RCTs reported on the impact of the interventions on various biomarkers. C-reactive protein was the most frequently reported marker (18 RCTs, 15%). Other biomarkers included white blood cell count, several interleukins, tumour necrosis factors, interferons, leukotrienes, procalcitonin, α1-antitrypsin and other biomarkers specific to the interventions evaluated.
Outcomes of systematic reviews
Outcomes reported in systematic reviews were similar to those reported in primary studies. Details are available in table 2.
Frequency that different outcome measures were reported in the 123 randomised controlled trials (RCTs) and 38 systematic reviews (SRs) included in this methodological review
Discussion
In this methodological study, evaluating 38 systematic reviews and 123 RCTs conducted during the last decade, we identified significant heterogeneity in the selection and definition of outcome measures, which might render their findings incomparable, and hinder the production of evidence-based syntheses and recommendations.
We included both RCTs and systematic reviews of RCTs as the latter are based on methodology independently developed by the systematic reviewers, who select primary and secondary outcome measures that are pertinent to patients and clinicians, and not necessarily the same as the primary studies they evaluate.
Only half of the identified trials and <20% of the systematic reviews were based on a prospectively registered protocol. Prospective publication of an RCT or systematic review protocol could limit the phenomenon of selective reporting of trial results, as well as the impact of publication bias on systematic reviews and guidelines. For this reason, International Committee of Medical Journal Editors introduced a requirement for a prospective protocol for all clinical trials published in participating journals from 2004 [13]. Since then, the proportion of RCTs that were based on a prospectively registered protocol has been consistently increasing. Unfortunately, a similar requirement for the publication of systematic reviews is still lacking, but the Cochrane Collaboration has been advocating for it.
Another significant methodological limitation of the included studies was the lack of a clear separation between primary (powered) and secondary outcomes. Specifically, 32% of the included RCTs and 57% of the systematic reviews did not clearly distinguish primary and secondary outcomes, complicating the interpretation of their results.
RCTs and systematic reviews in AECOPD evaluate a variety of outcomes, including several patient-important outcomes. Frequently reported outcomes include mortality, clinical success or failure, adverse effects, improvement in health status, symptoms and quality of life, spirometry, duration of exacerbation, and re-exacerbation (table 3). Only mortality and adverse effects were assessed and reported in a standardised manner that allows comparability.
Advantages and disadvantages of the main outcome measures
Clinical success or failure represent one of the most important outcomes of a COPD exacerbation. Unfortunately, there is significant variability in the definitions of this outcome across different trials. We consider most appropriate the simplest definition used. This classifies complete resolution of all signs and symptoms as cure, incomplete resolution of signs and symptoms with resolution of fever as improvement, and the persistence or deterioration of symptoms as treatment failure. The selection of timepoints to evaluate this outcome (and other exacerbation outcomes) is of utmost importance, given the dynamic nature of exacerbations. Previous studies estimated the length of symptom duration of AECOPD and found median durations of 11–13 days from symptom onset [14] or 7–10 days after seeking medical advice [11, 12]. While logically, an exacerbation starts with symptom onset, identification of that timepoint is often impractical and it is simpler to define as onset the day when patient seeks medical advice. This timepoint has the additional advantage that it usually coincides with the onset of the intervention. Therefore, evaluation of the treatment failure rate between 1–2 weeks after presentation could provide meaningful results.
Many studies evaluated improvement of health status, quality of life or symptoms over time. The numerous distinct measures of these outcomes and use of diverse timepoints significantly limit comparability. Time from presentation to treatment success (defined as the reversion of symptoms back to the patient's baseline level, prior to the AECOPD) has been assessed by a surprisingly limited number of studies, but it is a simple to capture and comparable outcome. It would also be very sensitive, as it could reveal small effects that could be missed by more crude measures such as treatment success or failure rate. While the three identified trials used the Anthonisen criteria to confirm symptom resolution, other, noninterventional, studies have used other symptom scores, such as EXACT [15] or the London COPD Cohort diary card scores [16], which are more comprehensive.
Time to treatment success is a more accurate outcome than the length of hospitalisation, which can be affected by comorbidities, social circumstances, the clinician's perception of patients' symptoms or even the hospital's structure, availability of hospital beds and delays in administrative processes [17–19]. It is also not limited to hospitalised AECOPD, in contrast to the length of hospitalisation.
Some treatments of AECOPD not only aim to treat the acute episode but also to delay further events. Time to next exacerbation or frequency of exacerbations during follow-up represent simple measures of this outcome that have been consistently evaluated.
Our systematic review has several strengths. It is the first study evaluating the methodology and outcome measures of RCTs and systematic reviews comparing interventions aimed to treat AECOPD. Our findings are based on all RCTs (n=123) and systematic reviews (n=38) published during the last decade. All stages of our systematic review, which was based on a prospectively registered protocol, were conducted by two authors independently and the strong reproducibility of the investigators' judgements strengthen the validity of our findings.
A limitation of our review is that we only searched Medline and PubMed for published RCTs and systematic reviews; we did not screen other publication databases or registries. However, we included a very large number of high-quality studies, and we are confident that our findings are complete and representative of the available work. Our review is also limited by inherent limitations of the included studies. Most importantly, study protocols were not prospectively registered or available online for half of the included trials and most systematic reviews. Therefore, we extracted our data from the published manuscripts and we cannot exclude the possibility of selective outcome reporting by some studies. Finally, we do not report intervention-specific, infrequently reported outcomes, such as muscle strength (outcome of some physiotherapy-related interventions) or adherence to the intervention, as they were not considered relevant to the aims of this report.
As stated in our introduction, AECOPD are frequent events, yet their treatment has seen little change over many years and the intervention research does not in any way match the size of the problem. Other medical emergencies have seen vast progress [20, 21] and the respiratory community needs to address how to catch up. Defining a core outcome set is a first step [10]. Subsequently, collaborative efforts are needed and societies such as the European Respiratory Society ought to take a leading role here.
In conclusion, our study revealed significant methodological limitations and heterogeneity in the selection and definition of outcomes in trials on AECOPD that hinder the repeatability and comparability of their findings. This could be remedied by the development of a core outcomes set for trials on AECOPD management.
Supplementary material
Supplementary Material
Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.
Supplementary material 00072-2019.supp
Footnotes
This article has supplementary material available from openres.ersjournals.com
This study is registered at https://www.crd.york.ac.uk/prospero/ with identifier number CRD42016052437
Author contributions: A.G. Mathioudakis and J. Vestbo contributed to the conception and design of the study. A.G. Mathioudakis, M. Moberg and J. Janner contributed to data collection. All authors contributed to interpretation of the data. A.G. Mathioudakis, P. Alonso-Coello and J. Vestbo provided methodological input. A.G. Mathioudakis drafted the manuscript. All authors contributed to critical revision of the paper for intellectual content.
Conflict of interest: A.G. Mathioudakis reports consultancy fees for COPD research programmes from Boehringer Ingelheim and GlaxoSmithKline, and an investigator-initiated grant for an RCT of COPD exacerbations from Boehringer Ingelheim, outside the submitted work.
Conflict of interest: M. Moberg has nothing to disclose.
Conflict of interest: J. Janner has nothing to disclose.
Conflict of interest: P. Alonso-Coello has nothing to disclose.
Conflict of interest: J. Vestbo reports consultancy fees for COPD phase 2 and 3 programmes, and payment for lectures including service in speaker bureaus from Chiesi Pharmaceuticals, Boehringer-Ingelheim, Novartis and AstraZeneca, outside the submitted work.
Support statement: A.G. Mathioudakis is funded by an Academic Clinical Fellowship in Respiratory Medicine from the National Institute for Health Research (NIHR), UK (ACF-2016-06-008), and A.G. Mathioudakis and J. Vestbo are supported by the NIHR Manchester Biomedical Research Centre. The views expressed are those of the authors, and not necessarily those of the National Health Service, the NIHR or the Department of Health. Funding information for this article has been deposited with the Crossref Funder Registry.
- Received March 17, 2019.
- Accepted April 1, 2019.
- Copyright ©ERS 2019
This article is open access and distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0.
References
