Abstract
Guidelines for chronic obstructive pulmonary disease (COPD) management are based largely on results from double-blind randomised controlled trials (RCTs) of efficacy. These trials have high internal validity and test whether a drug is efficacious, but they are conducted in highly selected populations that may differ significantly from patients with COPD seen in routine practice.
We compared the baseline characteristics, healthcare use and outcomes between the Salford Lung Study (SLS), an open-label effectiveness RCT, with six recent large-scale efficacy RCTs. We also calculated the proportion of SLS patients who would have been eligible for inclusion in an efficacy RCT by applying the inclusion criteria used in efficacy trials of combination treatments.
SLS patients were older, included more females and more current smokers, had more comorbidities (including asthma), and had more often experienced exacerbations prior to inclusion. In the SLS, rates of moderate or severe exacerbations, incidence of overall serious adverse events (SAEs), and SAEs of pneumonia were more frequent. A maximum of 30% of patients enrolled in the SLS would have been eligible for a phase IIIa regulatory exacerbation study.
Patients in large COPD efficacy RCTs have limited representativeness compared with an effectiveness trial. This should be considered when interpreting efficacy RCT outcomes and their inclusion into guidelines.
Abstract
Efficacy trials are less representative than effectiveness trials and both are required when evaluating treatments http://ow.ly/plwd30he4a2
Introduction
Chronic obstructive pulmonary disease (COPD) is a major public health concern worldwide most commonly managed in primary care, and is proving to be a major burden to patients and healthcare systems [1, 2]. Clinical guidelines for COPD management are based largely on results from double-blind randomised controlled trials (RCTs) of efficacy, which are accorded the strongest category of evidence [1, 3]. Such “efficacy RCTs” are usually conducted for the regulatory approval of new therapies to answer the question of whether the new therapy actually works as intended [4, 5]. They focus on maximising internal validity and are designed to determine the effect of a medicine in ideal conditions.
However, it is questionable whether conventional efficacy RCTs are relevant to routine clinical practice and thus sufficient as the only source for registration. These trials are conducted in highly selected and homogeneous patient populations to avoid confounding factors such as comorbidities or diagnostic uncertainty. Inclusion criteria include strict spirometric criteria and smoking history, and patients with significant comorbidities (e.g. cardiovascular disease, or a current diagnosis or history of asthma, allergic rhinitis or atopy) are usually excluded [6–8]. Patients in RCTs are usually recruited in research clinics, are often healthier than patients in the general population [9] and frequently participate in multiple trials. Evidence suggests that this group of “persistent participators” may differ significantly from patients with COPD seen in routine primary care, typically ranging from <10% to 30% of COPD patients seen in routine care [10–14].
In addition, efficacy RCTs involve intensive monitoring of the patients with frequent visits and procedures. Inhaler technique is rigorously checked, adherence is actively monitored and encouraged, and treatments are provided directly to patients by the investigator [15, 16]. This highly controlled environment of an efficacy RCT does not reflect everyday practice, where patients with COPD are reviewed less frequently, inhaler technique is rarely checked and adherence to inhaled medicines is lower [17]. Therefore, study findings may be poorly generalisable to patients with COPD managed in everyday clinical practice despite a high internal validity.
In view of the aforementioned considerations, healthcare decision makers and providers are calling for data from more representative patients treated in a setting of routine care [15]. This can be achieved by effectiveness trials that are more able to fully explore the true benefit/risk ratio and value of a medicine, allowing clinicians to make more informed management decisions with patients [15]. In brief, whereas efficacy studies are carefully carried out experimental trials that test if a drug can work, an effectiveness study is a more simplistic trial testing if the drug does work. As a consequence, clinical guidelines may also be enhanced by integrating effectiveness as well as efficacy data [18]. Although improving external validity, effectiveness trials come at a price; often patients may have a less stringent diagnosis, are less well characterised and have poorer quality spirometry.
The Salford Lung Study (SLS) in COPD was designed to be conducted in a setting of everyday clinical practice to meet the need for effectiveness data to complement existing evidence from standard efficacy RCTs [19, 20]. It was the world's first large-scale prospective, randomised study to evaluate the clinical effectiveness and safety of initiating a pre-licensed COPD medicine (fluticasone furoate/vilanterol (FF/VI); once daily in a novel dry powder inhaler) compared with continuing usual care in everyday clinical practice [20].
The aims of this article are 1) to compare the generalisability of the SLS by describing the similarities and differences of the patient characteristics and study conduct between the SLS and large COPD exacerbations efficacy trials carried out in the last 10 years, 2) to demonstrate the impact of patient selection and study conduct on study outcomes, and 3) to determine the proportion of SLS COPD patients that would have been eligible for inclusion in FF/VI phase IIIa regulatory exacerbation studies [6].
Methods
SLS COPD study design
The SLS was a 12-month open-label effectiveness RCT conducted in UK primary care that evaluated the effectiveness and safety of initiating FF/VI 100/25 µg once daily compared with continuing COPD maintenance therapy (usual care) (figure 1) [21].
Patients
Broad inclusion and minimal exclusion criteria were employed. Patients aged ≥40 years, who had been diagnosed with COPD by their primary care physicians, with a history of exacerbations in the last 3 years and taking a regular maintenance inhaled therapy (inhaled corticosteroid (ICS) and/or long-acting muscarinic antagonist (LAMA) and/or long-acting β-agonist (LABA)), were randomised 1:1 to initiate FF/VI or to continue their usual care [20, 21].
Recruitment and monitoring
Patients were recruited on general practice. Patients were only seen face-to-face by the study team at baseline and at exit from the trial at 12 months. Study end-points and safety monitoring was carried out with an integrated primary and secondary care electronic health record (EHR). Patients collected their prescriptions from their usual pharmacy. General practitioners (GPs) were able to adjust medication throughout the study to allow for optimal treatment of COPD, as would be normal clinical practice. Patients were allowed to switch from FF/VI to usual care.
End-points
The primary end-point was the mean annual rate of moderate or severe exacerbations (symptoms that led to treatment with antibiotic agents or systemic glucocorticoids (or both), to hospital admission, or to scheduled or unscheduled hospital visits), with symptoms assessed by the COPD Assessment Test (CAT) and healthcare resource utilisation as secondary end-points. Safety outcomes included SAEs of pneumonia (defined as an adverse event of special interest, i.e. an adverse event that was considered to be possibly related to ICS and LABA), and other SAEs including fatal SAEs and adverse drug reactions (ADRs) [20, 21].
Selection of large efficacy trials
Based on a PubMed search using the search terms “COPD”, “randomised clinical trial” and “exacerbations” (resulting in 994 publications), we selected a range of large double-blind efficacy RCTs using the following criteria: 1) efficacy RCTs recruiting COPD outpatients aged ≥40 years, 2) enrolled at least 1000 patients with a diagnosis of COPD, 3) conducted in the last 10 years (2007–2016), 4) assessing inhaled ICS/LABA and/or LAMA/LABA at licensed dose, 5) COPD exacerbations as the primary end-point, and 6) duration of treatment of 1 year.
Methods of analysis
We compared baseline characteristics and healthcare use, and efficacy/effectiveness and safety outcomes, between the SLS and the selected large efficacy RCTs. Baseline characteristics included age, sex, smoking status, post-bronchodilator forced expiratory volume in 1 s (FEV1) % pred, CAT score, exacerbation history and comorbidities. Where CAT score values from efficacy RCTs were not available, the values were derived from the St George's Respiratory Questionnaire score [22]. Outcomes included the rate of moderate or severe COPD exacerbations, the rate of hospitalised (severe) COPD exacerbations, overall SAEs, fatal events, SAEs of pneumonia and ADRs. In addition, rates of patient withdrawal were also compared.
In a separate post hoc analysis, we evaluated the proportion of patients who would have been eligible for the FF/VI phase IIIa regulatory studies with exacerbation as a primary outcome, using a stepwise approach [6]. The criteria we examined in sequence included baseline spirometry, available post-bronchodilator FEV1/forced vital capacity (FVC) <0.70, post-bronchodilator FEV1 ≤70%, smoking status and number of pack-years, history of current asthma, and history of at least one moderate or severe exacerbation in the prior 12 months.
Results
Selection of efficacy RCTs
We assessed six large efficacy RCTs conducted in patients with COPD [6–8, 23–25]. The trials were carried out between 2007 and 2016, included patients aged ≥40 years with a smoking history of at least 10 pack-years, with a post-bronchodilator spirometric FEV1/FVC <0.70 combined with post-bronchodilator FEV1 ≤70%. The exclusion criteria included long-term oxygen therapy, acute phase of pulmonary rehabilitation, patients with comorbid asthma or other pulmonary disease, or other significant conditions (e.g. carcinoma, heart disease or diabetes). Table 1 shows the key study design characteristics of the six selected large RCTs.
Patients characteristics and healthcare use at baseline
Key patient characteristics in the ITT populations of the SLS and efficacy RCTs are summarised in table 2. SLS patients were older (mean age 67 versus 63–65 years in efficacy RCTs), included more females (51% versus 24–43%), a high proportion of current smokers (46% versus 36–48%) and had a high rate of comorbidities (77%), including patients with asthma (22%). Not all patients in the SLS underwent spirometry (21% had not) and 5% of patients had never smoked. SLS patients had more exacerbations prior to inclusion (mean 2.01 moderate or severe exacerbations in the prior 12 months) compared with those included in efficacy RCTs (mean 1.2–1.7).
Study outcomes in the SLS versus large efficacy trials
Key findings are summarised in figure 2a and b, and are detailed in table 3. In the SLS, the rate of patient withdrawals from treatment was very low (7%) compared with efficacy RCTs (11–30%) (table 1). The mean annual rate of moderate or severe exacerbations was substantially higher in both arms in the SLS (1.74 in the FF/VI arm and 1.90 in the usual care arm) compared with the efficacy RCTs, where the rate ranged from 0.45 to 1.19 (table 3 and figure 2a). However, there was no difference in the rate of severe exacerbations when comparing the SLS and the large efficacy trials (table 3). The incidence of overall SAEs (including fatal SAEs) and pneumonia SAEs was higher in the SLS (both arms) at 27–29% and 6–7%, respectively, compared with efficacy RCTs (13–24% and 1–3.2% in ICS-containing arms, respectively) (table 3 and figure 2b). The incidence of ADRs in the FF/VI arm from the SLS COPD was similar to the incidence observed in the FF/VI arm in the FF/VI phase III exacerbation studies (15% versus 17%), The incidence of ADRs in the usual care arm of the SLS was lower than that in the FF/VI arm in both the SLS and FF/VI phase IIIa regulatory studies [6]. The incidence of fatal SAEs was low, but varied across the FF/VI and usual care arms of the SLS and the arms of efficacy RCTs (3% versus 2% and 1–3%, respectively). The patterns of SAEs and ADRs in the SLS COPD were as expected for this medicine class.
Proportion of SLS patients eligible for inclusion in FF/VI phase IIIa regulatory exacerbation studies
Almost half (5658 out of 11 720 (48%)) of COPD patients registered at the GP practices in Salford and South Manchester area taking part in the SLS were eligible for the SLS, based on a retrospective analysis of a database study carried out for the feasibility of the SLS [27], and of those eligible patients around half entered the study.
Of the 2802 patients enrolled in the SLS, 841 (30%) would have been eligible for the phase III FF/VI studies (figure 3a) [6]. Most patients were excluded by spirometry (49%), especially because of missing spirometry data at baseline (figure 3b). Of all the COPD patients in Salford eligible for inclusion in the SLS, ∼15% (841 out of 5658 patients) would have been eligible for entry to an efficacy RCT.
Discussion
In the SLS COPD study carried out in routine clinical practice, patients had a high burden of disease, more symptoms, more frequent exacerbations, more comorbidities and more SAEs (including SAEs of pneumonia) compared with patients in large COPD efficacy RCTs conducted for registration purposes.
Efficacy RCTs exclude patients because of age, disease severity and presence of comorbidities [10, 11, 13, 14, 28]. Data from an analysis of seven primary care databases in Europe has previously been compared with six large COPD efficacy RCTs [28]. In that analysis, as with the SLS, patients with COPD followed in primary care tended to be older, more commonly female and to have moderate airflow obstruction, but unlike the SLS had lower exacerbation rates. This probably relates to the SLS selecting patients with at least one exacerbation in the previous 3 years (81% had an exacerbation within the last year). These data combined with the SLS show definitively that COPD patients enrolled in efficacy RCTs are unrepresentative of those seen in primary care.
Efficacy RCTs also have other subtle enrolment criteria which make them less relevant to the population to whom the drug will be marketed. For example, patients may be excluded for lack of compliance during run-in or poor inhaler technique. This could eliminate any benefits in routine practice from an easier-to-use inhaler. In addition, the tight supervision in an RCT with repeated training on inhaler technique and encouragement to adherence just does not take place in routine care. In contrast, in the SLS, apart from the baseline and 12-month visits, there were no planned face-to-face study visits with the study team. The very low dropout rate in the SLS compared with efficacy RCTs probably reflects the “passive” nature of the SLS, with all routine care being carried out by the patient's GP, and that subjects could change from FF/VI back to usual care while remaining in the study. The dropout was up to 4 times higher in the efficacy trials compared with the SLS. With higher dropout rates, there is always a concern about the relevance of trial outcomes for those who did drop out. In addition, the data from the healthy survivor population remaining in an efficacy RCT may be even less relevant to everyday care. Effectiveness trials have other limitations. They often rely on assessments made as part of routine care, often lacking the rigour employed in an efficacy trial; for COPD this especially relates to the quality of spirometry. The study design also allows patients to change treatment, something that leads to exclusion from an efficacy trial. In the SLS, patients were allowed to switch from FF/VI to usual care (but not from usual care to FF/VI). Given the study findings, treatment switching is likely to dilute the study result as all analyses were done on an intention-to-treat basis, but more studies are needed on this aspect of effectiveness trials. An SLS supportive analysis was conducted, but not yet published, to assess the impact of switching on effectiveness outcomes and no impact was found. Finally, effectiveness trials are often carried out in a specific geographical location, as was the SLS. The impact of this on study outcomes has not been examined as, unfortunately, very few effectiveness trials in COPD exist. We are of course also limited in having only one effectiveness trial to compare with the larger number of published efficacy trials and subsequent effectiveness trials could differ from the SLS.
The diagnosis of COPD was based on GP standard clinical practice, which did not always incorporate spirometric confirmation; indeed, 21% of patients in the SLS did not have spirometry. COPD efficacy RCTs usually exclude never-smokers, yet in the SLS, 5% of subjects reported that they had never smoked. In the SLS, the 22% of patients with a concurrent diagnosis of asthma would have been excluded in a registration RCT, even though in routine practice many patients will commonly be labelled as both asthma and COPD [1, 29].
Only a small proportion of SLS patients (30%) would have been eligible for the FF/VI phase IIIa exacerbation studies due to the typical multiple inclusion and exclusion criteria, which is consistent with the findings reported in the literature [10, 11, 13, 14, 28]. This proportion is likely an overestimate, since many patients could have been excluded on the basis of poor inhaler technique or poor adherence.
There is a lack of data comparing safety outcomes from efficacy versus effectiveness RCTs. In the SLS, there were more exacerbations, and a higher incidence of SAEs and fatal SAEs, likely due to enrolment of COPD patients with severe comorbidities. The higher rates of ADRs (where causality is attributed to the investigational medicine) may be due to a number of factors, including the open-label nature of the study, the unlicensed and unfamiliar FF/VI in a novel inhaler, and the “passive” continuous EHR monitoring for SAEs compared with the periodic active patient SAE reporting at infrequent face-to-face follow-up. In the SLS, all SAEs were captured by the EHR and reviewed weekly by the safety team with alerts to doctors.
In conclusion, we have confirmed the clear limitations of large efficacy RCTs in relation to their transferability to usual care. The patient population is more limited and the research environment is substantially different. Efficacy RCTs are essential to show that novel treatments have efficacy and are safe during research and development programmes. However, they should not be transferred straight into routine care guidelines without careful consideration. The SLS is a clinical effectiveness study designed to maintain scientific rigour through a prospective design, randomisation and stratification. We have enhanced the external validity and transferability into routine practice by recruiting a broad range of COPD patients and using remote monitoring with an EHR, albeit at the cost of less stringent diagnostics and poorer spirometric assessments. To fully evaluate any new treatments, we will need to provide additional effectiveness data in large unselected populations in routine clinical practice.
Disclosures
Supplementary Material
N.D. Bakerly ERJ-01531-2017_Bakerly
I. Boucot ERJ-01531-2017_Boucot
S. Collier ERJ-01531-2017_Collier
J. Crawford ERJ-01531-2017_Crawford
E. Hilton ERJ-01531-2017_Hilton
D.A. Leather ERJ-01531-2017_Leather
J. Vestbo ERJ-01531-2017_Vestbo
A. Woodcock ERJ-01531-2017_Woodcock
Acknowledgements
The authors wish to acknowledge the support of Andy Nicholls and Jamie Rees for the review and quality check of the data presented in this manuscript, both of whom are employees of GlaxoSmithKline (London, UK).
Author contributions: A. Woodcock, J. Vestbo, D.A. Leather, J. Crawford, S. Collier and N.D. Bakerly contributed to the study design and drafting the study protocol. I. Boucot, J. Crawford, D.A. Leather and E. Hilton were involved in the data analysis and interpretation.
Footnotes
This study is registered at ClinicalTrials.gov with identifier number NCT01551758.
Support statement: This study was sponsored and funded by GlaxoSmithKline (study number: HZC115151). Funding information for this article has been deposited with the Open Funder Registry.
Conflict of interest: Disclosures can be found alongside this article at erj.ersjournals.com
- Received July 27, 2017.
- Accepted December 9, 2017.
- Copyright ©ERS 2018