Abstract
Rationale Acquiring high-quality spirometry data in clinical trials is important, particularly when using FEV1 or FVC as primary endpoints. In addition to quantitative criteria, the ATS/ERS standards include subjective evaluation which introduces inter-rater variability and potential mistakes. We explored the value of AI-based software (ArtiQ.QC) to assess spirometry quality and compared it to traditional over-reading control.
Methods A random sample of 2000 sessions (8258 curves) was selected from Chiesi COPD and Asthma trials (N=1000 per disease). Acceptability using the 2005 ATS/ERS standards was determined by over-reader review and by ArtiQ.QC. Additionally, three respiratory physicians jointly reviewed a subset of curves (N=150).
Results The majority of curves (N=7267, 88%) were of good quality. The AI agreed with over-readers in 91% of cases, with 97% sensitivity and 93% positive predictive value. Performance was significantly better in the asthma group. In the revised subset, N=50 curves were repeated to assess intra-rater reliability, (Kappa: 0.83, 0.86 and 0.80). All reviewers agreed on 63% of 100 unique tests (Kappa=0.5). When reviewers set the consensus (gold-standard), individual agreement with it was 88%, 94% and 70%. The agreement between AI and “gold-standard” was 73%, over reader agreement was 46%.
Conclusion AI-based software can be used to measure spirometry data quality with comparable accuracy as experts. The assessment is a subjective exercise, with intra- and inter-rater variability even when the criteria are defined very precisely and objectively. By providing consistent results and immediate feedback to the sites, AI may benefit clinical trial conduct and variability reduction.
Footnotes
This manuscript has recently been accepted for publication in the ERJ Open Research. It is published here in its accepted form prior to copyediting and typesetting by our production team. After these production processes are complete and the authors have approved the resulting proofs, the article will move to the latest issue of the ERJOR online. Please open or download the PDF to view this article.
Conflict of interest: Sandrine CORRE received support for the present manuscript from Chiesi, for which they are an employee.
Conflict of interest: Marko Topalovic received support for the present manuscript from ArtiQ, for which they are a co-founder.
Conflict of interest: Eva Topole received support for the present manuscript from Chiesi, for which they are an employee.
Conflict of interest: Isabella Montagna received support for the present manuscript from Chiesi, for which they are an employee.
Conflict of interest: BIONDARO Sonia received support for the present manuscript from Chiesi, for which they are an employee.
Conflict of interest: Sanja Stanojevic has received consulting fees from Chiesi Farmaceuticals, outside the submitted work.
Conflict of interest: Brian L Graham reports the following relationships outside the submitted work; received professional fees from Chiesi Farmaceutici S.p.A., MGC Diagnostics, Vyaire Medical, and the Lung Association of Saskatchewan. BG has a patent licensing agreement with Hans Rudolph Inc.
Conflict of interest: Kevin Ray received support for the present manuscript from ArtiQ, for which they are an employee.
Conflict of interest: NILAKASH DAS received support for the present manuscript from KU Leuven university for which they were a Post-doctoral researcher; patent pending for ArtiQ; stock options in lieu of patents held for ArtiQ.
Conflict of interest: Massimo Corradi has a patent pending.
This is a PDF-only article. Please click on the PDF link above to read it.
- Received June 24, 2022.
- Accepted September 23, 2022.
- Copyright ©The authors 2022
This version is distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0. For commercial reproduction rights and permissions contact permissions{at}ersnet.org
