Original Article
GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences—Risk of bias and indirectness

https://doi.org/10.1016/j.jclinepi.2018.01.013Get rights and content

Abstract

Objectives

The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group defines patient values and preferences as the relative importance patients place on the main health outcomes. We provide GRADE guidance for assessing the risk of bias and indirectness domains for certainty of evidence about the relative importance of outcomes.

Study Design and Setting

We applied the GRADE domains to rate the certainty of evidence in the importance of outcomes to several systematic reviews, iteratively reviewed draft guidance and consulted GRADE members and other stakeholders for feedback.

Results

This is the first of two articles. A body of evidence addressing the importance of outcomes starts at “high certainty”; concerns with risk of bias, indirectness, inconsistency, imprecision, and publication bias lead to downgrading to moderate, low, or very low certainty. We propose subdomains of risk of bias as selection of the study population, missing data, the type of measurement instrument, and confounding; we have developed items for each subdomain. The population, intervention, comparison, and outcome elements associated with the evidence determine the degree of indirectness.

Conclusion

This article provides guidance and examples for rating the risk of bias and indirectness for a body of evidence summarizing the importance of outcomes.

Introduction

Decisions in health care require not only evidence about the effects of interventions (eg, the absolute risk reduction or increase for an outcome in a particular population resulting from a specific intervention when compared with an alternative) but also knowledge of the relative importance of the outcomes that interventions prevent or cause (see Box 1 for a hypothetical example).

Incorporating these concepts in health-care decision-making often refers to considerations of values and preferences [1], [2], [3], [4], [5], [6], [7]. In the context of decision-making, values and preferences represent the relative importance people place on the outcomes of interest resulting from a decision (eg, about accepting a treatment or undergoing a test) [1], [2], [3], [4], [5], [6], [7].

The methods that investigators use to ascertain the relative importance of the outcomes include (a) direct measurement of the utility or value of outcomes, for example, with the standard gamble [8], [9], [10], time trade-off [11], [12], or rating scales [9], [13], [14]. Conjoint analysis is another category to elicit utility and indicate outcome importance, which includes discrete choice experiments [15], [16], contingent valuation and willingness to pay [17], probability trade-off [18], [19], paired comparison; (b) indirect measurement of utility: results from instruments such as the EuroQual-5-dimension (EQ-5D) utility, or Short form-6-dimension (SF-6D) utility, which would transform the measurement results across several domains, that is, pain, mobility, into the utility [20], [21]; or (c) other quantitative surveys and questionnaires that provide outcome importance information in a nonutility manner [22], [23]. In addition, qualitative studies can provide evidence about the relative importance of outcomes [24], [25] (see Appendix 1).

Given health-care decisions will be influenced by both the health effects of interventions, and the relative importance of the outcomes of interest, they both require appropriate methods of certainty assessment. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group has developed approaches to assess certainty of evidence addressing intervention effects [26], [27], test accuracy [28], resources [29], prognosis [30], and qualitative evidence [31]. However, GRADE recognized the increasing need to develop a transparent and structured approach to assess the certainty of evidence for relative importance of the outcomes. Conceptually the GRADE process has required judgments of the certainty in relative importance of the outcomes to draw conclusions in its Evidence to Decision (EtD) tables and frameworks [32], [33], [34], [35], [36], [37]. In the last iteration of the GRADE EtD frameworks, the question related to the relative importance of outcomes is “is there important uncertainty about or variability in how much people value the main outcomes?”

Having illustrated the rationale for these considerations, we will describe our terminology (See Box 2) [3], [38]. We recognize inconsistent use of the terms in the scientific community. For example, not all scientists agree that a visual analogue scale (VAS) is a utility instrument because it does not require a choice under uncertainty. While acknowledging this fact, we use “outcome importance,” which includes but goes beyond the strict definition of “utility.” The merit of using “outcome importance” is that it is consistent with the conceptual process of balancing health benefits and harms. In addition, we focus on “relative” importance of outcomes to express that the importance relates to anchors (eg, 0 indicating death, and 1 indicating perfect health) or other outcomes that an intervention causes and which may be balanced against each other to make informed decisions.

The aim of this and the next article related to relative importance of the outcomes is to provide guidance about the GRADE approach for assessing the certainty of a body of evidence dealing with relative importance of the outcomes. In this article, we describe the definitions and methods of this project, and the GRADE approach for rating the domains' risk of bias and indirectness of relative importance of the outcomes. The second article will focus on the domains of inconsistency, imprecision, and publication bias and rating up the certainty of evidence. The second article will also clarify what variability of values and preferences or the relative importance of the outcomes means in this context.

Section snippets

Methods

This document presents formal guidance by the GRADE working group for rating the relative importance of outcomes. We developed the guidance using an iterative multipronged approach to develop this GRADE guidance. The work was presented at GRADE working group meetings and reviewed by members of the GRADE working group before approval through a vote at a GRADE working group meeting in Rome, Italy, on April 27, 2017. It was then formally approved by the GRADE guidance group.

Guidance for GRADE domains

We did not identify additional domains for assessing the certainty of the body of evidence describing relative importance of the outcomes, beyond what the GRADE working group had suggested previously: risk of bias, inconsistency, indirectness, imprecision, publication bias, and domains to rate up the evidence [38]. Here, we focus on the detailed guidance for the GRADE domains risk of bias and indirectness.

Risk of bias or limitations in the detailed study design or execution

Risk of bias may be a concern at different stages of an investigation into relative importance of the outcomes, including study design, study execution, data analysis, and reporting [42]. Assessing risk of bias for the relative importance of outcomes is similar to that for intervention effects in that it requires first an assessment of the risk of bias for individual studies, followed by an assessment for the body of evidence. However, it differs in several important ways. First, unlike studies

Indirectness

Indirectness can be a reason to rate down the certainty of evidence in relative importance of the outcomes [48]. The assessment of indirectness for relative importance of the outcomes has its specific features. First, studies usually do not directly compare the intervention options; rather, the focus is on outcomes. Second, surrogate outcomes or outcomes that are not patient-important are a source of indirectness for treatment questions–this may not be the case for the evidence of relative

Summary

This article describes the use of GRADE to assess the certainty of evidence for the relative importance of outcomes when considering risk of bias and indirectness. When assessing certainty of evidence for the relative importance of outcomes, evidence starts at “high” for all study designs, with rating down if risk of bias or indirectness is a serious concern. Users rate down by one or more levels depending on the specific considerations for the two domains.

Risk of bias assessment in this

Acknowledgments

The authors are grateful to Dr Amiram Gafni from McMaster University for comments on the manuscripts.

Authors' contributions: Y.Z., P.A., G.G., and H.J.S. designed the methodology for this project. H.J.S. conceived the project and approach. Y.Z., J.J.Y.N., and Y.C. summarized the certainty assessment for relevant items in systematic reviews; Y.Z., P.A., G.G., and H.J.S. proposed the subdomains for certainty of evidence assessment; all authors participated in the methodological discussions. All

References (51)

  • G. Guyatt et al.

    GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables

    J Clin Epidemiol

    (2011)
  • P.J. Karanicolas et al.

    A new “mechanistic-practical” framework for designing and interpreting randomized trials

    J Clin Epidemiol

    (2009)
  • J.E. Fincham

    Response rates and responsiveness for surveys, standards, and the Journal

    Am J Pharm Educ

    (2008)
  • G.H. Guyatt et al.

    GRADE guidelines: 2. Framing the question and deciding on important outcomes

    J Clin Epidemiol

    (2011)
  • G.H. Guyatt et al.

    GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias)

    J Clin Epidemiol

    (2011)
  • G.H. Guyatt et al.

    GRADE guidelines: 8. Rating the quality of evidence–indirectness

    J Clin Epidemiol

    (2011)
  • H.J. Schunemann et al.

    Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement

    Health Res Policy Syst

    (2006)
  • M.H. Murad et al.

    Incorporating patient preferences in evidence-based medicine

    JAMA

    (2008)
  • H.J. Schunemann et al.

    Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise

    CMAJ

    (2014)
  • M. Krahn et al.

    The next step in guideline development: incorporating patient preferences

    JAMA

    (2008)
  • T. van der Weijden et al.

    How to integrate individual patient values and preferences in clinical practice guidelines? A research protocol

    Implement Sci

    (2010)
  • A. Gafni

    The standard gamble method: what is being measured and how it is interpreted

    Health Serv Res

    (1994)
  • G.W. Torrance

    Utility measurement in healthcare: the things I never got to

    PharmacoEconomics

    (2006)
  • D.N. Churchill et al.

    Measurement of quality of life in end-stage renal disease: the time trade-off approach

    Clin Invest Med

    (1987)
  • P. Dolan et al.

    The time trade-off method: results from a general population study

    Health Econ

    (1996)
  • Cited by (0)

    Ethics approval and consent to participate: Not required. This study does not involve de novo patient data collection. No patient informed consent and Institutional Review Board approval have been sought.

    Availability of data and materials: The data sets supporting the conclusions of this article are included within the article and its additional file.

    Conflicts of interest: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/conflicts-of-interest/).

    Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. It was funded through internal research funds at McMaster University available to H.J.S. G.H. is supported by a CIHR New Investigator Salary Award and a The Arthritis Society Young Investigator Salary Award, neither of which is directly related to this research project.

    View full text