Original ArticleGRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences—Risk of bias and indirectness
Introduction
Decisions in health care require not only evidence about the effects of interventions (eg, the absolute risk reduction or increase for an outcome in a particular population resulting from a specific intervention when compared with an alternative) but also knowledge of the relative importance of the outcomes that interventions prevent or cause (see Box 1 for a hypothetical example).
Incorporating these concepts in health-care decision-making often refers to considerations of values and preferences [1], [2], [3], [4], [5], [6], [7]. In the context of decision-making, values and preferences represent the relative importance people place on the outcomes of interest resulting from a decision (eg, about accepting a treatment or undergoing a test) [1], [2], [3], [4], [5], [6], [7].
The methods that investigators use to ascertain the relative importance of the outcomes include (a) direct measurement of the utility or value of outcomes, for example, with the standard gamble [8], [9], [10], time trade-off [11], [12], or rating scales [9], [13], [14]. Conjoint analysis is another category to elicit utility and indicate outcome importance, which includes discrete choice experiments [15], [16], contingent valuation and willingness to pay [17], probability trade-off [18], [19], paired comparison; (b) indirect measurement of utility: results from instruments such as the EuroQual-5-dimension (EQ-5D) utility, or Short form-6-dimension (SF-6D) utility, which would transform the measurement results across several domains, that is, pain, mobility, into the utility [20], [21]; or (c) other quantitative surveys and questionnaires that provide outcome importance information in a nonutility manner [22], [23]. In addition, qualitative studies can provide evidence about the relative importance of outcomes [24], [25] (see Appendix 1).
Given health-care decisions will be influenced by both the health effects of interventions, and the relative importance of the outcomes of interest, they both require appropriate methods of certainty assessment. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group has developed approaches to assess certainty of evidence addressing intervention effects [26], [27], test accuracy [28], resources [29], prognosis [30], and qualitative evidence [31]. However, GRADE recognized the increasing need to develop a transparent and structured approach to assess the certainty of evidence for relative importance of the outcomes. Conceptually the GRADE process has required judgments of the certainty in relative importance of the outcomes to draw conclusions in its Evidence to Decision (EtD) tables and frameworks [32], [33], [34], [35], [36], [37]. In the last iteration of the GRADE EtD frameworks, the question related to the relative importance of outcomes is “is there important uncertainty about or variability in how much people value the main outcomes?”
Having illustrated the rationale for these considerations, we will describe our terminology (See Box 2) [3], [38]. We recognize inconsistent use of the terms in the scientific community. For example, not all scientists agree that a visual analogue scale (VAS) is a utility instrument because it does not require a choice under uncertainty. While acknowledging this fact, we use “outcome importance,” which includes but goes beyond the strict definition of “utility.” The merit of using “outcome importance” is that it is consistent with the conceptual process of balancing health benefits and harms. In addition, we focus on “relative” importance of outcomes to express that the importance relates to anchors (eg, 0 indicating death, and 1 indicating perfect health) or other outcomes that an intervention causes and which may be balanced against each other to make informed decisions.
The aim of this and the next article related to relative importance of the outcomes is to provide guidance about the GRADE approach for assessing the certainty of a body of evidence dealing with relative importance of the outcomes. In this article, we describe the definitions and methods of this project, and the GRADE approach for rating the domains' risk of bias and indirectness of relative importance of the outcomes. The second article will focus on the domains of inconsistency, imprecision, and publication bias and rating up the certainty of evidence. The second article will also clarify what variability of values and preferences or the relative importance of the outcomes means in this context.
Section snippets
Methods
This document presents formal guidance by the GRADE working group for rating the relative importance of outcomes. We developed the guidance using an iterative multipronged approach to develop this GRADE guidance. The work was presented at GRADE working group meetings and reviewed by members of the GRADE working group before approval through a vote at a GRADE working group meeting in Rome, Italy, on April 27, 2017. It was then formally approved by the GRADE guidance group.
Guidance for GRADE domains
We did not identify additional domains for assessing the certainty of the body of evidence describing relative importance of the outcomes, beyond what the GRADE working group had suggested previously: risk of bias, inconsistency, indirectness, imprecision, publication bias, and domains to rate up the evidence [38]. Here, we focus on the detailed guidance for the GRADE domains risk of bias and indirectness.
Risk of bias or limitations in the detailed study design or execution
Risk of bias may be a concern at different stages of an investigation into relative importance of the outcomes, including study design, study execution, data analysis, and reporting [42]. Assessing risk of bias for the relative importance of outcomes is similar to that for intervention effects in that it requires first an assessment of the risk of bias for individual studies, followed by an assessment for the body of evidence. However, it differs in several important ways. First, unlike studies
Indirectness
Indirectness can be a reason to rate down the certainty of evidence in relative importance of the outcomes [48]. The assessment of indirectness for relative importance of the outcomes has its specific features. First, studies usually do not directly compare the intervention options; rather, the focus is on outcomes. Second, surrogate outcomes or outcomes that are not patient-important are a source of indirectness for treatment questions–this may not be the case for the evidence of relative
Summary
This article describes the use of GRADE to assess the certainty of evidence for the relative importance of outcomes when considering risk of bias and indirectness. When assessing certainty of evidence for the relative importance of outcomes, evidence starts at “high” for all study designs, with rating down if risk of bias or indirectness is a serious concern. Users rate down by one or more levels depending on the specific considerations for the two domains.
Risk of bias assessment in this
Acknowledgments
The authors are grateful to Dr Amiram Gafni from McMaster University for comments on the manuscripts.
Authors' contributions: Y.Z., P.A., G.G., and H.J.S. designed the methodology for this project. H.J.S. conceived the project and approach. Y.Z., J.J.Y.N., and Y.C. summarized the certainty assessment for relevant items in systematic reviews; Y.Z., P.A., G.G., and H.J.S. proposed the subdomains for certainty of evidence assessment; all authors participated in the methodological discussions. All
References (51)
- et al.
Patient values and preferences in decision making for antithrombotic therapy: a systematic review: antithrombotic therapy and prevention of thrombosis, 9th ed.: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines
Chest
(2012) - et al.
GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength
J Clin Epidemiol
(2013) Measurement of health state utilities for economic appraisal
J Health Econ
(1986)- et al.
Comparison of contingent valuation and conjoint analysis in ecosystem management
Ecol Econ
(2000) - et al.
How to define and measure concordance between patients' preferences and medical treatments: a systematic review of approaches and recommendations for standardization
Patient Educ Couns
(2010) - et al.
GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence
J Clin Epidemiol
(2013) - et al.
GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health
J Clin Epidemiol
(2016) - et al.
GRADE guidelines: 3. Rating the quality of evidence
J Clin Epidemiol
(2011) - et al.
Forty-two systematic reviews generated 23 items for assessing the risk of bias in values and preferences' studies
J Clin Epidemiol
(2017) - et al.
GRADE guidelines: 9. Rating up the quality of evidence
J Clin Epidemiol
(2011)
GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables
J Clin Epidemiol
A new “mechanistic-practical” framework for designing and interpreting randomized trials
J Clin Epidemiol
Response rates and responsiveness for surveys, standards, and the Journal
Am J Pharm Educ
GRADE guidelines: 2. Framing the question and deciding on important outcomes
J Clin Epidemiol
GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias)
J Clin Epidemiol
GRADE guidelines: 8. Rating the quality of evidence–indirectness
J Clin Epidemiol
Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement
Health Res Policy Syst
Incorporating patient preferences in evidence-based medicine
JAMA
Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise
CMAJ
The next step in guideline development: incorporating patient preferences
JAMA
How to integrate individual patient values and preferences in clinical practice guidelines? A research protocol
Implement Sci
The standard gamble method: what is being measured and how it is interpreted
Health Serv Res
Utility measurement in healthcare: the things I never got to
PharmacoEconomics
Measurement of quality of life in end-stage renal disease: the time trade-off approach
Clin Invest Med
The time trade-off method: results from a general population study
Health Econ
Cited by (0)
Ethics approval and consent to participate: Not required. This study does not involve de novo patient data collection. No patient informed consent and Institutional Review Board approval have been sought.
Availability of data and materials: The data sets supporting the conclusions of this article are included within the article and its additional file.
Conflicts of interest: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/conflicts-of-interest/).
Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. It was funded through internal research funds at McMaster University available to H.J.S. G.H. is supported by a CIHR New Investigator Salary Award and a The Arthritis Society Young Investigator Salary Award, neither of which is directly related to this research project.