Consensus measurement in Delphi studies: Review and implications for future quality assurance
Highlights
► Research reveals deficits in the practice and rigour of consensus measurement for Delphi research. ► Qualitative criteria, descriptive and inferential statistics are used for consensus measurement. ► Consensus is not a suitable stopping criterion in Delphi research. ► Consensus measurement is an important component for Delphi analysis and interpretation.
Introduction
Mankind has always desired to know what the future will be like. Throughout history, people have consulted chosen individuals who were said to be able to envision the future. Among the most famous prophesiers is Michel de Notredame (1503–1566) or Marie Anne Adélaide Lenormand (1772–1843). For over thousand years, oracles constituted the lives of Romans and Greeks. From the eighth century B.C. until the third century A.D., people primarily consulted oracles regarding fortune, success, marital affairs, professional advancement, and judicial disputes [1], [2]. In these times, oracular sites were spread all over Greece. The two greatest were in Delphi, associated with Apollo, and in Dodona, associated with Zeus [3]. The Greek word Delphoi means “hollow” or “womb”. Historians interpret it as a reference to Gaia, the great mother of all creatures on Earth or the primordial Earth goddess, in the Ancient Greek religion.
In the 1950s, the term Delphi was adopted by the U.S. RAND Corporation for its research purposes. The RAND Corporation was a research institution that initially focused on national security issues and later concentrated on scientific, educational, and charitable endeavours for public welfare. Within the scope of the “Project Delphi”, RAND researchers developed a structured survey in written form in order to estimate bombing requirements. For confidentiality reasons, the contents of the experiment were first published by Dalkey and Helmer [4] in their article “An Experimental Application of the Delphi Method for the Use of Experts” 10 years later. Project Delphi was sponsored by the United States Air Force and included the application of “expert opinion of the selection, from the viewpoint of a Soviet strategic planner, of an optimal U.S. industrial target system and to the estimation of the number of A-bombs required to reduce the munitions output by a prescribed amount” [4, p.458]. The expert panel consisted of seven specialists in the areas of economics, physics, systems analysis, and electronics. Dalkey and Helmer [4] reported that the experts' first evaluation of possible industry targets did not result in consensus. However, in a second estimate, consensus was achieved and the procedure was said to have yielded more reliable results than comparable techniques. Shortly after the technique's introduction to the public in 1963, various studies using the technique on non-military issues followed.
Since the 1950s, the usage of the Delphi survey method has undergone different stages of development, which were described by Rieger [5]:
- 1.
Secrecy/obscurity (1950s): exclusive application in the military context
- 2.
Novelty (1960s): declassification by the U.S. military and introduction to the public
- 3.
Popularity (1970–1975): spread to Western Europe, Eastern Europe, and Asia; major forecasting tool in business
- 4.
Scrutiny (1975–1980): critical evaluation of the technique's reliability and validity
- 5.
Continuity (1980–1986): acceptance in science and practice; stable application patterns
After a time of stagnation in the 1980s, the Delphi technique received increasing interest in the early 1990s again. As an extensive literature review by Landeta [6] shows, this trend prevails. In total, 414 Delphi-related articles were published in the two major databases “Science Direct” and “ABI/Inform” during 1995 and 1999 [6]. This number increased to 677 articles in the period between 2000 and 2004. Similar patterns can be observed for book publications. Google Ngram viewer displays graphs showing how certain phrases have occurred in a collection of books (generated in July 2009) during time (see [7]). From 1950 until 2008, a bigram search in a sample of more than one million English books for “Delphi study” reveals a strong increase from almost zero in 1960 to peak in 1962, but then to decrease to about half of the 1980 frequency around 1990. Since then, the term “Delphi study” was used increasingly and after 2005 reached a level higher than all years before.
More recent applications concentrate on the web-based implementation of the Delphi procedure, but still follow the technique's fundamental rationale and consider consensus measurement to be a key component of analysis [8], [9]. The facts presented here underline that the Delphi technique is widely accepted as a research technique today and that its value has been scientifically and practically proven.
This paper contributes to research on the Delphi techniques in three ways. First, it identifies and explains 15 different types of measures related to the field of consensus measurement and/or stability over Delphi rounds. Second, it gives an overview of the criteria that have been defined for these 15 different types of measurement. Third, an overall assessment is presented that gives guideline for future Delphi work. The overall research questions are therefore: (1) how has consensus been measured in Delphi studies since its emergence in the 1960s until today?; (2) which levels of measurement have been used to define consensus?; and (3) which implications should be considered for future quality assurance in Delphi research? The research presented here is therefore conceptual in nature and represents a comprehensive literature review across multiple disciplines and research strands.
The remainder of the paper is organized as follows: Section 2 summarizes the fundamental characteristics and rationale of Delphi in order to set the basis for the following core sections on consensus measurement. Section 3 is dedicated to general definitions of “consensus”, its relationship with stability and the differentiation from dissent-oriented Delphi studies. 4 The use of subjective criteria and descriptive statistics, 5 Inferential statistics for consensus measurement provide an overview of different types of consensus measurement and the defined consensus criteria. This includes subjective and descriptive types of measurement as well as inferential statistics. Conclusions will be stated in Section 6; limitations and future research will be addressed in Section 7.
Section snippets
Fundamental characteristics of the Delphi method
The Delphi technique is survey technique in order to facilitate an efficient group dynamic process. This is done in the form of an anonymous, written, multi-stage survey process, where feedback of group opinion is provided after each round. In most cases such studies further focus on opinion building, usually consensus, among experts. Although there are differences in the focus of definitions and the procedure of the technique, four distinct characteristics of Delphi usually remain the same [10]
The concept of consensus in Delphi studies
The efficient structuring of a group communication process can be considered the primary goal of a Delphi study. Consensus measurement in turn should be considered a valuable component of data analysis and interpretation in Delphi research. Nevertheless, many researchers have used it as a sole stopping criterion of rounds, which does not match the original idea of Delphi and is not recommended. In fact, it is important to distinguish between the two different concepts “consensus/agreement” and
The use of subjective criteria and descriptive statistics
Many Delphi studies have used subjective criteria or descriptive statistics for the determination of consensus and the quantification of its degree. The criteria have, however, sometimes been chosen rather arbitrarily. The literature review revealed that researchers have actually used all kinds of descriptive statistics in order to measure consensus. One can find applications of measures of association as well as measures of central tendency and dispersion. Table 1 summarizes the research
Inferential statistics for consensus measurement
Inferential statistics are statistics that help to establish relationships among variables and draw conclusions therefrom [86]. The application of such statistical tests depends on the level of data and whether this data conforms approximately to a normal distribution [83]. If the latter is the case and the data is interval/ratio-scaled, parametric tests can be used. Nonparametric tests, on the other hand, can be used on nominal- or ordinal-scaled data not conforming to a normal frequency
Conclusion
The previous sections have presented the results of an extensive literature review on consensus measurement in Delphi studies including accompanying tests for stability. This review revealed that a general standard of how to measure consensus in Delphi studies does not yet exist. Researchers have applied subjective criteria as well as descriptive and inferential statistics to measure consensus and convergence. Especially in the case of the latter, violations in basic assumptions have been found
Limitations and future research
As with any research, the discussions within this article have some limitations and generate questions for future research. First, the review within this article strongly focused on consensus measurement, which is by far the most often considered concept in Delphi studies since its development. However, the author is aware of the Delphi research that focusses on dissent rather than consensus, e.g. policy Delphi or emerging dissensus-based Delphi methods. Recently, there have been an increasing
Acknowledgments
The content of this publication is partly based on work of the joint research project “Competitiveness Monitor”, funded by the German Federal Ministry of Education and Research (project reference number: 01IC10L18 A) in the course of its leading-edge cluster initiative. I would like to thank all involved in the leading-edge cluster as well as its partners and sponsors. Furthermore, I would like to thank Janice Magel, Dr. Tobias Gnatzy, Philipp Ecken and Dr. Inga-Lena Darkow for their valuable
Dr. Heiko A. von der Gracht is Founder and Director of the Institute for Futures Studies and Knowledge Management (IFK) and post-doctoral researcher at EBS Business School in Wiesbaden, Germany. His research interests are corporate foresight, Delphi and scenario techniques, decision support, and quality in futures research. His works have been published in several books and in peer-reviewed journals, among them Technological Forecasting & Social Change, Futures, and International Journal of
References (114)
- et al.
The Delphi method as a ritual: inquiring the Delphic Oracle
Technol. Forecast. Soc. Chang.
(2011) Towards a history of futurism: the Greeks
Futures
(1974)Directions in Delphi developments: dissertations and their quality
Technol. Forecast. Soc. Chang.
(1986)Current validity of the Delphi method in social sciences
Technol. Forecast. Soc. Chang.
(2006)- et al.
Validating an innovative real-time Delphi approach — a methodological comparison between real-time and conventional Delphi studies
Technol. Forecast. Soc. Chang.
(2011) - et al.
RT Delphi: an efficient, “round-less” almost real time Delphi method
Technol. Forecast. Soc. Chang.
(2006) - et al.
Delphi, political philosophy and the future
Futures
(1975) - et al.
Delphi: a brief look backward and forward
Technol. Forecast. Soc. Chang.
(2011) - et al.
Stability and agreement criteria for the termination of Delphi studies
Technol. Forecast. Soc. Chang.
(1979) - et al.
Individual stability in delphi studies
Technol. Forecast. Soc. Chang.
(1980)
The design of a policy Delphi
Technol. Forecast. Soc. Chang.
The decision Delphi
Technol. Forecast. Soc. Chang.
A dissensus based online Delphi approach: an explorative research tool
Technol. Forecast. Soc. Chang.
Review of Delphi-based scenario studies: quality and design considerations
Technol. Forecast. Soc. Chang.
Applying a risk management process (RMP) to manage cost risk for an EHV transmission line project
Int. J. Proj. Manag.
E-commerce project development risks: evidence from a Delphi survey
Int. J. Inf. Manag.
The Delphi method as early warning: linking global societal trends to future radicalization and terrorism in The Netherlands
Technol. Forecast. Soc. Chang.
Disaggregative policy Delphi: using cluster analysis as a tool for systematic scenario formation
Technol. Forecast. Soc. Chang.
People consultation to construct the future: a Delphi application
Int. J. Forecast.
Enhancing rigour in the Delphi technique research
Technol. Forecast. Soc. Chang.
Modified Delphi methodology for technology forecasting case study of electronics and information technology in India
Technol. Forecast. Soc. Chang.
Analytical search of problems and prospects of power sector through Delphi study: case study of Kerala State, India
Energy Policy
Identifying critical cross-cultural school psychology competencies
J. Sch. Psychol.
Decision variables for selecting prototyping in information systems development: a Delphi study of MIS managers
Inf. Manag.
Scenarios for the logistics services industry: a Delphi-based analysis for 2025
Int. J. Prod. Econ.
The Delphi technique as a forecasting tool: issues and analysis
Int. J. Forecast.
Improving the quality of group judgment: social judgment analysis and the Delphi technique
Organ. Behav. Hum. Perform.
The role of knowledge codification in the emergence of consensus under uncertainty: empirical analysis and policy implications
Res. Policy
Participants' response to the Delphi method: an attitudinal perspective
Technol. Forecast. Soc. Chang.
Global prevalence of dementia: a Delphi consensus study
Lancet
A Delphi study of goals and evaluation criteria of state and privately owned latin American airlines
Socioecon. Plann. Sci.
The Delphi process as a tool for decision making
A Brief History of the Future. How Visionary Thinkers Changed the World and Tomorrow's Trends are ‘Made’ and Marketed
An experimental application of the Delphi method to the use of experts
Manag. Sci.
Quantitative analysis of culture using millions of digitized books
Science
Expert opinions in forecasting: the role of the Delphi technique
The Delphi method: a description
Rev. Critic. J. Acad. Librariansh.
Delphi-Befragungen. Ein Arbeitsbuch
Public Policy Analysis. An Introduction
An overview of futures methods
Experiments in Delphi methodology
The Delphi method?
Nurs. Res.
Building consensus using the policy Delphi method
Policy Polit. Nurs. Pract.
Testing the stability of experts' opinions between successive rounds of Delphi studies
The Delphi technique: an exposition and application
Technol. Anal. Strateg. Manag.
Consensus methods: characteristics and guidelines for use
Am. J. Public Health
American Heritage Dictionary of the English Language
Principles of Forecasting: A Handbook for Researchers and Practitioners
Decisions, decisions, decisions
Psychol. Today
The Policy Delphi
Cited by (1043)
Expert consensus-based clinical recommendation for an integrative anthroposophic treatment approach to acute tonsillitis in childhood
2024, Complementary Therapies in MedicineCan graphene fuel a transformative change in energy storage technologies? A scenario analysis for the next two decades
2024, Technological Forecasting and Social ChangeOrange grower's perception of drought impacts and strategies for mitigation and adaptation: A study of the Vidarbha region in India
2024, International Journal of Disaster Risk ReductionEliciting expert judgements to underpin our understanding of faecal indicator organism loss from septic tank systems
2024, Science of the Total Environment
Dr. Heiko A. von der Gracht is Founder and Director of the Institute for Futures Studies and Knowledge Management (IFK) and post-doctoral researcher at EBS Business School in Wiesbaden, Germany. His research interests are corporate foresight, Delphi and scenario techniques, decision support, and quality in futures research. His works have been published in several books and in peer-reviewed journals, among them Technological Forecasting & Social Change, Futures, and International Journal of Production Economics.