Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. Facilitation plays a pivotal role.
Facilitators should understand the strengths and weaknesses of the qualitative assessments we facilitate and design. Qualitative assessments measure the opinions, attitudes, knowledge, perceived behaviors, observations, beliefs, and experiences of individuals who use a system the most. The correct understanding is fundamental as we lead participants to solutions that are created, understood, and accepted by all.
The #1 Thing Facilitators and Technical Experts Get Wrong About Qualitative Assessments
The number #1 thing is how to focus on design, analysis, and administration to make qualitative assessments effective. We traced the rich history of qualitative assessments for more than 100 years. Many of the same debates about human judgment, statistics, and math are still being debated today.
It seems that most people fall into either the qualitative or quantitative camp and fight with each other from a binary position. The truth is that most decisions are made from a hybrid, weight-of-evidence approach.
The article, The #1 Thing Facilitators and Technical Experts Get Wrong About Qualitative Assessments, has more helpful background.
Why Qualitative Assessment Design Is Important
Facilitators often overlook the impact of the design of qualitative assessments that they use in their facilitated sessions. Several modern-era sources of good practices are available. In addition to the author’s own experiences, the basis for the ones provided in the article relates closely to the field of human reliability engineering.
Facilitators are more effective when they understand the strengths and weaknesses associated with the design of assessments that they are using. See more in the article, FINESSE Facilitation: Why Qualitative Assessment Design Is Important.
What Are Best Practices for Qualitative Assessment Analysis?
Qualitative assessments use ordinal data, which means traditional math and statistics are not technically valid. One hundred years ago, Rensis Likert concluded that opinions and social attitudes are best analyzed by “clustering.” Likert cited the need to “cut through the statistical confusion” created even in his time.
Likert’s analysis recommendations are summarized as follows:
- Plot the data. Examine for visual inconsistencies.
- Report the most representative clustered value (central tendency)
- Report the range of responses and, even better, report the value that most responses were above or below
- Compare the clustered values (medians) between data sets to evaluate reliability (repeatability) and validity
The article FINESSE Facilitation has many more details: What Are Best Practices for Qualitative Assessment Analysis?
What Are Best Practices for Facilitating Qualitative Assessments?
Qualitative assessments are cost-effective and highly flexible tools. They measure the opinions, attitudes, knowledge, perceived behaviors, observations, beliefs, and experiences of individuals who use a system the most. Human factors should be included in any system evaluation and aspects of system performance that we cannot or do not understand how to measure.
“The motivations of the respondents are affected by what the respondents were asked to do before the survey, their beliefs as to what will happen with the survey data, how interested they are in the topic of the survey, and how frequently they are surveyed. Motivation can be increased or decreased by the design of the test (e.g., completing the same survey multiple times), the design of the survey (e.g., length, question wording), and the administration of the survey (e.g., when administered).”
The Institute for Defense Analysis
Arguably, correctly administering surveys is the most important aspect of the facilitator’s job when it comes to qualitative assessment. See more tips in the article, What Are Best Practices for Facilitating Qualitative Assessments?
Qualitative Assessments: 3 Bad Examples That Will Improve Your Effectiveness
This article provides three examples of poor qualitative analysis to avoid and, in turn, improve the effectiveness of your work as a facilitator. Poorly executed qualitative assessment from a professional association, customer surveys from a Fortune 500 company, and a management assessment of a public utility were provided for learning purposes.
More on what not to do is found in the article, Qualitative Assessments: 3 Bad Examples That Will Improve Your Effectiveness.
Qualitative Assessments: Do the Fine Points of Risk Matrices Really Matter?
Examination of ten examples yielded a conclusion that, in practice, qualitative assessments provide acceptable but not optimal results. However, in most cases, optimal results are not needed.
Some more specific conclusions include the following:
- A minority of risk matrices are designed, analyzed, and administered properly.
- When done correctly, the results of risk matrices are acceptable for initial prioritization purposes. (more analysis and decision making may be needed, depending on the context)
- There is a significant practitioner problem in analyzing ordinal scales and associated data.
- There is a significant practitioner problem in designing and analyzing multi-attribute variables, particularly related to the consequences of failure (CoF).
- Common sense or luck often prevails in the finalized prioritization.
- Organizations have some slop in the normal selection of priorities. Using risk matrices produces no worse-than-normal results.
- Many organizations can only afford to invest in high priorities, so the fine differences in the middle often do not matter.
The article concludes with the question, “Do the fine points of risk matrices really matter?” Yes, because someone pays a trained professional to do the technical assessment correctly. Qualitative assessments used to develop a risk matrix are not designed, constructed, and administered appropriately.
The article, Qualitative Assessments: Do the Fine Points of Risk Matrices Really Matter?, provides more details.
Facilitating with FINESSE
Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration.
Facilitators should understand the strengths and weaknesses of the qualitative assessments we facilitate and design. Qualitative assessments measure the opinions, attitudes, knowledge, perceived behaviors, observations, beliefs, and experiences of individuals who use a system the most. The correct understanding is fundamental as we lead participants to solutions that are created, understood, and accepted by all.
References
Rensis Likert, R.S. Woodruff, editor, Archives of Psychology, Columbia University, New York, Volume XXII, Nos. 146-146, 1932-1933, pp.4-43.
Handbook of Human Factors Testing and Evaluation, 2nd edition, edited by S.G. Charlton and T.G. O’Brien, publishers Lawrence Erlbaum Associates, Inc., 2002.
Institute of Defense Analysis, “ICH Q9 Briefing Pack II”, July 2006.
European Association of Methodology (EAM), International Handbook of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 2008.
J.D. Solomon, Daniel Vallero, and Kathryn Benson, “Evaluating Risk: A Revisit of the Scales, Measurement Theory, and Statistical Analysis Controversy,” Proceedings of the 2017 international Reliability and Maintainability Symposium.
JD Solomon Inc provides solutions for facilitation, asset management, and program development at the nexus of facilities, infrastructure, and the environment. Founded by JD Solomon, Communicating with FINESSE is the community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.
Leave a Reply