Opinion-based data is the foundation of qualitative assessments. Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. This article provides three examples of poor qualitative analysis to avoid and, in turn, improve the effectiveness of your work.
Qualitative Assessment Scales
Likert scales are five-point ordinal scales where participants select a label (or numerical value) that equates to an opinion or attitude. Labels are paired (i.e., agree-disagree, strongly agree-strongly disagree) symmetrically around a neutral center. Today, we see 5-, 7-, and 9-point versions of Likert scales in everything from customer surveys to risk assessments.
Example 1
The first example comes from an evaluation conducted by a professional engineering association, which suspected it was not positioned correctly in the eyes of regulators, the public, and elected officials. Survey questions were developed and analyzed by a public relations firm and reviewed by licensed engineers. However, the data were treated as continuous and the mean (average), extended to two decimal places, was used to reach sweeping conclusions.
By way of example, a question was asked to most members of the state assembly (elected officials) concerning the opinion of numerous professions on a 5-point Likert scale. The analysis yielded the following results: physicians (4.47); educators (4.33), research scientists (4.28), engineers (4.24), accountants (4.22); architects (4.21), home builders (4.05); contractors (3.91), and attorneys (3.60).
Example 1 Analysis
The conclusion was the higher score indicated a more favorable impression. Therefore, engineers were better regarded than certain professions when analyzing averages reported to two decimals.
However, after examining the data using the median rounded to the whole number, all professions except attorneys received a favorable score of four. Remember, no one provided an answer to a decimal place.
Example 1 Design
When reviewing the individual responses, the validity of the line of the questions is also of concern – none of the politicians seemed to dislike any profession, as indicated by few individual scores of 1 or 2. The politicians also seemed to have had a favorable impression of all professions since each profession clustered around 4, or favorable.
The assessment design did not reflect that most politicians are not inclined to degrade any particular profession. To understand how politicians actually feel about a particular profession, different and more layered questions would need to be incorporated into the assessment design.
Example 1 Consequence
A series of action items to improve the scores was errantly developed to improve the profession’s standing. Once again, the survey design was poor and heavily relied on analysis to two decimal places. The leaders of the organization blindly followed the flawed recommendations. The result was over $80,000 of expended resources on a public relations campaign to address problems that did not exist.
Example 2
An international technical services company began conducting a 5-point Likert scale for customer service. The results in all categories were impressive, with customers indicating that they were “satisfied,” or 4.
Example 2 Initial Analysis
The company’s analysis used the mean (average) rounded to two decimal places. An erroneous conclusion was drawn about the top three areas of needed improvement compared to other areas. In reality, rounded to the whole number (or even to one decimal place), there was no significant difference in the top 8 areas of potential improvement. Resources were wasted in the name of continual improvement.
Example 2 Subsequent Analysis
Subsequent surveys were compared to the original survey. Trend analysis was performed to two decimal places rather than properly considering the ordinal data clustered around “4”.
Example 2 Consequences
Conclusions of progress and backsliding were drawn over several years, which the data when properly analyzed did not support. More corporate resources were expended to address areas of perceived risk and needed improvement that did not exist. Only one person questioned the assessment tool’s design, analysis, or administration.
Example 3
A regulatory-mandated management performance survey of a public utility was conducted every five years. The management consulting firm that conducted the survey measured the current and desired states of management practices using a 5-point Likert scale using the mean rounded to 3 decimal places.
Example 3 Analysis
Ten categories were identified as needing improvement for nineteen performance categories related to operations and maintenance practices. After two years of expending resources for improvement and seeing limited results, a third party was retained for a second opinion.
Example 3 Consequences
The third-party review indicated the following.
- The categories of the assessment were consistent with industry practice.
- The assessment design was appropriately done.
- The assessment administration was appropriately done.
- The assessment analysis was poorly done.
When analyzed using clustering techniques and rounding results to the whole number, the areas of needed improvement dropped from ten to seven. The seven areas of improvement were consistent with opinion and evidence, both before and after the initial assessment. The three other areas were de-emphasized to allow resources to be shifted to the seven areas that needed better performance.
Applying It with FINESSE
Qualitative assessments are cost-effective and highly flexible tools. However, using Likert-type questionnaires as a principal evaluation test must be accompanied by the appropriate analysis. Keeping it simple and consistent with good practices for ordinal data is the best approach for facilitators and most technically trained professionals.
Founded by JD Solomon, Communicating with FINESSE is a not-for-profit community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.
This article also appears on www.substack.com.
Leave a Reply