How to Increase the ROI for Your Customer Satisfaction Measurement Efforts (article 2 of 4)
by Service 800, Inc.

Article 2 – How Can I Analyze Customer Satisfaction and Benchmarking Data Like The Pros?

Imagine that you are a college professor. At the end the year, the students in each of your ten classes evaluate your teaching style. Year after year, you’re pleased that you rank well above average for verbal skills, but you continue to place low on the scale for preparation and presentation of materials.

You participate in various skills enhancement programs offered by the university and apply the techniques learned, but the results do not show improvement. In frustration, you seek the help of a friend who is a statistics professor. After carefully studying the data, your associate discovers several problems with how you are analyzing the data.

Learn to properly analyze the satisfaction feedback by asking questions about data collection.

Are Response Rates Consistent?
First, your colleague explains that the sample collected was dramatically different for each class. For example, each class has 250 students. In five of the classes, only 10-15 students volunteer to complete the questionnaire in each class. You pass out questionnaires to everyone in the next three classes, but do not require that they fill it out, so the response rates vary. In the last two classes, all students are required to complete the questionnaire.

Question – Are the response rates consistent and proportionate from each group for which you collect data? If response rates are not consistent from each class, results may become more like ‘general feedback’ rather than actionable data.

Does the Sample Size Yield a 95% (+/- 5%) Confidence Level?
Given the total number of students for the time period you are measuring, it is best to use a sample size that yields a 95% (+/- 5%) confidence level. In other words, a certain number (or sample) of people must respond in order to understand how the total student population ranks your teaching style. And remember, this does not mean that 95% of your students need to complete the questionnaire, just the proper number (or sample) of individuals to reflect the views of the majority of your students.

Question – Will the sample size yield a 95% (+/- 5%) confidence level? If it does not, you may not be able to make confident decisions on ways to make improvements to your teaching style.

Can you clearly identify high and low satisfaction scores?
Once you are certain the data collected is valid, you can make confident decision based on the high and low scores. While it is not recommended that major changes are implemented on the findings of only a few months of data, studying the high and low scores is a good place to start and can lead to immediate change in some cases. For example, you may discover that students rated you low in accessibility outside of the classroom in the third month of data collection, while the first two months of data were more favorable. After a few phone calls to colleagues and students you find out that the new phone menu for the department is confusing and too long.

Question…Am I investigating unusual high and low scores? React to unusually high or low scores by investigating the cause of the problem before making any major changes. If the cause is not apparent, trend the data over time to determine if the score was a one-time occurrence, and review specific comments made to help determine the cause of the low scores.

Are You Trending Data to Over Time to Pinpoint Problems?
When you study the data by morning, mid-day, and night classes by semester, you see that your preparation and presentation skills are only weak in the night classes, during the final semester of the each year. Until this point, findings from night classes appeared slightly lower than other classes, but did not seem significant. By comparing the results for several months, or even an entire year, trends become more apparent and help you pinpoint areas that need improvement.

Question…Am I studying the data over time to determine trends? Looking only at one or two month’s data may not provide you with a full assessment of your teaching strengths and weaknesses.

Do You Pay Attention to Large Swings in Results?
Small data swings may only be a sampling error and may not be significant. You should not rush to employ changes based on small swings. On the contrary, you should question large swings in your results or trends that create large swings in data over time. The first step should be to find out if the change is inside or outside the range of the 95% (+/- 5%) confidence level. Remember you are working from a sample, not the entire population, and it is important to understand what is and what is not an acceptable variance. By taking action on every little down swing or upswing, you may find yourself working in circles.

Question – What is causing large swings in my satisfaction results? Could it be an inappropriate sample size, response rate, or confidence level?

The best way to get the most out of your customer satisfaction data is to ask questions, questions, and more questions before making assumptions or changes.

Next Week - Why is it Important to Study Satisfaction Data for Different Customer Segments?

To Contact Us — To discuss this topic, any other Benchmark Benefits article, or to provide topic suggestions, please contact Jan DeMatteo at jan@service800.com.

For More Information — For additional information about the SSPA SoftwareMetric Customer Satisfaction Benchmark, SERVICE 800, or other benchmark programs, visit www.service800.com/benchmarkprograms.asp.

About SERVICE 800
Founded in 1989, SERVICE 800 designs and administrators real-time customer satisfaction measurement programs, helping service organizations follow up with their customers within hours or days of service events. The company utilizes a distinctive follow up telephone interview process along with e-mail, web, and other survey techniques to measure customer satisfaction. With offices in Minneapolis and London, SERVICE 800 has been measuring customer satisfaction for corporations throughout the world for over a decade.