Customer surveys can be conducted using many different technologies. Phone surveys, mail surveys and web surveys are the best known means. A more recent addition to this list is IVR surveys, also known as automated phone surveys. IVR stands for “Interactive Voice Response.” When you call a company for service, your phone call – and those often-annoying phone menus – is managed with an IVR, and IVR technology provides the capability to conduct surveys. In this article I will examine the strengths and weaknesses of the IVR survey approach, and I will use a recent experience with a large consumer retail company to illustrate some points.

What is an IVR survey?
An IVR survey shares many characteristics with a telephone survey since the survey is conducted over the telephone. But rather than have a live interviewer conduct the survey, the survey is delivered using a pre-recorded script. The respondent is asked to enter his responses using the keypad of the telephone and by verbalizing free-form comments. (IVR survey programs can even analyze the tone of the responses.) Since there's no interviewer to clarify questions, the script must be clean and anticipate all the possible responses someone being surveyed may have.

IVR surveys are most commonly used to measure a customer's reactions to service just delivered through a call center environment, known as a transactional survey. When you call a company before being connected to an agent, you may be asked if you would be willing to take a short survey at the end of the call. When the service is complete and the agent disconnects from the caller, the caller is automatically transferred into the survey program. It's important that the agent not know who is going to be surveyed to keep the measurements clean.

IVR surveys need not be limited to this situation. You could invite a respondent group to take an IVR survey by calling them or by emailing them with a toll free number to call to take a survey.

My recent experience with a larger US-based consumer company shows the IVR method's strengths and weaknesses.

The Good:
What are the Strengths of the IVR Survey Method?

Speed of Survey Data Collection: The most striking advantage of the IVR survey method is how quickly a company can collect feedback from an individual. Within 5 minutes after a service interaction is finished, the company can get feedback from the customer about her experiences. No faster means of collecting feedback exists.

Service Recovery: The completed survey from the respondent gets analyzed immediately by the IVR survey program. Management can establish criteria to flag a specific survey response for review. For example, if someone provides a very low rating for overall service quality, information about that service transaction and the subsequent survey can be forwarded to a designated manager.

Lower Survey Response Bias: Response bias results from a respondent's predispositions when they enter into a surveying routine. Common ones are acquiescence where the respondent gives the answer he feels the surveyor wants and auspices where the respondent gives an answer to please the interviewer. Since the respondent is interacting with a machine, the likelihood of any response bias is much reduced. We may not understand the script, but few of us will be intimidated by the process. However, the flip side may prove true should a low score result in an immediate follow-up phone call by a designated manager. That call could prove intimidating to some -- and make them leery to take future IVR surveys.

Lower Cost for Larger Survey Programs: IVR surveys have a cost model that has a high fixed cost, but lower variable cost. The script must be developed and recorded, which can be costly. But the cost to deliver each individual survey is relatively small.

The Bad:
What are the Weaknesses of the IVR Survey Method?

Lack of Anonymity: Like a telephone survey, no anonymity exists. They know who we are. Thus, any survey research purpose that relies heavily on anonymity, such as an employee survey, is best done through some other method.

Keep it Very Simple: Again, like telephone surveys, the questionnaire or script must be kept simple. Complex survey questions that can need to be administered with the visual aid of a printed page or web page cannot use an IVR survey approach.

Narrow Scope: When an IVR survey is conducted as a transactional survey at the end of an interaction with a contact center, e.g., a phone call, the only things that can really be evaluated in the survey is what happened during the immediate interaction. Yet, that interaction is likely one part of a broader transaction, which is composed of a series of interactions.

Telephone Turn-off: Some people just dislike telephone interaction, including telephone surveys. We can thank telemarketers, especially those who use surveys as a pretense for a sales call. Some people also don't like interacting with a machine. And those of us who have grown to hate IVR phone trees -- and I count myself in that group -- may also hate anything that involves an IVR.

In summary, IVR surveys can be a very useful tool to get very quick feedback for transactional surveys and they are flexible enough to be used in other situations. However, care must be taken in the survey questionnaire design, which wasn’t the case with the consumer retailer example reviewed in the next case.

The Ugly:
What NOT to do with IVR survey design (a case study)

You would probably think that mistakes in survey design would be made by small companies with limited resources and knowledge. Yet, some of the best examples of bad survey design I find in big companies. In this section, I'll illustrate the mistakes made in the IVR survey process to measure a popular retail company’s customer satisfaction.
In a nutshell, this IVR customer satisfaction survey:

• Didn't allow me, the customer, to fully express my feelings about my entire transaction with the company
• Forced me to answer irrelevant (to me) survey questions
• Force-fit a survey question to a scale to the point where the data generated are not interpretable (that is, they generate garbage data)
• Asked me for comments -- and then ignored me
• Most importantly, the survey process led to greater customer dissatisfaction and destroyed any customer loyalty I had to this company

The last bullet should startle you. Customer surveys are typically meant to provide customer measurements to increase customer satisfaction. So, how could a survey program design actually lead to greater customer dissatisfaction? The answer lies in a poorly conceived survey project and program design that leads the customer to come away with a more negative attitude toward the company in addition to generating survey data of questionable value.

My Service Interaction

I bought a hot water heater in 1997 from a large US consumer retail company that you have all likely done business with. It had a lifetime warranty. In September of 2007 the heater failed; it cracked. My transaction with this company evolved into the following series of interactions.

1. Claim Initialization: I called the company’s call center to inquire how I would invoke my warranty protection for the failed hot water heater. This interaction was great. They actually had my purchase on record, so I didn't even need my receipt, which I did have. They arranged for a repair tech to come to my house two days later to verify whether the tank failure was covered under warranty. I wasn't thrilled with the 2-day wait, but I had backup hot water capabilities. The agent told me that should the technician verify that the failure was covered by my warranty, then I would not be levied the $69 service charge for the technician visit.

2. Technician Verification of My Claim: The technician came to my house, verified my claim, gave me the paperwork to get a new hot water heater - and charged me the $69 service charge. I figured getting my refund from a company of this stature would be no problem.

3. Fulfillment of my claim: The field tech gave me a form to take to my local store to get my replacement heater. I went to the store on the same day to get my new heater, but they told me the paperwork would take 24 hours to get into the IT system. Why didn't the technician tell me this?

4. Resolving the Improper Charge: When questioning the charge with the field tech at my house, he told me to call the local office rather than call the service center. He said I could talk to a “customer relationship specialist” there. The person with whom I spoke was the most useless, aggravating person I have ever engaged. I called the main service center again and verified that I should not have been charged. They told me to get my refund by working through my local store. I tried unsuccessfully for months. A month after my letter was delivered to headquarters I got a call from a troubleshooter. She told me the charge was legitimate. The service agents at the service center gave me wrong information and the company is not responsible. My jaw dropped.

Soon after the technician's visit, I received an automated phone message asking me to take an IVR survey. The 6 or 7 question survey posed all its questions on a satisfaction scale, where 1 represented Very Dissatisfied and 5 represented Very Satisfied.

What Lessons Can We Learn from this Survey?

Keep the Survey Design Focused. I teach in my Survey Design Workshop to keep a tight focus on the contents of a survey and not let other departments meddle their way into the survey design process. This I believe happened at the company in question. The first 6 questions were about the technician's performance, but the last question, asked about my “satisfaction with the technician making me aware of the company’s products and services.” Clearly, the technician is tasked to promote the company’s products, which is one of those strange roles for service technicians. Survey focus is achieved by having a good Statement of Purpose or Research Objectives -- and sticking to it.

Word Questions to the Scale: The last question of the survey—about the tech making me aware of the company’s products and services—also displays a classic problem in survey question design: wording a question to fit the scale. The question was posed on a 1 to 5 satisfaction scale. I had no idea how to respond. The technician didn't make me aware of any of the company’s products, which satisfied me. So should I have scored that a 5? But I’m sure the marketing folks would want me to score that a 1. I did the cop-out, middle-of-the-road score of 3. I have no doubt, the marketing folks at this company are interpreting the results from this survey question that is simply generating garbage data.

Develop Comprehensive Response Options: That last question really needed a Not Applicable option. I tried just not answering, but the system would not let me do that. So, as said, I entered a 3 hoping for an open-ended comment box.

Develop Comprehensive Question Set: While we want to keep any survey, but especially an IVR transactional survey, short and sweet, you still need to pose all the needed questions. Usually, my criticism of surveys is that they go too far-a-field. This company’s survey was too narrow. The technician gave me factually incomplete information about how to get my replacement heater, and I was charged improperly. Those critical flaws in the transaction went unaddressed in the survey questions.

Think Beyond the Interaction to the Transaction: A service transaction is a chain of interactions, and the weak link needs to be identified. This company ’s survey only asked me about my experience with the technician. My transaction with this company went well beyond the technician's visit, including my interaction with the service center and the store.

Need for a Service Recovery Program: The most disastrous error in this survey example is what they did NOT do with my data. The last question presented the opportunity to make a free-form comment about my experience. I was waiting for that opportunity. I even developed notes to be sure I covered everything. I listed the litany of mistakes that this company made. You would think that the person who analyzes that textual data would flag me for a follow-up call. However, they ignored my cry for help, making me angrier than I already was. Yes, the survey -- designed to measure customer satisfaction and identify needed operational improvements -- actually made the situation worse!

Missed Value in a Service Recovery Act. After the headquarters’ troubleshooter told me to get lost, I started looking at my legal options and initiated an arbitration claim through my state’s attorney general office. Then, 6 weeks after that phone call, a check for $69 arrives. No accompanying letter. No accompanying phone call. Part of the purpose of a service recovery act is to win back the hearts and minds of the customer. The check won back my mind, but my heart? I'm still seething at the process and how I was treated. This company missed an opportunity here to win me back.

Will I ever shop at this company again? Doubtful. Why? I have no confidence that they will make a bad situation right without me having to jump through hoops. But I still don’t understand why they waste the money on the survey program.

About Fred Van Bennekom……………………………………………….
Fred Van Bennekom founded Great Brook to help organizations collect and apply feedback, especially customer feedback learned by support groups. Fred served as an information systems consultant for Digital Equipment Corporation’s field service organization before earning his doctorate from Boston University's School of Management. Fred teaches operations management in Northeastern University’s Executive MBA and Harvard’s Certificate in Management programs, and he conducts survey workshops and projects based on his book, Customer Surveying: A Guidebook for Service Manager.

Please contact Fred to find out what company was reviewed in this case study or with feedback on this article or other topics that you might find interesting. Visit Great Brook’s website by clicking here.