Show all articles in this section

Measuring customer satisfaction: how to get reliable results for less cost

AUTHOR: Ujwal Kayande   DATE: 30.11.02   ISSUE 3, 2002

Consider the customer satisfaction measurement methods of a large bank. It has 780 branches and surveys 100 customers per branch per quarter – that’s 78,000 customers per survey at a cost of $500,000.

In an analysis of the bank’s survey data Dr Ujwal Kayande found that if the bank reduced the number of customers surveyed from 100 to 25 it could significantly reduce the cost of its quarterly surveys to $121,000.

He found that while there would be more errors in the measurement of satisfaction because fewer people would be surveyed, the data indicated that reliability would only fall from 0.98 to 0.95 (on a scale of 0 to 1).

The important thing, says Kayande, is to understand the inherent variability of your branches, customers and survey questions in order to get dependable responses. Only then can you achieve cost efficiency by using your understanding of marketplace variability to determine the minimum number of customer responses you need to obtain reliable data.

“If a bank or retail chain is surveying customer satisfaction to use for bench­marking itself against its competition or for benchmarking its branches or retail outlets against one another, there are ways in which you can evaluate whether the survey is doing a good job,” says Kayande.

“Unfortunately, most companies will not try to assess that. Instead, they implicitly assume the data is indicative of the truth whereas, in fact, the data can be subject to a lot of measurement errors,” he says.

What Kayande’s work does is identify the cause and magnitude of errors (or unrelia­bility) in survey data so that companies can ascertain whether the data means something substantial or just represents random numbers.

“If you are providing incentives to your employees and to your branches or retail outlets on the basis of customer responses that have a substantial error component, that are ‘noisy’, then you can expect your incentive scheme, your benchmarking methods to lead to employee dissatisfaction over time,” says Kayande.

What goes wrong?
Benchmarking typically produces data that represents responses to customer satisfaction surveys that ask questions such as: ‘How well do our staff help to make your banking easy?’ or ‘Do you perceive that we treat you with respect?’. “What many companies fail to do,” says Kayande, “is recognise there are typically a lot of errors that come from the fact that customers are not all the same.

“Now marketers might reasonably worry about a marketing lecturer who says the fact that customers are different implies the data is unreliable. However, there is another way to think about it.

“If all customers in the marketplace were identical then the number of customers you would have to ask to get the answers would be equal to exactly one, because if you asked one you’d know everybody’s answer.

“However, the interesting thing about this world is that customers are not all the same, which begs the question, ‘Well, how different are they?’

“If they are all terribly different then you have to ask every one of them a question, but if they are not all terribly different then you can ask just a few and get the answer you want. What I am trying to point out is if they were all the same and you asked only one person, you would have no errors in the data. If they were all very different and you asked only one person, your information would be so unreliable there would be no meaning to it.

“What you have to do is go through a process of matching how different your customers are to how many people you should be surveying. This helps to reduce the unreliability to the greatest extent possible and that’s what my work has been all about,” says Kayande.

But there is another angle to it. It is not just about minimising errors in customer response data.The other part of good survey design is about capturing the inherent variability in the units (the branches or retail outlets) you are benchmarking – what Kayande calls a ‘signal’.

“If you start off with an assumption that these retail outlets are different – in terms of sales and market share and other factors – but your survey data comes back indicating they are not actually different at all in terms of satisfaction, then you have a problem – because your benchmarking isn’t capturing those things that make some outlets better than others on sales or share,” says Kayande. (See Figure 2: Concept of Reliability.)

Asking the right questions
“You have to try to design a survey in which your questions produce the largest possible signal [the best possible capture of the inherent variability between the units being surveyed] and the smallest error or unreli­ability in customers’ responses – that’s the critical matching you have to go through,” says Kayande.

“For example, suppose you have 1000 retail outlets. You randomly select 20 or 50 branches, randomly select customers within each branch to understand how different those customers are, and randomly select 20 questions from a pool of 50 that tap into customers’ satisfaction.

“This random selection allows you to generalise to your 1000 outlets, millions of customers and the 50 questions. You then conduct a survey on these customers to assess how much of a signal this set of 20 questions is giving you.

“If it is giving you a lot of signal relative to the differences between customers then you are doing pretty well. But if it is giving you a small signal relative to the differences between customers, then your survey is doing really badly, and you have to start rethinking your questions.

“For example, if you asked about whether a bank branch is clean or not when you know that most of them are, in fact, pretty clean, then most of these branches would come out with a similar score and what that means is your signal is going to be close to zero.

“On the other hand, if you asked a question about friendliness when you know that few branches are, in fact, very friendly, you are going to find a lot of difference in the data and that means there will be quite a bit of signal coming through,” he says.

Good, or dependable, survey design relies on an understanding that if you start asking questions that generate little or no signal, then you are wasting your time and money. The whole point, says Kayande, of randomly selecting questions from a pool of 50, is that you can determine which questions are good and which are bad – and use the better ones for future surveys. “But, of course, the questions you select all should be conceptually linked to customer satisfaction,” he says.

Achieving efficiency
It is all about efficiency at the end of the day, says Kayande, and “you have to match this thing called reliability – which is how much of a signal there is relative to how much error there is”. (See Figure 1: Reliability and Efficiency).

“You have that, on the one hand, and then you have the cost of doing the survey – which is determined by how many people you survey, how many stores you include and how many questions you ask,” says Kayande.

The trick is to balance these things against each other. If a bench­marking exercise surveys too many customers the implicit assumption is that the differences in customers are large compared to differences between retail outlets (the signal). However, if customers are not very different and a huge number of people are still being surveyed then your benchmarking is not efficient and you are wasting money. On the other hand, if customers are very different and, in addition, very few people are being surveyed, the benchmarking outcome will be unreliable for decision-making.

The importance of random selection
“Random selection is the way you can determine the inherent variability that exists among people, questions, retail outlets and occasions,” says Kayande.

“Suppose you are conducting mystery shopping at petrol stations and your mystery shopper visits the petrol station only in the mornings; that’s a problem because you can’t generalise the results [to your marketplace of customers] because there hasn’t been a spread of measurement across the day.

“If you randomise this then you are much better off because you can generalise you get an understanding of the inherent variability, which you can’t if you specifically choose certain times. Your survey responses should reflect the inherent variability across branches or stores, among people, questions and occasions,” he says.

Survey design checklist
· Understand the purpose of doing a customer satisfaction survey – this is critical. Many companies collect data but don’t know what to do with it.

· Use the data you have already collected to determine what you should do in the future; this will make future data collection efficient.

· Ask questions that find differences and, therefore, give you a better signal (or reflection of the inherent variability of the retail outlets, branches or other units you are surveying).

· Watch out for indications of high unreliability in your benchmarking data. If, over a year, you find that stores drop in and out of being good at customer satisfaction when you would expect a consistent pattern, then you need to reassess the number of customers or range of stores being surveyed, or the type of questions you are asking. Another indicator of data unreliability would be a situation where the stores which do well at customer satisfaction are the ones that have poorer sales.

· Recognise that good survey design will, in most instances, result in a correlation between satisfaction and sales over time.

*Dr Ujwal Kayande is a senior lecturer in marketing at the AGSM. He received the American Marketing Association’s Donald R. Lehmann Award for best dissertation-based article in Journal of Marketing Research or Journal of Marketing (1998), and he was awarded Researcher of the Year by the Australia New Zealand Marketing Academy (ANZMAC) in 2000. He is continuing to work with Australian and US professors on improving survey design methodology.

Footnote
1 Lee J. Cronbach, ‘Co-efficient alpha and the internal structure of tests’, Psychometrika, vol. 16, no. 3, pp. 297—334, 1951.

Further reading
1 Ujwal Kayande and Adam Finn, ‘Unmasking the phantom: a psychometric assessment of mystery shopping’, Journal of Retailing, vol. 75, no. 2, pp. 195—217, 1999.

2 Ujwal Kayande and Adam Finn, ‘Reliability assessment and optimisation of marketing measurement’, Journal of Marketing Research, vol. 34, no. 2, pp. 262—275, 1997.