Slashdot Mirror
Customer Feedback Surveys Could Be Considered Harmful (easydns.org)
Longtime Slashdot reader Stunt Pope writes: Customer Feedback surveys are now near-ubiquitous, subjecting us all to near-Black Mirror-esque pursuit to "rate your experience" for everything from going to the bank to ordering a pizza. Thanks to The Curse of Goodhart's Law, all of these surveys are beyond useless and even damaging. Mark Jeftovic writes in a blog post: "The shop/hire-rate-reward feedback loop has become baked-in to some systems. Many live marketplaces incorporate these feedback transactions into ratings, which then become a score which then impacts future prospects of whomever is being rated. And that's where the trouble starts. There is a point where this stops being useful and the knock-on effects of a ratings system predicated on feedback results becomes counter-productive. That point is when the ratings become targets. When a company decrees 'All customer feedback ratings must score a minimum of X, or else...' the company has just commenced the process of invalidating and corrupting all useful information to be gleaned from that feedback/survey process. A label which captures this concept is 'Goodhart's Law' -- after economist Charles Goodhart, who posited in essence that 'when a measure becomes a target, it becomes useless.'"
How would you rate the quality of this story?
[ ] Ehh, good enough.
[ ] Could have been better, I guess.
" A label which captures this concept is 'Goodhart's Law' -- after economist Charles Goodhart, who posited in essence that 'when a measure becomes a target, it becomes useless.'"
I've seen a similar effect in places where I've worked. A poorly defined metric that is used to rate employee performance will suddenly become the primary focus of the job, instead of actually doing the job.
Would you rate this story a second time if we gave you a 5% off coupon?
There are two types of people in the world: Those who crave closure
Due to the growing abusiveness or corporations, invasions of privacy, and wide spread deceit, I have decided to follow the principles that many corporations expose and purposefully lie in feedback to companies I dislike. They want quality consulting services, well, the fuckers can bloody well pay for them, nothing is for free according to them, for free, they just get lies. Turnabout is fair play after all, lie to me, well I'll lie to you ;) (only for poorly behaved corporations, which seems to be the majority, especially multi-nationals).
Chaos - everything, everywhere, everywhen
An experience at a local biz in town with a customer service rep... Was told that anything less than 5 out of 5 on his customer review is considered a bad review, and he all but begged me to give him 5-stars.
He was so overly friendly it was past creepy. I felt conflicted: he did a good job, but I felt I was rating for his sake, not to give an honest assessment of how well I was served by him.
And as a person filling out surveys who knows (just a little) about math and statistics, I think of ratings on a bell curve. On a 1-10 scale almost nothing is actually a 1 or a 10. On that scale I would rate a 5 as average service and give a 7 or 8 to what I think is well above average service, 9 would be excellent service. You would only get a 10 if there was no possible way to do any better under any circumstances and you completely exceeded all of my expectations. Unfortunately people get dinged if they don't get all 10s. Sucks to be you if I have to fill out your survey.
It's no surprise to know that targets can be gamed, and that performance metrics can be poorly implemented. But this is a false dichotomy: the choice is not between poorly implemented metrics/targets and no metrics/targets. There's also the option of implementing metrics/targets well. Not perfectly: what is, in this life? But certainly possible to implement them well -- and it would be damaging for the organisation not to do so. And if the behaviours and mindsets of the organisation are broken in the first place, then an absence of metrics/targets can be just as disastrous as poorly implemented metrics/targets -- and what's really needed is effort to work on the underlying issues.
I don't generally fill out surveys. When I do, though, I am as honest as they let me be. Sometimes, the pre-provided answers don't conform to my true feelings or are not applicable.
I prefer the freeform surveys where I can state exactly what I like and don't like.
My eyes reflect the stars and a smile lights up my face.
I follow Joshua's advice and rarely, if ever, fill out the "customer feedback survey".
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
It's supposed to demoralize the front-line people. It's supposed to make them hate every second with the customers. This isn't a big revelation. It's applied psychology being used for anti-social ends.
The ends which state that time is money.
For every second you're not selling, you're costing someone money. For every second you're not adding value to that sale, you're costing someone money. For every second you're spending getting to know that other human, you're wasting someone else's money. That's how retail works now. How's it's been working for the past 15 years or so when those got first introduced. It was never about getting someone to better themselves. It was always about manipulation.
Will this work for you? ;)
On a scale of 1 to 10, please rate to the nearest ten-thousandth place, EXACTLY how you felt about your service today:
1. Your Service reps handshake: 7.67565
The reps hand placement was not in perfect alignment with mine, and I detected a slight amount of clamminess on the skin...
2. The quality of the tires you received: 3.14159
The tires were very round, but the font face of the sidewall was less than ideal for viewing at high speeds.
Thank You for your thoughtful response. Please take 5 minutes and complete a free survey regarding your recent Slashdot posting experience!
1. On a scale of 1 to 9.99, 10 being the best, how would rate the grammatical content of that post you replied too?
2. On a scale of noon to midnight how would you rate the timeliness of the post?
3. On a scale from A to Z, how was the speling?
For completing this *FRee survey, you get a sticker!
If it's a phone number based system, just give them Jenny's phone number.
If you want to reward someone for doing a good job, you can tell a manager (look for someone walking around the store who looks like a manager), or go to the store's "Customer Service" department, and tell them. Be specific as to how that person was helpful.
Once in a busy pre-Christmas shopping season, a store employee went out of his way to help me. I told a manager, who was walking around the store, how much that employee had helped me. About 1/2 later, the employee rushed up to me all happy. He thanked me for telling the manager how helpful he had been. He said that because of what I'd said, he'd gotten a star (whatever that is) and a bonus.
I think it's also important to consider that satisfaction surveys tend to suffer from a sort of selection bias. You're only getting feedback from people who feel compelled to give feedback. In my personal experience you'll get:
- Sometimes people who are angry
- Occasional people who are extremely pleased
- Often people who have excessive esteem of their own opinions**
- Rarely people who just want to give helpful feedback
I'm not pointing this out to necessarily disparage these groups or say that their opinions aren't valid, but it's important to understand you're unlikely to get a true random sampling.
**I know someone is going to take issue with my third item, "people who have excessive esteem of their own opinions", so I'll try to explain what I mean by that. Obviously people's opinions are important, or you wouldn't be asking for feedback. And yes, everyone values their opinions more than others'. However, there are some people who... you read their online review, and you can tell that they believe their review will impress everyone and settle all disputes. Like you'll read a negative Yelp review, and the reviewer isn't just saying, "I had an bad experience," or "I didn't like it," but something more like, "This place is simply objectively terrible and though I see other people saying that they like the place, they're all wrong and stupid and not worth listening to." You can almost imagine that they've finished writing the review, leaned back in their chair, and thought, "Well that waiter crossed the wrong person. I expect they'll go out of business any day now."
So my
I also once read that customer satisfaction surveys are, in general, only answered by people who were on the extreme ends of the customer satisfaction ratings; that is, most of the time the only people who bother with a survey are those who had a really awesome or a really bad experience. The experience left them emotionally charged, and they feel the need to share/vent, and the survey gives them that opportunity. The average customer, on the other hand, doesn't have this emotional need and so - when offered an opportunity to rate their experience, gives the whole thing a pass and - if forced into it - tend to give wishy-washy middle-of-the-road answers (e.g., all 5 out of 10s) just so they don't have to think about it.
I imagine it is also easier to be on the receiving end of a "terrible experience" than an awesome one too, which only further biases the reviews. I mean, as a customer I can imagine a dozen ways in which a cashier could piss me off, but have a hard time thinking up a way that cashier could make me EXCITED about paying for my groceries). So the end result is very biased against the person getting reviewed, because most of the people who bother to respond to the reviews are those who had a bad experience. If they forced /all/ customers to take the survey and somehow ensured their honesty, you might get a better overview of how the employee is performing, but by leaving it up to the customer the employer is getting a very unbalanced response.
Not to mention I never know how to answer those because I never know what is considered "average". Is a 5/10 average or do I have to base it on the US grading system, where 7/10 is "average"?
I work in the auto industry. Consider all manufacturer surveys on a logarithmic scale. On a 1 through 10 rating system with 1 being the lowest and 10 being the highest, most dealership operators and manufacturers consider 9 a failing grade for the salesperson. Yes, you read that right. A survey filled out with 9's all the way down will get most salespeople called on the carpet. And if you average a 90% rating for the month you may not work there the next month.
Many salespeople's living wage is ties to bonuses that come from customer satisfaction ratings. You get paid one amount for selling a car. You get paid a higher amount if your average customer satisfaction for the month (or sometimes a 3 month rolling average) is above a certain very high number (like 97%.) Some dealerships tie all performance bonuses to high customer satisfaction scores, easily halving the pay of the salesperson if they get one survey with low scores.
For instance, as a salesperson I have had one customer who scored a survey in the 85% range and due to the circumstances of that month, it cost me over $3500 in lost bonuses. This is not an isolated instance. It happens routinely at dealerships all over the country.
The first time I heard from a manufacturer's rep that customer satisfaction ratings are directly related to not only future sales for the marque but also current resale value I knew it was a sham.
When the only tool you have is a claw hammer every problem starts to look like the back of someone's skull.
Had the same pressure from an Audi salesman a few years ago. Told me I had to rate him a 10.
Well, I didn't because he gave me the usual car salesman sleazy misinformation/ scam dog and pony show as all car salesman are trained to do.
He was pissed.
Last car I bought was a Tesla. No scams. No pressure. Just helped me pick out the options. Best car buying experience ever. (No survey)
I don't read your sig. Why are you reading mine?
I always give service people a 10 (unless the service really sucked).
They have shitty jobs and the ratings are just another stick to make their lives miserable. I won't play that game. Hopefully they can get a better job some day.
I don't read your sig. Why are you reading mine?
I don't think these surveys are worth crap. The companies that rely on them to rate employees have HR policies that suck. I usually just ignore the pleas from "anonymous data collectors" about my visit to see my doctor, or the pizza order (really!). When an employee implores me to rate my visit (for which she is being held responsible) highly, in what might be a personalized way, I just rate the "experience" at 10's from start to finish unless they performed some egregious bit of malpractice on me or my car. It's like tossing a fiver into the hat of a street musician. He needs it and it costs nothing to help him out. I'm hoping that after seeing a few of these the companies will stop asking for my meaningless "feedback". Otherwise, I will continue (with little enthusiasm or optimism) to try to improve the quality of life for those who depend on 10's.
"it cost me over $3500 in lost bonuses."
That's part of the problem. Bonuses, by its very definition, can't be lost, only earned.
Most of these places will look you up by phone number if you tell them you don't have the card with you.
The local area code plus "867-5309" has worked at any place I've ever been. (From the "Jenny" song, from those of us who wouldn't remember it.) And I'm apparently not the only one who knows this. According to my store receipt, Jenny spent over $30,000 at my local grocery store last year.
To fight the war on terror, stop being afraid.