Customer Feedback Surveys Could Be Considered Harmful (easydns.org)

Longtime Slashdot reader Stunt Pope writes: Customer Feedback surveys are now near-ubiquitous, subjecting us all to near-Black Mirror-esque pursuit to "rate your experience" for everything from going to the bank to ordering a pizza. Thanks to The Curse of Goodhart's Law, all of these surveys are beyond useless and even damaging. Mark Jeftovic writes in a blog post: "The shop/hire-rate-reward feedback loop has become baked-in to some systems. Many live marketplaces incorporate these feedback transactions into ratings, which then become a score which then impacts future prospects of whomever is being rated. And that's where the trouble starts. There is a point where this stops being useful and the knock-on effects of a ratings system predicated on feedback results becomes counter-productive. That point is when the ratings become targets. When a company decrees 'All customer feedback ratings must score a minimum of X, or else...' the company has just commenced the process of invalidating and corrupting all useful information to be gleaned from that feedback/survey process. A label which captures this concept is 'Goodhart's Law' -- after economist Charles Goodhart, who posited in essence that 'when a measure becomes a target, it becomes useless.'"

Comments

Most forms of metric are like this

[Zephyn]

" A label which captures this concept is 'Goodhart's Law' -- after economist Charles Goodhart, who posited in essence that 'when a measure becomes a target, it becomes useless.'"

I've seen a similar effect in places where I've worked. A poorly defined metric that is used to rate employee performance will suddenly become the primary focus of the job, instead of actually doing the job.

Re:Most forms of metric are like this

[Dread_ed]

This is how people become hopelessly dissatisfied with their work.

I am in sales and I live and die by a simple axiom that resonates with my internal model of the way the world should be:

If I help enough people get what they want I never have to ask for what I need.

My focus is always on the customer, never on money. My objective is to help others, not myself. Magically this has led to a low 6 figure income in the most unlikely of places. It has been sufficient to support my wife and I and our three children with only me working for 20 years.

I see people who get into sales because they think they are motivated by money. These people become disillusioned when they start earning money and find they are terribly dissatisfied with their life. I relate this to misapprehension of their true motivation. Finding what truly motivates you, what makes your heart sing, what fulfills you and gives meaning to your life, and aligning your life with those principles in a way that generates income is what leads to long term success and happiness.

My $0.02, for what its worth.

I've seen it in action

[npslider]

An experience at a local biz in town with a customer service rep... Was told that anything less than 5 out of 5 on his customer review is considered a bad review, and he all but begged me to give him 5-stars.

He was so overly friendly it was past creepy. I felt conflicted: he did a good job, but I felt I was rating for his sake, not to give an honest assessment of how well I was served by him.

Re:I've seen it in action

[npslider]

The end of my story was that I did not fill out any survey. Life is too short for me to be filling out surveys every time I am asked. I get them from the doctor's office, the car dealership, random items bought online, and my cat.

Re:agreed

[Nkwe]

And as a person filling out surveys who knows (just a little) about math and statistics, I think of ratings on a bell curve. On a 1-10 scale almost nothing is actually a 1 or a 10. On that scale I would rate a 5 as average service and give a 7 or 8 to what I think is well above average service, 9 would be excellent service. You would only get a 10 if there was no possible way to do any better under any circumstances and you completely exceeded all of my expectations. Unfortunately people get dinged if they don't get all 10s. Sucks to be you if I have to fill out your survey.

Overblown story, in my view

[shilly]

It's no surprise to know that targets can be gamed, and that performance metrics can be poorly implemented. But this is a false dichotomy: the choice is not between poorly implemented metrics/targets and no metrics/targets. There's also the option of implementing metrics/targets well. Not perfectly: what is, in this life? But certainly possible to implement them well -- and it would be damaging for the organisation not to do so. And if the behaviours and mindsets of the organisation are broken in the first place, then an absence of metrics/targets can be just as disastrous as poorly implemented metrics/targets -- and what's really needed is effort to work on the underlying issues.

Tell a manager in person

[myid]

If you want to reward someone for doing a good job, you can tell a manager (look for someone walking around the store who looks like a manager), or go to the store's "Customer Service" department, and tell them. Be specific as to how that person was helpful.

Once in a busy pre-Christmas shopping season, a store employee went out of his way to help me. I told a manager, who was walking around the store, how much that employee had helped me. About 1/2 later, the employee rushed up to me all happy. He thanked me for telling the manager how helpful he had been. He said that because of what I'd said, he'd gotten a star (whatever that is) and a bonus.

Selection bias

[nine-times]

I think it's also important to consider that satisfaction surveys tend to suffer from a sort of selection bias. You're only getting feedback from people who feel compelled to give feedback. In my personal experience you'll get:

- Sometimes people who are angry
- Occasional people who are extremely pleased
- Often people who have excessive esteem of their own opinions**
- Rarely people who just want to give helpful feedback

I'm not pointing this out to necessarily disparage these groups or say that their opinions aren't valid, but it's important to understand you're unlikely to get a true random sampling.

**I know someone is going to take issue with my third item, "people who have excessive esteem of their own opinions", so I'll try to explain what I mean by that. Obviously people's opinions are important, or you wouldn't be asking for feedback. And yes, everyone values their opinions more than others'. However, there are some people who... you read their online review, and you can tell that they believe their review will impress everyone and settle all disputes. Like you'll read a negative Yelp review, and the reviewer isn't just saying, "I had an bad experience," or "I didn't like it," but something more like, "This place is simply objectively terrible and though I see other people saying that they like the place, they're all wrong and stupid and not worth listening to." You can almost imagine that they've finished writing the review, leaned back in their chair, and thought, "Well that waiter crossed the wrong person. I expect they'll go out of business any day now."

So my

Re:agreed

[mspohr]

I always give service people a 10 (unless the service really sucked).
They have shitty jobs and the ratings are just another stick to make their lives miserable. I won't play that game. Hopefully they can get a better job some day.