Slashdot Mirror
The Four Fallacies of IT Metrics
snydeq writes "Advice Line's Bob Lewis discusses an all-too-familiar IT mistake: the use of incidents resolved per analyst per week as a metric for assessing help-desk performance. 'If you managed the help desk in question or worked on it as an analyst, would you resist the temptation to ask every friend you had in the business to call in on a regular basis with easy-to-fix problems? Maybe you would. I'm guessing that if you resisted the temptation, not only would you be the exception, but you'd be the exception most likely to be included in the next round of layoffs,' Lewis writes. 'The fact of the matter is it's a lot easier to get metrics wrong than right, and the damage done from getting them wrong usually exceeds the potential benefit from getting them right.' In other words, when it comes to IT metrics, you get what you measure — that's the risk you take."
It's bad business planning, but it's also the way any big name linux distroy works. Something not working on your Red Hat Linux? No problem, call us! And that's how they make money. They make money on the promise of fixing problems, and that includes saying that their OS is broken.
Losers realize this simple fact, instantly think of several ways to game the metric, then don't do it figuring that "obviously" the decisionmakers realize the metric is horribly broken. Then they get laid off. Winners spend hours, days, or weeks coming up with one way to game the metric, pat themselves on the back for being so clever, and do it. Then they get promoted, eventually to a position where they come up with metrics of their own.
Such metrisc also disincentivize people taking proactive steps to reduce the number of incoming tickets (i.e. making the system/environment more robust or your users more educated), and disincentivizes managers for so doing by reducing the number of people needed to service incoming tickets (thus reducing the size of the empire and the pay grade of the manager).
I've seen both "disincentives" in action. It ain't pretty.
Everybody gets what the majority deserves.
Winners understand that tech support is a stepping stone and treat it as such. Which means that they move up as soon as possible.
Tech support managers are under pressure to keep their costs down. So unless you're okay with working for less money than the others there (but still solving as many problems / answering as many calls) you will be replaced with a new, cheaper person as soon as they can find one.
The metrics are just there to justify replacing you.
Probably the same people who consider number lines of code written per hour as a good metric to evaluate their employees productivity.
Troll is not a replacement for I disagree.
Metrics work if you are comparing two workers on an assembly line doing the exact same work - you can compare their widgets built-per-hour rate (offset by any QA problems).
But when you're dealing with a helpdesk team, the work is no longer homogeneous. The more senior helpdesk person usually gets the hard problems... and he spends more time mentoring his peers (at least he'll do that in a well run team). But tell him that his time-to-resolve metric will determine his bonus and suddenly he'll focus on solving tickets as quickly as possible and instead of volunteering to track down that intermittent printing problem reported by the finance team, he'll leave that for his cohorts and instead will jump on the fast easy tickets.
That's an example of the Dunning-Kruger effect.
http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
Anyone can design a metric that they themselves cannot figure out how to game.
Its not just the losers. Talented and rational technicians and engineers bend to the rules of the system too. Basically you get what you incentivize, what your reward. If you reward people for complying to some metric then they will generally comply. It does not matter what everyone agrees is right, it does matter if management says quality is important. If the metric decides whether you get to keep your job or get that raise then the metric is what the company gets regardless of what the company asks for or whether the company's goals are actually advanced.
Yeah, I used to work tech support for what was then the largest Mac products reseller in the US, and that's the kind of metric they used (just calls per hour, not even resolved issues).
There was one tech so bad that people would just hang up and call back. When asked about my long call times, I showed them a dozen calls from the logs where they talked to 'Hank' for 5 minutes, got back in the queue, and then talked to me for 20-50 minutes (the source phone # was in the logs with destinations and timestamps). I never left a customer with an unresolved problem, but that's not what was being measured.
They did understand that the real waste of money was the guy who had 'great' call times, but they also had no way to measure our actual performance, so they used the reports they did have as proxies.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Good. glad to see that some VP did the smart thing for once and cut the middle managers instead of the people who actually get the work done.
It is deliciously ironic that you would take a swipe at "middle managers" in this conversation about metrics.
The only way to eliminate middle management, is for upper management to utilize metrics in order to evaluate lower management. There is no time for hands-on management and evaluation with a keen eye in one of these vaunted "flat organizations" with no middle management. And so lower management quickly realizes that their jobs and bonuses depend on the metric, rather than on quality or long-ranged action.
After that, the company is humped... but by then, the "aggressive VP" who wiped out middle management has collected his bonus and moved on.
FATMOUSE + YOU = FATMOUSE
on Raymond Chen's site (The old new thing) a sporting goods store wanted to increase the upsales of "shoe protecting" sprays. They offered the staff a kickback since they had a ridiculously high margin on the spray (like 80%). Anyways the sales people were allowed to use discounts at their discrecion and the store had coupons frequntly. So ... smart employees gave away the spray and used coupons or "discounts" to make up the difference so they'd get the kick back. In general any behavior you offer a reward for that isn't exactly what you want as a company will result in you getting what you are incentivizing regardless of whether or not you get what you want out of the deal. The only solution: tie the reward directly to what you want, eg. if you want more profit for the company than give profit sharing. You still have to work hard to remove disincentives, ie crappy employees that make the goal unachievable and so make even your good employees just take it easy since there is no reason to put in the extra effort, but at least you make your employees incentives tied to your organization wide goals.
"Seriously? That's the only way to evaluate? You can't think of a single other way?"
I take umbrage with this sort of response. Your whole purpose is trying to make the parent look stupid, without any sort of constructive input of your own.
By all means, if you know how to perform the job of 10 middle managers with just one top level manager without using metrics, please tell us, otherwise I'm calling your bluff. The parent poster's whole point is that you need hands on management to do proper evaluations. They need to observe you, talk to you, critically evaluate your work, etc. A single top level manager, managing 100 employees can't do that, and if you try to ask the other employees, most decent ones tend to not want to sell out their co-workers.
Metrics was invented for this exact purpose. But a number based on some work stats can't replace critical evaluation.