The Four Fallacies of IT Metrics

snydeq writes "Advice Line's Bob Lewis discusses an all-too-familiar IT mistake: the use of incidents resolved per analyst per week as a metric for assessing help-desk performance. 'If you managed the help desk in question or worked on it as an analyst, would you resist the temptation to ask every friend you had in the business to call in on a regular basis with easy-to-fix problems? Maybe you would. I'm guessing that if you resisted the temptation, not only would you be the exception, but you'd be the exception most likely to be included in the next round of layoffs,' Lewis writes. 'The fact of the matter is it's a lot easier to get metrics wrong than right, and the damage done from getting them wrong usually exceeds the potential benefit from getting them right.' In other words, when it comes to IT metrics, you get what you measure — that's the risk you take."

Comments

Business planning

[InsightIn140Bytes]

It's bad business planning, but it's also the way any big name linux distroy works. Something not working on your Red Hat Linux? No problem, call us! And that's how they make money. They make money on the promise of fixing problems, and that includes saying that their OS is broken.

Re:Business planning

[fsckmnky]

SCO was famous for this. $5,000 minimum support contracts, with $1,000 per incident fees, whether they fixed it or not.

"Thank you for calling SCO may I help you ?"

"Yeah, my manufacturing plant just shut down because your kernel panic'd."

"We're sorry to hear that, but you have the newest version, so there are no updates you can apply to resolve the issue. ($1,000 cha ching)"

Re:Business planning

[AdamWill]

Support doesn't just mean 'fixing bugs'. It also means 'helping you set things up right', 'helping you optimize your configuration', 'helping you figure out what tool you need for the job at hand', and so on.

Selling support does not require that the underlying product be broken.

Re:Business planning

[Anonymous Coward]

Flipping burgers ain't bad if you own the restaurant.

Re:Business planning

[justsayin]

My father was an auto and large truck mechanic, pretty good one too. He had three questions you can ask to begin diags on pretty much any system with humans involved.

1) Did it ever work?
2) When did it quit?
3) What have you done to it lately?

Pretty much the foundation of my IT Career right there.

Any metric can be gamed

[russotto]

Losers realize this simple fact, instantly think of several ways to game the metric, then don't do it figuring that "obviously" the decisionmakers realize the metric is horribly broken. Then they get laid off. Winners spend hours, days, or weeks coming up with one way to game the metric, pat themselves on the back for being so clever, and do it. Then they get promoted, eventually to a position where they come up with metrics of their own.

Re:Any metric can be gamed

[Hatta]

Unfortunately, this is true. Evil will always triumph because good is dumb.

Re:Any metric can be gamed

[AdamWill]

Yeah, this is pretty much the problem. Performance evaluation should really be done by crazy, high-tech methods such as you and your peers and manager sitting down and discussing what you've achieved, but that kind of thing is way too hard to stick into an Excel macro, after all...

Another classic example: call centres which measure 'performance' mainly by the average call time metric. Which gives tech support workers all the incentive in the world to give out any piece of bogus advice that'll get the customer to hang up as fast as possible. Or just hang up on them, if the phone system isn't sophisticated enough to detect it.

Re:Any metric can be gamed

[Ethanol-fueled]

As a widget-fixer, our analog is obviously how much shit we can fix in a given time frame. One of the biggest mistakes I've seen in multiple companies(ranging from laptop to medical device repair) is that the PHB keeps a board or chart showing how many widgets each tech fixed during a given timespan.

Any idiot can see that the misguided sweatshop-style metrics cause the following problems:

Cherry-picking - Techs choosing and even stashing away (!) the returns with the easiest and quickest problems to fix. It matters not that your expensive gadget has been sitting there for a month, there are numbers to be made and we'll get to yours when we want to regardless of the order they came in.

Racing - When there are no "easy" ones to be cherry-picked, then the techs will race to fix your item. They will ignore problems and cut corners on others. Stripped screw hole? Super-glue the screw in. Low output? Game the settings so the tests will pass. Part shortage? Cannibalize and rob Peter to pay Paul in a hardware-sort of Ponzi-scheme.
Status Quo and mediocrity - The top performers will become accustomed to the attaboys and will continue to produce slipshod repairs, even if there is a slowdown in work when they can do their job right. Meanwhile, the low performers will become used to it and feel no need to better their work.

My idiot boss in the company I'm in now considered it and was shot down by every tech. In this company, due to the variety of products, one person could make tens of thousands of dollars with 1-2 days work while another tech working on a different product will have to spend more labor and overhead juggling external vendors and all the headaches it involves only to make a couple thousand dollars. Yeah.

Fortunately, the consultants we brought in are smart. They listed generic milestones and a cheeky "100%" as the goal with the smiling disclaimer that it will probably never happen.

Re:Any metric can be gamed

[inviolet]

Losers realize this simple fact, instantly think of several ways to game the metric, then don't do it figuring that "obviously" the decisionmakers realize the metric is horribly broken. Then they get laid off. Winners spend hours, days, or weeks coming up with one way to game the metric, pat themselves on the back for being so clever, and do it. Then they get promoted, eventually to a position where they come up with metrics of their own.

It's not just IT. Our entire society has converted over to metrics. An easy example comes to mind: the stock market versus a company's quarterly performance. Another set of particularly nasty examples is found in our justice system: police officers evaluated by their number of citations, prosecutors by their number of convictions, prisons by their dollars per inmate per day.

I get the financial impetus to switch to metrics. Where it used to be one skilled manager overseeing per 5-7 employees, it can now be one schmuck manager with an Excel spreadsheet overseeing 30 employees.

I even get the psychological impetus. Numbers give us that all-important feeling of certainty, and at low cost too... while the traditional alternative requires legwork, mindwork, judgment, contemplation, and mistakes.

But it's wrecking our society.

ain't pretty.

[Caerdwyn]

Such metrisc also disincentivize people taking proactive steps to reduce the number of incoming tickets (i.e. making the system/environment more robust or your users more educated), and disincentivizes managers for so doing by reducing the number of people needed to service incoming tickets (thus reducing the size of the empire and the pay grade of the manager).

I've seen both "disincentives" in action. It ain't pretty.

Re:Who does this?

[wisnoskij]

Probably the same people who consider number lines of code written per hour as a good metric to evaluate their employees productivity.

You get what you reward

[perpenso]

Its not just the losers. Talented and rational technicians and engineers bend to the rules of the system too. Basically you get what you incentivize, what your reward. If you reward people for complying to some metric then they will generally comply. It does not matter what everyone agrees is right, it does matter if management says quality is important. If the metric decides whether you get to keep your job or get that raise then the metric is what the company gets regardless of what the company asks for or whether the company's goals are actually advanced.