Slashdot Mirror
How Would You Benchmark an IT/IS Department?
ferretworks asks: "Our IS/IT department has been asked by our CEO to find a way to benchmark ourselves against IS/IT departments from other companies with similar technologies (none specific). This sounds like an innocent enough request, but diving into it has made me realize that this is, not necessarily undiscovered country, but a desolate one and rarely visited. So, my poll to the community is: In your Opinion, what is best way to benchmark an IS/IT department and what categories/sub categories would you base your judgment and ratings on?"
I think the first facet you should look for is uptime / accurate data rates (eg. 1% lost data etc). Beyond that, while being nearly a crapshoot, I think the satisfaction that the rest of your company is getting from your department is paramount - perhaps having a anonymous survey given company-wide to see how you're doing. Also, your upper managers may want to hear numbers such as ROI as well as IT costs as a percentage of revenue brought in...maybe even what revenue would be lost without the department.
One metric you could use is downtime/loss of work due to IT. This could be because you don't do backups right, to you don't have a test/production setup, to you upgraded to vista without training first, etc etc. Though you'd be reporting stuff you do badly, you can use this for a lot of justification.
"Email was down about 25 times for up to a day over the last year. This is because we don't have to budget to buy a redundant system. If you give us more money, we can increase uptime." etc etc.
If there are benchmarks, then the terrorists win.
I would start a consulting firm, get the CEO to hire your consulting firm. Spend a lot of time compiling a bunch of numbers, then because the other companies won't want their data revealed by name sort them into averages based on Fortune 20, Fortune 100, Fortune 500.
Make up an "average" for these three sets in which your company does better in most metrics, take the $250,000 you got from this consulting gig and live on it while you go around with your initial report selling it to other companies.
This idea is so stupid and useless that only a consulting firm would offer this service.
did you get that memo?
I would survey the users.
Having worked on both sides of the IT vs Business user / programmer fence, I think any measure of the success of IT needs to include some form of customer satisfaction. This is important because IT's customers tend to be internal and most customer satisfaction queries are focused on external customers.
I've seen battle lines drawn between IT and everyone else, and nobody wins because energies are focused on battling with the other side instead of finding ways to help the company make money. While I understand that IT's responsibility is to the infrastructure, it should be done in a way that makes it easy for their customers (the rest of the company) to do their jobs quickly and efficiently.
If you are being asked to do that, then the PHB's are looking for a cheaper option. Work on your resume and get out NOW. Otherwise, you will training your replacements.
Been there, done that, don't have the t-shirt.
Seriously, they are looking at cheaping labor costs in India/China/Godknowswhat3rdworldhellhole to send your jobs to. If they are asking for a lot of procedures on your work and maps on how it gets done, leave faster.
Just a dude. Stuck in IT.
Operations(IT) isn't like sales. It's not exactly easy to quantify achievements, etc. As long as your teams hit their deadlines within the allotted time frames, your department is doing better than most.
Don't think that a small group of dedicated individuals can't change the world. It's the only thing that ever has.
I got a benchmark for ya. "Are things running well?" Yes? Yes they are? Then shut the fuck up and stop asking me to waste time comparing myself to other people.
(This method of managerial interaction is derived from the groundbreaking book "Shut the fuck up" by Dr Denis Leary)
Seriously, it's not an "innocent" request. It never is. First, they just want to know how you compare. Then, they want to know why you're not the absolute bestest in the whole universe. Then they compare you against departments with fewer people. "Hmm, seems we could get comparable results with 2 fewer bodies". Then each employee is evaluating their specific performance trying to justify their job. Or you're filling out "Time code sheets" to tell them what you do in each 6 minute block of time. Pretty soon you're hearing about their buddy Steve, who has his entire office and web presence run by one part time summer intern who happily works for $6/hour plus tips.
Evaluate your performance based on internal expectations. You want great uptime on the web server? Why, we were only down for 2 hours last month. Compared to previous months, that's great! You want quick response to employee problems? Our average response time for properly filed tickets was down %4 compared to last quarter. That's the way you evaluate. If you start giving these fuckers examples of how other companies are doing, they'll cherry-pick and start challenging you to match. "Why, there's a place in Wisconsin that only has 1 tech covering 400 people! I don't know what this 'Wyse terminal' thing is, but surely if just one guy can cover all those people, we're overstaffed here!"
I know it seems like it could be good. "Why, we're outperforming most other companies, surely they'll see this and use it as criteria to give us better raises/more benefits/better perks". NO. NO, they won't. Just NO. Come review time, you could be leading 90% of the field, and they'll go on about the top 10% and how those guys earn less than you, so they shouldn't give you a raise. Or they'll go on about how other companies have their helpdesk techs do server support too, so they don't need your $80k server admin. If you're underpaid, and everyone else makes more than you, you can show them that too, and they'll nod in an interested fashion, they'll hum and haw, and they'll end up giving you some bullshit excuse about budget constraints. If you make more than other people, expect a pay freeze or cut. Even if fall in the middle of the bell curve, expect them to find a reason to lump you with the folks on the bottom of the curve.
Management doesn't understand you or what you do. They're scared of you. They want to control you and marginalize you so they can eliminate you. Don't let them, don't help them, don't give them any excuse. No matter how much you train them, or explain to them, or draw them little fucking flowcharts in Visio with pretty colours, what you do is still voodoo to them. You push keys on the magic box, typing 10x faster than they can, and you get it to do things they couldn't even dream of, and you do it really easily, and that makes them feel dumb. And just like stupid bullies on the playground, they want to get back at you, subconciously perhaps, but they still do. They want to sit there across from you, at their big fancy desks in their spotless shiny offices, because that's their one time to feel superior to you, to be in control of you, unlike all the other times when you're the magician who just makes miracles happen.
Don't fall for it. Rephrase their request, turn it around on them, avoid it at all costs. Does your accounting department have to compare themselves to other businesses? Do your managers have to go around finding out how other managers in other companies are doing? Fuck no. Don't let them treat you any different than any other department. They're just doing this because they don't understand what you do. Don't explain it to them, just give them numbers based on internal statistics so they can sit back and go "phew, IT is performing well, I'm happy".
Just remember folks, any time anyone in management opens their mouth, echo these words in your head:
"It's a trap!"
If I knew the wedgies I gave you back in 6th grade would have resulted in this . . . I might have taken a moments pause.
(Monitors on Desk * cups of coffer per day) - (downtime of mailserver + vista installs) / (|the temperature of the server room|) = Productivity
....pray
x2 multiplier for Linux install on desktop
x5 multiplier for BSD install on desktop + 1 free very much needed hooker
x10 multiplier for *nix install on managements desktop
Lather rinse repeat for each member of IT department.
Add them together, if the department has somehow manipulated these values to = the number 42, give them all a free car If all of these values are stored in text files encrypted with 4096 bit encryption and spread across a 3x redundant cluster of 10 load balancing servers so that this value can be calculated constantly to the 10 millionth digit....
NewslilySocial News. No lolcats allowed.
Helpdesk (4 employees): Their mainly measured on how many helpdesk "tickets" they can close without sending the problem off to a 2nd level tech. They can reset passwords, add printers, etc.. They can remote also into users desktops to provide assistance. Their resolution rate of about 40%. The remaining tickets are passed up to the 2nd level.
2nd level techs (6 employees): They are expected to close "standard" tickets within 16 business hours from the time the call is taken at helpdesk. They will visit users in their cubes to assist. They're expected to close 95% of their tickets each month within the 16 hour time frame. Anything they cannot fix goes to 3rd level techs.
3rd level techs (4 employees): They get 8 additional hours to close ticket (24 total). They provide final resolution or call the proper vendor for support. They are held to 95% completion rate for non-vendor issue calls.
Telecom (3 employees): They handle call flow, broken phones, fax server, etc.. They have 16 hours to provide resolution from the time call is taken. They also are expected to close 95% of their tickets in that time.
IT Ops (15 employees): They handle AS/400 (logins, reports, administration issues) and backups. Same 95% within 16 hours without vendor assistance.
We also have programmers, database admins, and special system admins. Not sure how they're measured.
About 100 employees in all making up 10% of the company. Here are some other things we measure monthly:
Spam blockage as percentage of total incoming email, viruses stopped/detected, faxes in/out, calls in/out, internet usage as percentage of total pipe ("tube" :-) size, # helpdesk tickets opened vs number resolved by helpdesk/2nd level/3rd level/telecom/ops. Wish I had more but I can't find the metric reports on out Intranet but I hope this post assists a little.
"Trusting every aspect of our lives to a giant computer was the smartest thing we ever did.." Homer Simpson
Why would you do such a thing to yourself? With those metrics, the absolute very best you could do is, 'we didn't cost the company anything...other than salaries and resources.'
Why not use metrics/goals that actually help justify your job? Why not use metrics to show how much IT is adding to the bottom line? Seriously.
BTW, before you go out and waste resources on another system you don't understand, how about learning how to handle what you have? (Hint: It's not downtown; it's 'scheduled maintenance.')
1: Management is clueless as to what the IT staff actually does. 2: Management is looking for ways to reduce headcount.
When you see this nightmare raise its head at your place of employment - be worried. Especially if you're at the top of the pay scale - you'll go first.
If you see this at the same time as there's handwringing about the cost of health benefits, don't just get your resume in order, start looking for another job. Even if your job survives the reorg that's just around the corner, you'll be working twice as hard.
The sign of an incompetent department/person is one who doesn't want to be benchmarked.
... then one day they wonder why they are obsolete/getting offshored/getting downsized and then they bitterly complain about their situation and how unfair it is. If you have ever read the book 'Who moved my cheese', you will recognise this position.
Look at it this way:
If you are better than the rest, you should make sure everyone knows about it and ask for a payrise.
If you are doing worse than others, then you better improve.
It's the latter situation that most people such as the parent end up fearing (and hence ranting about). Simply, they are scared to change their current work practices - they are content doing a mediocre job. They are the sort who spend enormous effort trying to maintain the status quo whilst the rest of the world improves around them
If the CEO's asking you to benchmark your IT group, do not JUST measure your uptime or other simple availability metrics, that's just silly.
Either you're reliable or you're not. But reliability, while important, is only part of the picture.
For struggling, barely competent IT guys, maybe reliability is all they can hope to achieve, but let's assume that's not you and yours. Let's assume you've got reliability down pat. Why do I make this blithe assumption? Because we can purchase reliability off the shelf today for not very much money. Windows XP boxes on programmers desks regularly go 2-3 weeks without requiring a reboot. Servers can run even longer if properly configured, fed and cared for, even running Windows.
So, where do we go from 99.999% reliable with an IT department? We start consulting with our business managers. We work with them as a partner to help select and adopt technology that will make the business more efficient and/or more effective. And we benchmark these efforts by measuring the productivity of the business function before and after the new technology/system is deployed. This is where we show the strategic value of IT. Not in uptime.
And, we take into account cost. If we can get 99.999% reliable and your competitors IT group can get 99.999% reliable, the budget becomes the tiebreaker and the winner is the one that does it for less money.
For instance, if a business is indexing large document sets for the litigation support market and we can figure out how to move that function from Dallas to India thus cutting labor costs by 60% without a loss in quality or responsiveness by cleverly deploying networking technology and low-end PC's into Madras, we have provided an easily quantifiable cost structure improvement which can give our company an important competitive advantage. This is an example of the big payout from IT. The other big one is quality improvement. And the really rare one is opening up a whole new way of doing business...disruptive technology.
I do this for a living, so you'll probably hate me for some of my suggestions....
Number one thing I'd like to get out of the way is that the people who qualify this request as a waste of time are idiots who shouldn't be in an IT department. I've been to too many places where no one can tell me what they're running, where the servers are, what patch levels they are at, which machine is up or down and what they need to keep pace with growing demand (your business IS growing, right?). IT departments like these are the root cause for bloated IT initiatives, downed web servers and crappy customer service.
Number two is the realization that the IT department is in the business of the keeping the rest of the business up and running. This means two things: your responsibility is to keep the rest of the company happy and productive, and you get to charge them (system doesn't matter, as long as you keep track of what it means to service 200 requests a day for lost passwords) when you perform work to keep someones servers up, the mail flowing and the network lit.
Once you understand the place of IT in a business, metrics are easy to come by:
- how well you're doing is measured by how available your servers and your apps are.
- how efficient you are is measured by how much money you save the rest of the business in avoided downtimes and increased productivity.
- Bonus if you keep track of how long it takes to service requests and what their resolution was.
- Extra Bonus if you can do performance forecasting and figure out when you need to expand your infrastructure.
The performance of an IT department is NOT measured by any of the following:
- budgets of IT departments in other companies
- how many new products you've rolled out
- how many people you fired (or hired)
Anybody who suggets those needs to be smacked around and told not to speak in public anymore.
With that in mind, there are a million products to track your statistics. My company will be happy to quote you anything from a 4-figure to a 10-figure deal for this. There are open-source tools that will do similar things for free (though in my opinion, you get what you pay for). There are in-house and hosted answers to any of these questions. But the one thing you need to remember is that you need to know the answers to the CEOs questions about what the status of your machines and your people is. If you can tell him that your hardware uptime runs at 99.99% for mission-critical servers (the Oracle RAC that holds your financial information, for example) and your app availability is 99.7% during business hours, that it takes you 4 hours to respond to a priority 1 request and 3 days to a priority 3 request, and that based on current response time trends, you need to double the processing power and database space in 6 months, you're golden.
If you can't tell your CEO these things, you will be replaced by someone who can. Or even worse, your job will be outsourced to my company, and I get to work out these things while you're flipping burgers. Yeah, I'm being a jerk. That's because I'm flabbergasted at some of the comments (and their moderation) and how their authors still have a job. IT metrics are simple, and don't have to start with complex SLA measurements and other crap. You can start with a basic ping monitor of your critical servers, and go from there. But for heaven's sake, do something. You'll be a hero if you play it right.
Oh, and just to repeat - do not benchmark yourself against other companies. You don't have access to valid data, and you won't find an identical business. Instead, find out what your other departments need from you, and benchmark from there.
Those who can, do. Those who can't, sue.
First off, as many people have commented, you don't want to blindly give management what they are looking for here, but you also don't want to ignore such a situation, as they are probably trying to justify the cost of IT.
You need to turn this around in favor of your department, and it would be a very good idea to take a look at what metrics you can apply that will serve a twofold purpose - to set a baseline of current performance, and to set a moving target for constant improvement. In other words, the things you measure should paint a picture of things that your department can and should improve on, if given the resources to do so. An open ended reporting task like this is a setup from the start, but you need to turn it into a chance to show both how well the department is working with what it has, and how much better it could be working if management would let it. That means finding out where your human resources are being wasted, and making recommendations for refocusing those efforts, finding out what parts of your infrastructure are money sinks, finding support costs that can be reduced with changes, and justifying expenditures that will have a concrete benefit to your department's ability to meet the business needs.
If you don't have them already, now's also a good time to start setting realistic SLAs and tracking compliance with them - for everything from backup/recovery time, recovery time objectives, recovery point objectives, helpdesk call resolution times, server reliability, etc. Just make sure they are realistic, and within reach, and re-evaluate them frequently.
Formal statistical process control methodology such as Six Sigma can be useful in the IT department, but only with the people to make it work, and an organization large enough that it can see enough cost savings to justify having a formal quality team. In order for statistical process control to help, everybody has to be onboard, from management to the lowliest member of the department. If you can achieve this, the rigid define, measure, analyze, improve, control methods of Six Sigma can probably create a savings of cost, labor, and sanity within your department.
I would also look at regulatory compliance as a benchmark. Even if your company is not publicly traded, the tight controls that are necessary for public companies to comply with Sarbanes Oxley (SoX) often lend themselves to better IT practices - formal validation of your IT controls (access to information, physical and logical access to systems, authentication credentials, least privilege, auditing, etc) is a good idea.
Do users have local administrator access to their desktops? If so, why? What applications are requiring it?
Do you have an audited software inventory? If not, what's stopping you?
Do you have controls to prevent unauthorized software installation?
Do you have controls to insure virus scanning and security patching are done appropriately?
Do you have controls to make sure than users have no more access rights than their responsibilities require?
Do you have controls to keep track of why users have the access rights they have, and who approved it?
Measuring your disaster recovery capabilities, and realistic evaluation of the scenarios they have to survive are also a must:
What are your worst single-failure scenarios?
What are your worst two-failure scenarios?
What can be done to mitigate the above?
What has already been done to mitigate the above?
I'd also take a good look to see whether your department is spending it's time putting out fires, or keeping them from starting - if you are just barely keeping up, then it would be good to look at why, and what can be done to change that. Take a good hard look at your helpdesk, and apply the 80/20 principle - roughly 20% of the causes will be responsible for 80% of the calls - what can be changed that will improve the situation there?
From the end user perspective, what issues are most disruptive to users? What issues are hurting the productivity of use
How much money does your inhouse ERP apps bring? How about your wharehouse database that can be used to cut down misplaced inventory? How about your accounting apps that help the CFO find out which areas are hte most profitable?
Forecasting supply/demand apps?
These all make money and help streamline business processes and work with the CEO and his company.
Thats what CEO's want to hear and what a good I.T. department does in a fortune 1000 company.
Not that your high tech janitors like another poster mentioned, which in this case would mean its time to either move to India or update your resume because your a cost center.
http://saveie6.com/
First, all the posters who are saying "It's a trap. If they're delving into this, you're gonna get shafted down the road." are absolutely right. Pay attention to that.
Second, different tasks get measured different ways, so I'm going to concentrate on one aspect, front-line support. That's what I do.
When my upper management got on a kick about assessing skills and measuring performance, my local manager starting doing her best to protect us from the crapola and just let us do our jobs. She's great. Still, I felt the need to pinch-hit for her so at a recent meeting with the big executive who's far enough up the org chart that we almost never actually see her in the flesh, I gave a little speech that went, approximately, like this:
"You want to measure our performance? Fine. For the last several years, you've made a big deal about how we're a support organization and our job is to make sure other folks can do their jobs. So measure that. Assign me a hundred employees (The target user-to-support tech ratio in my organization is currently 113-to-1.) with roughly similar jobs or, for smaller posts of duty, just assign me the entire office. Tell me my job is to keep those people happy. Then randomly and continuously survey those people and ask them if I'm keeping them happy. If I am, I'm doing my job. Nothing else matters. I don't care how many numbers you aggregate off of how many closed trouble tickets and how you compile them and present them in fancy charts and feed them into service delivery models. None of that means anything. All the numbers you're seeing are presented to you by analysts who've never actually been in the field fixing things and dealing with users; you can't trust a thing they say because they don't know what they're talking about. If my customers and my manager think I'm doing a good job, then I'm doing a good job. What I want to know is: Do you have the testicular fortitude to abandon all those meaningless performance metrics and actually start measuring performance by, you know, actual performance? "
Two side comments: My manager is about to give me a perfect performance rating for the year. I know she's going to do that because she had me write the thing. I'm getting it less than a month after that little speech. Also, for you young whippersnappers who think everything in government is bad and anyone who spends more than 6 months on a single job is a loser, I'd like to point out that the speech cited above is a perfect example of why it's great to work under civil service protections and even better to work there long enough that even if management tried to fire you, you could just tell 'me to FO, retire, and take your pension.
I love my job and I do it well; that's why I'm willing to stand up for it. Stand up for yours, if you've got the balls. NOW is the time to start; once management starts measuring the wrong stuff in the wrong ways, you're screwed a dozen ways, none of them involving a happy ending.