Slashdot Mirror
How Would You Benchmark an IT/IS Department?
ferretworks asks: "Our IS/IT department has been asked by our CEO to find a way to benchmark ourselves against IS/IT departments from other companies with similar technologies (none specific). This sounds like an innocent enough request, but diving into it has made me realize that this is, not necessarily undiscovered country, but a desolate one and rarely visited. So, my poll to the community is: In your Opinion, what is best way to benchmark an IS/IT department and what categories/sub categories would you base your judgment and ratings on?"
I think the first facet you should look for is uptime / accurate data rates (eg. 1% lost data etc). Beyond that, while being nearly a crapshoot, I think the satisfaction that the rest of your company is getting from your department is paramount - perhaps having a anonymous survey given company-wide to see how you're doing. Also, your upper managers may want to hear numbers such as ROI as well as IT costs as a percentage of revenue brought in...maybe even what revenue would be lost without the department.
One metric you could use is downtime/loss of work due to IT. This could be because you don't do backups right, to you don't have a test/production setup, to you upgraded to vista without training first, etc etc. Though you'd be reporting stuff you do badly, you can use this for a lot of justification.
"Email was down about 25 times for up to a day over the last year. This is because we don't have to budget to buy a redundant system. If you give us more money, we can increase uptime." etc etc.
If there are benchmarks, then the terrorists win.
I think Futuremark has a benchmark for that! Although I hear some IT/IS Department are hiring custom techs just to get high results on it.
I would start a consulting firm, get the CEO to hire your consulting firm. Spend a lot of time compiling a bunch of numbers, then because the other companies won't want their data revealed by name sort them into averages based on Fortune 20, Fortune 100, Fortune 500.
Make up an "average" for these three sets in which your company does better in most metrics, take the $250,000 you got from this consulting gig and live on it while you go around with your initial report selling it to other companies.
This idea is so stupid and useless that only a consulting firm would offer this service.
did you get that memo?
One trend out there is if an IT project doesn't really to appear to have lowered costs, didn't really improve profits, and didn't really improve productivity, what was the over-all point? Was it a failure or is tehre some other measurement of success?
But another measure of success is to measure the increase in Consumer Surplus the IT project/dept adds to the firm. Consumer surplus is one possible way to measure a competitive advantage (because it measures the increased value to your customers).
Estimating consumer surplus is another story though...
Just an idea
http://en.wikipedia.org/wiki/Consumer_surplus
I have never let my schooling interfere with my education.
you will never get any numbers from any other company to benchmark against.
your best bet is to chart your own org over a few years so you can show how much improvement everyone has made. this has the added benefit of being a great weapon to reverse any decisions you didn't agree with or to attack your enemies, just show that the numbers got worse. for example, if you are pissed they outsourced your support department, show that the support resolution numbers tanked. if you hate the network engineer's manager show that his department has been getting worse on a few measurements.
I would survey the users.
Having worked on both sides of the IT vs Business user / programmer fence, I think any measure of the success of IT needs to include some form of customer satisfaction. This is important because IT's customers tend to be internal and most customer satisfaction queries are focused on external customers.
I've seen battle lines drawn between IT and everyone else, and nobody wins because energies are focused on battling with the other side instead of finding ways to help the company make money. While I understand that IT's responsibility is to the infrastructure, it should be done in a way that makes it easy for their customers (the rest of the company) to do their jobs quickly and efficiently.
Take a poll of the people who use the IT department. What do they think?
Other metrics are just silly and are going to have the IT depart trying to do things not their job, which is to make everyone else work more productively.
>> How Would You Benchmark an IT/IS Department?
1. Coversheets per TPS report
2. PHBs per employee
3. Salesmen per server
Higher is better for all metrics.
I'm highly dissatisfied with the IT department in my workplace.
They recently did an anonymous satisfaction survey across the 4000 employees in the company.
The survey was USELESS!
- All of the questions were True/False,
- Many of the questions were not even applicable to me, (but being T/F, I couldn't put a NA answer)
- There was no way to adequately express my dissatisfaction on most issues.
- Many questions were ambiguous.
If a survey is done...
1. Get someone else to write it, NOT just the IT department.
2. Grab some random employees to EVALUATE the survey before it's sent out, and see if it could be improved.
3. When the results have come in, publish them and any conclusions drawn, inviting anyone who disagrees to anonymously comment upon them.
I'm sure my IT department is erroneously patting themselves on the back for their interpretation of the results... don't let it happen to you!
We do consulting for a lot of major companies, and we try to always give reports based on standards such as the ISO 17799 standard for IT management practices and the NIST 800-30 standard for risk assessments. Both of these should give you actionable items to improve your practices and reduce your risk profile.
(Plug: Or we can do this for you, web.interhack.com)
I would use bonnie++, hdparm tends not to flush buffers and so gives inaccurate results.
Oh, wait...
If you are being asked to do that, then the PHB's are looking for a cheaper option. Work on your resume and get out NOW. Otherwise, you will training your replacements.
Been there, done that, don't have the t-shirt.
Seriously, they are looking at cheaping labor costs in India/China/Godknowswhat3rdworldhellhole to send your jobs to. If they are asking for a lot of procedures on your work and maps on how it gets done, leave faster.
Just a dude. Stuck in IT.
Operations(IT) isn't like sales. It's not exactly easy to quantify achievements, etc. As long as your teams hit their deadlines within the allotted time frames, your department is doing better than most.
Don't think that a small group of dedicated individuals can't change the world. It's the only thing that ever has.
The less the IT department is noticed that it takes care of things behind the scenes the better. It should be perfectly fluid. They shall fix problems BEFORE they happen. If problems happen out of the blue there should be fast replacements/workarounds that have minimum impact on users and the entire systems.
And remember EVERY second counts!
If you have a little money,ok more than a little, to come up the results from a third party call Gartner (or another big firm) and ask to participate in a study. They will find other companies your size in and out of your industry to compare you against. They use a common and repeatable criteria that you can use in the future again.
We had one done in the IT department I work in, and while we came in above average in one or two areas it gave us a place to start from in proving we don't just throw money away.
We also directly bill other groups in the company for our services, so they understand where their money goes and how much extra some of there stupid requests cost.
Tracking by uptime, as a lot of other posts describe, simply is not enough. If you want to compare success by uptime, your management might as well just outsource your department to a datacenter company.
A Service Level Agreement can truly monitor your performance as a customer service organization. This idea would measure the soft side of IT support as well as uptime. Also if you do uptime measures, monitor the services, not just ping times. That is simply a waste.
HEAT is a very expensive system but can track service levels, and many open source or cheap packages can monitor uptimes.
A quick google has an example here.
Management always want things like this, unless it is a subsidiary its unlikely that you would get any meaningful data you could compare against in another company. Unless they agree to benchmark following the same rules and be completely honest with you... Who would tell you if they had a bad IT department?
A managers real question is how much money have you saved me or how much easier has my life become because of the IT department.
Someone suggested asking the users, thats not a bad idea... A simple questionaire on the corporate site (internal or external) would be good... Say 10 questions marked on a scale of 1-10, come up with an average of both including and excluding 1's and 10's on the scale. Your ones with low average scores are where you need to improve, high scores you rock and the overall average is where you are. Convince other companies to do the same thing, you have your comparison. Compare your score out of 100. The average without 1's and 10's would be more meaningful, people who put 1's and 10's either aren't thinking about it, or are trying to skew the results. Get the members of the IT department to do the same questions, comparing that might be interesting.
Self evaluation is another management favourite, but i have always hated that... But that would cover downtime or use of budget and things like that... Budget is a bad thing though, show you have saved money and your budget is cut
Have your IT department visit other companies...
I got a benchmark for ya. "Are things running well?" Yes? Yes they are? Then shut the fuck up and stop asking me to waste time comparing myself to other people.
(This method of managerial interaction is derived from the groundbreaking book "Shut the fuck up" by Dr Denis Leary)
Seriously, it's not an "innocent" request. It never is. First, they just want to know how you compare. Then, they want to know why you're not the absolute bestest in the whole universe. Then they compare you against departments with fewer people. "Hmm, seems we could get comparable results with 2 fewer bodies". Then each employee is evaluating their specific performance trying to justify their job. Or you're filling out "Time code sheets" to tell them what you do in each 6 minute block of time. Pretty soon you're hearing about their buddy Steve, who has his entire office and web presence run by one part time summer intern who happily works for $6/hour plus tips.
Evaluate your performance based on internal expectations. You want great uptime on the web server? Why, we were only down for 2 hours last month. Compared to previous months, that's great! You want quick response to employee problems? Our average response time for properly filed tickets was down %4 compared to last quarter. That's the way you evaluate. If you start giving these fuckers examples of how other companies are doing, they'll cherry-pick and start challenging you to match. "Why, there's a place in Wisconsin that only has 1 tech covering 400 people! I don't know what this 'Wyse terminal' thing is, but surely if just one guy can cover all those people, we're overstaffed here!"
I know it seems like it could be good. "Why, we're outperforming most other companies, surely they'll see this and use it as criteria to give us better raises/more benefits/better perks". NO. NO, they won't. Just NO. Come review time, you could be leading 90% of the field, and they'll go on about the top 10% and how those guys earn less than you, so they shouldn't give you a raise. Or they'll go on about how other companies have their helpdesk techs do server support too, so they don't need your $80k server admin. If you're underpaid, and everyone else makes more than you, you can show them that too, and they'll nod in an interested fashion, they'll hum and haw, and they'll end up giving you some bullshit excuse about budget constraints. If you make more than other people, expect a pay freeze or cut. Even if fall in the middle of the bell curve, expect them to find a reason to lump you with the folks on the bottom of the curve.
Management doesn't understand you or what you do. They're scared of you. They want to control you and marginalize you so they can eliminate you. Don't let them, don't help them, don't give them any excuse. No matter how much you train them, or explain to them, or draw them little fucking flowcharts in Visio with pretty colours, what you do is still voodoo to them. You push keys on the magic box, typing 10x faster than they can, and you get it to do things they couldn't even dream of, and you do it really easily, and that makes them feel dumb. And just like stupid bullies on the playground, they want to get back at you, subconciously perhaps, but they still do. They want to sit there across from you, at their big fancy desks in their spotless shiny offices, because that's their one time to feel superior to you, to be in control of you, unlike all the other times when you're the magician who just makes miracles happen.
Don't fall for it. Rephrase their request, turn it around on them, avoid it at all costs. Does your accounting department have to compare themselves to other businesses? Do your managers have to go around finding out how other managers in other companies are doing? Fuck no. Don't let them treat you any different than any other department. They're just doing this because they don't understand what you do. Don't explain it to them, just give them numbers based on internal statistics so they can sit back and go "phew, IT is performing well, I'm happy".
Just remember folks, any time anyone in management opens their mouth, echo these words in your head:
"It's a trap!"
If I knew the wedgies I gave you back in 6th grade would have resulted in this . . . I might have taken a moments pause.
(Monitors on Desk * cups of coffer per day) - (downtime of mailserver + vista installs) / (|the temperature of the server room|) = Productivity
....pray
x2 multiplier for Linux install on desktop
x5 multiplier for BSD install on desktop + 1 free very much needed hooker
x10 multiplier for *nix install on managements desktop
Lather rinse repeat for each member of IT department.
Add them together, if the department has somehow manipulated these values to = the number 42, give them all a free car If all of these values are stored in text files encrypted with 4096 bit encryption and spread across a 3x redundant cluster of 10 load balancing servers so that this value can be calculated constantly to the 10 millionth digit....
NewslilySocial News. No lolcats allowed.
I'd say in order, quality of services, security, and cost.
So, for quality of services, lots of downtime is bad.
But just about as bad are setups that are clunky to use. I've seen some custom apps that are pretty, but if the person has to enter forms, not being able to tab is bad. Cancel buttons that make the app pop up an error box instead of cancelling, and worst of all, do what you have to do to make the app fast. I use some clunky custom cash register app where I work, the database it uses is crap so it's steadily gotten slower over time. Soemtimes now it'll take over 15 seconds to add an item to the receipt, other random times it's seemingly instaneous.
Security. An IT dept. isn't top notch if it's left SSNs or credit card numbers up on a web site even if they're really nice otherwise 8-). Also, I don't think it's so good if there's systems failuers due to viruses etc. Beyond that, it might be hard to tell anything, I suppose most companies are tight-lipped with security statistics. If security goes beyond being secure to being a nuisance, that's a problem, which is why I didn't list security first.
Cost. If several departments have happy users, and good security, then the one that does it cheapest wins. If two departments provide similar service services, but one took 5x as much hardware (per user) to keep everything snappy, then the other department should get props for making things efficent rather than throwing more computers at the problem. This of course also saves electricity and cooling costs. Spending $ billions on an app should put them right out of the running, even if the app ends up being extra nice.
Walk into the IS department and pick a box at random. Tell them to shut it down, disconnect it, whatever. "This unit just died, utterly and completely. How do you recover?"
Where are you getting the data about the other companies? I think that's got to be your starting point. Obviously, if there's not data to be had to compare against you're dealing with an impossible task, and I can't imagine that many companies really publicize that type of stuff. Benchmarking your own team against itself is obviously quite easy as you should have some firm requirements/service level agreements/whatever.
It sounds like your CEO can't think of any new strategic improvements he can make to the company so he's falling back on busy work. Time to look for a company with a future.
Take a look at how mature the strategic alignment between IS/IT and the business is. Unless your business is actually IT, IT and the business usually sit on opposite sides of a fence. It seems to be a reasonable way to rate both IT and the business. http://cais.aisnet.org/articles/default.asp?vol=4& art=14
The more your staff makes the better you are. By paying the highest salaries you attract the most qualified and talented IT staff around. Just tell them it is the free market at work. That's economics, CEO's understand that stuff.
Also explain that another useful metric is the numbers of hours worked. If salaried IT staff are working fewer hours it is because they are operating more efficiently and doing a better job of earning their salaries.
--
edwardaux
You cannot benchmark things that are nstill evolving fast. If you have the good fortune to have some really good IT people on your staff, do anything to keep them happy. They may just save your neck.
Typical example: A really good dystem administrator catches most problems before others notice. Consequentially these people are often sacked by clueless management, because "'there was no problem''. Face it: IT is expensive and still experimantal today. And you absolutely need it to work.
If you really need benchmarks, go for things as desaster recovery capability and speed, incident handling speed and quality, flexibility of the IT staff with regard to using different technologies and ability of the IT staff to make competent technological decisions. If technological decisions are made by the management, then don't bother with benchmarking, since you are doomed anyways.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Helpdesk (4 employees): Their mainly measured on how many helpdesk "tickets" they can close without sending the problem off to a 2nd level tech. They can reset passwords, add printers, etc.. They can remote also into users desktops to provide assistance. Their resolution rate of about 40%. The remaining tickets are passed up to the 2nd level.
2nd level techs (6 employees): They are expected to close "standard" tickets within 16 business hours from the time the call is taken at helpdesk. They will visit users in their cubes to assist. They're expected to close 95% of their tickets each month within the 16 hour time frame. Anything they cannot fix goes to 3rd level techs.
3rd level techs (4 employees): They get 8 additional hours to close ticket (24 total). They provide final resolution or call the proper vendor for support. They are held to 95% completion rate for non-vendor issue calls.
Telecom (3 employees): They handle call flow, broken phones, fax server, etc.. They have 16 hours to provide resolution from the time call is taken. They also are expected to close 95% of their tickets in that time.
IT Ops (15 employees): They handle AS/400 (logins, reports, administration issues) and backups. Same 95% within 16 hours without vendor assistance.
We also have programmers, database admins, and special system admins. Not sure how they're measured.
About 100 employees in all making up 10% of the company. Here are some other things we measure monthly:
Spam blockage as percentage of total incoming email, viruses stopped/detected, faxes in/out, calls in/out, internet usage as percentage of total pipe ("tube" :-) size, # helpdesk tickets opened vs number resolved by helpdesk/2nd level/3rd level/telecom/ops. Wish I had more but I can't find the metric reports on out Intranet but I hope this post assists a little.
"Trusting every aspect of our lives to a giant computer was the smartest thing we ever did.." Homer Simpson
Wow you are really sticking it to the man aren't you? Keep at it, rebel hero!
and then compare yourself to it.
Why would you do such a thing to yourself? With those metrics, the absolute very best you could do is, 'we didn't cost the company anything...other than salaries and resources.'
Why not use metrics/goals that actually help justify your job? Why not use metrics to show how much IT is adding to the bottom line? Seriously.
BTW, before you go out and waste resources on another system you don't understand, how about learning how to handle what you have? (Hint: It's not downtown; it's 'scheduled maintenance.')
Have you ever lost an entire month's output due to a server crash?
/ 02/1215200/
http://hardware.slashdot.org/article.pl?sid=07/05
Seriously, the most gloomy of the responders about how badly your department is about to be hosed are optimists.
A meta-metric on this issue is easy.
Start your stopwatch when you start working on it, and stop it when done.
Do that for the duration of the project then add the total time all up.
File the report under "waste of fucking time"
Last time I checked, RAID arrays don't fail in sympathetically because the one next door did. They do or they don't. Your metric should be "did shit break?" and "did it get fixed fast or rendered irrelevant due to failover"? "Yes/No".
What the guy next door does is irrelevant and trying to compare your stuff with his stuff is middle-management claptrap.
www.strassmann.com has some of the best material around for
considering questions like this.
1: Management is clueless as to what the IT staff actually does. 2: Management is looking for ways to reduce headcount.
When you see this nightmare raise its head at your place of employment - be worried. Especially if you're at the top of the pay scale - you'll go first.
If you see this at the same time as there's handwringing about the cost of health benefits, don't just get your resume in order, start looking for another job. Even if your job survives the reorg that's just around the corner, you'll be working twice as hard.
IT is a service department (something many bigger IT departments forget) and their primary goal should be to keep the customers happy, within obvious budget constraints. If I were benchmarking IT (or HR, or accounting, or any other internal service organization) I would simply benchmark the level of satisfaction by their internal customers.
I think comparing with other companies is a waste of time simply because no two companies have the same requirements or specific networking setup - usually for historical reasons.
Of course, if you want a good company to benchmark against, my company's IT department routinely costs me days for downtime by pushing out broken OS patches, deleting random DLLs from my system directory because THEY don't want it any more (never mind the devs use it) and flagging internal applications as disallowed just because they were too lazy to even to a second or two of research. Oh yeah - they consider themselves a profit center too by charging us internally for the number of PCs we have sitting on our desk, never mind I wouldn't call them for help if they were my last option.
Fear: When you see B8 00 4C CD 21 and know what it means
Comparing your costs to that of another company can be tricky if you provide anything more than commodity services (desktop support, file servers, e-mail, etc.). Does your IT department support any sort of 'enterprise' processes (I can't think of a more descriptive buzzword at the moment? IOW, a business process that your management considers provides your company with a competitive advantage in your market. If so, its going to be difficult to account for additional IT costs to support this and it might not be something your management is willing to give up.
You might need to institute some sort of cost accounting system within IT to track how much is being spent on various categories of support (printers, file servers, etc.) and separate certain key functions (like supporting an e-commerce web site).
Have gnu, will travel.
Use something like Remedy for problem and change tracking. When users report problems, you write and document an audit trail and mark the problem resolution and put the root cause for the issue and the downtime suffered and impact of the issue. Then you can take fun statstics to see your downtime, how important it was, the components in question, and whose problem is was.
Obviously the hard part then is saying when everything's good enough, but knowing most corporations, it never is. There's always something to improve, and when you're done improving, there's new infrastructure to add with it's own bunch of issues. That's a good time to look at your favorite set of "industry standards" to say to management, "See, we're more than good enough. Go away".
The sign of an incompetent department/person is one who doesn't want to be benchmarked.
... then one day they wonder why they are obsolete/getting offshored/getting downsized and then they bitterly complain about their situation and how unfair it is. If you have ever read the book 'Who moved my cheese', you will recognise this position.
Look at it this way:
If you are better than the rest, you should make sure everyone knows about it and ask for a payrise.
If you are doing worse than others, then you better improve.
It's the latter situation that most people such as the parent end up fearing (and hence ranting about). Simply, they are scared to change their current work practices - they are content doing a mediocre job. They are the sort who spend enormous effort trying to maintain the status quo whilst the rest of the world improves around them
If your janitors are working hard all night long, the execs will come in, see how clean the office always is, and wonder why they have the janitors. They will fire them, (or replace them with cheap idiots) and only after the office turns into a filthy whole will they realize their mistake.
If IT is doing its job right, the normal day-to-day users should almost forget its there! Updates and upgrades are smooth and invisible, problems are fixed proactively, instead of reactively, and people will wonder why the hell were even there, since there is never a problem. And as long as were just sitting around just in case a problem ever happens (which to them never does) why not pay some guy $2.00/hour to babysit the computers/networks.
What are we going to do tonight Brain?
If the CEO's asking you to benchmark your IT group, do not JUST measure your uptime or other simple availability metrics, that's just silly.
Either you're reliable or you're not. But reliability, while important, is only part of the picture.
For struggling, barely competent IT guys, maybe reliability is all they can hope to achieve, but let's assume that's not you and yours. Let's assume you've got reliability down pat. Why do I make this blithe assumption? Because we can purchase reliability off the shelf today for not very much money. Windows XP boxes on programmers desks regularly go 2-3 weeks without requiring a reboot. Servers can run even longer if properly configured, fed and cared for, even running Windows.
So, where do we go from 99.999% reliable with an IT department? We start consulting with our business managers. We work with them as a partner to help select and adopt technology that will make the business more efficient and/or more effective. And we benchmark these efforts by measuring the productivity of the business function before and after the new technology/system is deployed. This is where we show the strategic value of IT. Not in uptime.
And, we take into account cost. If we can get 99.999% reliable and your competitors IT group can get 99.999% reliable, the budget becomes the tiebreaker and the winner is the one that does it for less money.
For instance, if a business is indexing large document sets for the litigation support market and we can figure out how to move that function from Dallas to India thus cutting labor costs by 60% without a loss in quality or responsiveness by cleverly deploying networking technology and low-end PC's into Madras, we have provided an easily quantifiable cost structure improvement which can give our company an important competitive advantage. This is an example of the big payout from IT. The other big one is quality improvement. And the really rare one is opening up a whole new way of doing business...disruptive technology.
BogoMips. What else?
How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?
I do this for a living, so you'll probably hate me for some of my suggestions....
Number one thing I'd like to get out of the way is that the people who qualify this request as a waste of time are idiots who shouldn't be in an IT department. I've been to too many places where no one can tell me what they're running, where the servers are, what patch levels they are at, which machine is up or down and what they need to keep pace with growing demand (your business IS growing, right?). IT departments like these are the root cause for bloated IT initiatives, downed web servers and crappy customer service.
Number two is the realization that the IT department is in the business of the keeping the rest of the business up and running. This means two things: your responsibility is to keep the rest of the company happy and productive, and you get to charge them (system doesn't matter, as long as you keep track of what it means to service 200 requests a day for lost passwords) when you perform work to keep someones servers up, the mail flowing and the network lit.
Once you understand the place of IT in a business, metrics are easy to come by:
- how well you're doing is measured by how available your servers and your apps are.
- how efficient you are is measured by how much money you save the rest of the business in avoided downtimes and increased productivity.
- Bonus if you keep track of how long it takes to service requests and what their resolution was.
- Extra Bonus if you can do performance forecasting and figure out when you need to expand your infrastructure.
The performance of an IT department is NOT measured by any of the following:
- budgets of IT departments in other companies
- how many new products you've rolled out
- how many people you fired (or hired)
Anybody who suggets those needs to be smacked around and told not to speak in public anymore.
With that in mind, there are a million products to track your statistics. My company will be happy to quote you anything from a 4-figure to a 10-figure deal for this. There are open-source tools that will do similar things for free (though in my opinion, you get what you pay for). There are in-house and hosted answers to any of these questions. But the one thing you need to remember is that you need to know the answers to the CEOs questions about what the status of your machines and your people is. If you can tell him that your hardware uptime runs at 99.99% for mission-critical servers (the Oracle RAC that holds your financial information, for example) and your app availability is 99.7% during business hours, that it takes you 4 hours to respond to a priority 1 request and 3 days to a priority 3 request, and that based on current response time trends, you need to double the processing power and database space in 6 months, you're golden.
If you can't tell your CEO these things, you will be replaced by someone who can. Or even worse, your job will be outsourced to my company, and I get to work out these things while you're flipping burgers. Yeah, I'm being a jerk. That's because I'm flabbergasted at some of the comments (and their moderation) and how their authors still have a job. IT metrics are simple, and don't have to start with complex SLA measurements and other crap. You can start with a basic ping monitor of your critical servers, and go from there. But for heaven's sake, do something. You'll be a hero if you play it right.
Oh, and just to repeat - do not benchmark yourself against other companies. You don't have access to valid data, and you won't find an identical business. Instead, find out what your other departments need from you, and benchmark from there.
Those who can, do. Those who can't, sue.
I would have thought that the request is fairly bogud to begin with. You would have to find a number of companies which mirror your size, profit, market, technology, payroll etc.. to be able to develop any comparative metrics that might mean anything.
First problem is how you would find enough information about the prospective companies to determine that they could be used in a represntative comparison. You might be able to get some of this from financial reports, but they usually are doctored a bit and might not include enough depth of detail.
second problem is how you would get this other company to provide ongoing data so that you could tune the metrics and compare your company to them anyway.
'Hi, this is Jim from ACME inc, you're competitors. Would you mind giving me a copy of all the financials and support documents pertaining to you IT department so that we can compare ourselves to you? We won't use it for competitive advantage, honest!'
My rule of thumb, never trust management that wants you to define your own KPIs. Actually never trust a management that uses KPIs, full stop.
use one that makes you look the best
First off, as many people have commented, you don't want to blindly give management what they are looking for here, but you also don't want to ignore such a situation, as they are probably trying to justify the cost of IT.
You need to turn this around in favor of your department, and it would be a very good idea to take a look at what metrics you can apply that will serve a twofold purpose - to set a baseline of current performance, and to set a moving target for constant improvement. In other words, the things you measure should paint a picture of things that your department can and should improve on, if given the resources to do so. An open ended reporting task like this is a setup from the start, but you need to turn it into a chance to show both how well the department is working with what it has, and how much better it could be working if management would let it. That means finding out where your human resources are being wasted, and making recommendations for refocusing those efforts, finding out what parts of your infrastructure are money sinks, finding support costs that can be reduced with changes, and justifying expenditures that will have a concrete benefit to your department's ability to meet the business needs.
If you don't have them already, now's also a good time to start setting realistic SLAs and tracking compliance with them - for everything from backup/recovery time, recovery time objectives, recovery point objectives, helpdesk call resolution times, server reliability, etc. Just make sure they are realistic, and within reach, and re-evaluate them frequently.
Formal statistical process control methodology such as Six Sigma can be useful in the IT department, but only with the people to make it work, and an organization large enough that it can see enough cost savings to justify having a formal quality team. In order for statistical process control to help, everybody has to be onboard, from management to the lowliest member of the department. If you can achieve this, the rigid define, measure, analyze, improve, control methods of Six Sigma can probably create a savings of cost, labor, and sanity within your department.
I would also look at regulatory compliance as a benchmark. Even if your company is not publicly traded, the tight controls that are necessary for public companies to comply with Sarbanes Oxley (SoX) often lend themselves to better IT practices - formal validation of your IT controls (access to information, physical and logical access to systems, authentication credentials, least privilege, auditing, etc) is a good idea.
Do users have local administrator access to their desktops? If so, why? What applications are requiring it?
Do you have an audited software inventory? If not, what's stopping you?
Do you have controls to prevent unauthorized software installation?
Do you have controls to insure virus scanning and security patching are done appropriately?
Do you have controls to make sure than users have no more access rights than their responsibilities require?
Do you have controls to keep track of why users have the access rights they have, and who approved it?
Measuring your disaster recovery capabilities, and realistic evaluation of the scenarios they have to survive are also a must:
What are your worst single-failure scenarios?
What are your worst two-failure scenarios?
What can be done to mitigate the above?
What has already been done to mitigate the above?
I'd also take a good look to see whether your department is spending it's time putting out fires, or keeping them from starting - if you are just barely keeping up, then it would be good to look at why, and what can be done to change that. Take a good hard look at your helpdesk, and apply the 80/20 principle - roughly 20% of the causes will be responsible for 80% of the calls - what can be changed that will improve the situation there?
From the end user perspective, what issues are most disruptive to users? What issues are hurting the productivity of use
How much money does your inhouse ERP apps bring? How about your wharehouse database that can be used to cut down misplaced inventory? How about your accounting apps that help the CFO find out which areas are hte most profitable?
Forecasting supply/demand apps?
These all make money and help streamline business processes and work with the CEO and his company.
Thats what CEO's want to hear and what a good I.T. department does in a fortune 1000 company.
Not that your high tech janitors like another poster mentioned, which in this case would mean its time to either move to India or update your resume because your a cost center.
http://saveie6.com/
1.) Make a random survey, asking insightful questions 2.) Have several emails go around the company stating how important it is for everybodies voice to be heard 3.) Pull metrics that make you look good out of your ass and use those instead of the answers 4.) profit!!
People who think they know everything are a great annoyance to those of us who do.
First of all what is your SLA? You should measure yourself firstly against the SLA in place. If you are way out you need to figure our why and/or maybe adjust the SLA. Do you offer a standard 9x5x5 (9am to 5am 5 days a week)? You need to have a clear cut definition of what services and at what level you will supply those services. Does your company outsource code infrastructure such as core network and telephony to another company? If so how does their SLA measure up to yours? Your SLA should be lower their your suppliers for obvious reasons.
How many 9s do you provide? In all honesty anything 97% or above should be acceptable to standard office users. Along with allowed scheduled downtime during office hours (such as lunch, first or last hour of the day) it should be fine. For core services this can be difficult (such as email) however you can resolve this in many ways most of which are rather simple (although require more hardware/software).
What is your DR plan? In case of fire/flood can you get your hands on the off site backups in 24 hours? Can you have a fully operational network with all data restored in 48-72 hours? These are realistic numbers for most businesses.
Other areas you should measure yourself on are how well your support your company with new technology? Have you looked into things such as server and storage virtualization? What are your storage technologies? SAN based? What backup systems do you use? Are they the most efficient for your business? I am not saying you have to use the latest greatest but you should know about them, their pros and cons, why they are/are not suitable for your business (And money is a perfectly acceptable answer).
What are your infrastructure forecasts? Things such as storage, CPU power, accessibility (internal and external, VPN?). In 1 year how much additional storage will you require? Can your current solution support this? Will you need to purchase a new system? Can your backup solution cope with this growth? Will you need a bigger tape library? Will this effect your scheduled downtime? What about DR numbers?
Providing you can answer all these questions you can consider yourself in good shape. Don't measure yourself just on uptime. Uptime is just one part of a much bigger picture, yes it's important but there are many other bits just as important.
...especially the part where you lead off by slamming the people who don't like the request, then join them a few paragraphs later. :)
And to add to the metrics, long-term bonus if you can report (when applicable) what chronic problems/issues you've resolved (or at least handled). Basically, it's an extension of metric #2 in the parent's list, just be aware that self-analysis can bring some large-scale benefits over the medium- or long- term.
Everything else is more or less made up -- you can calculate whatever stats make you look good.
The real test?
Two words: sack race.
You might want to look into the IT Infrastructure Library - http://www.itil.org/
;}
Best practices, key performance indicators, you name it
I have to answer with a question -- by benchmark, do you mean 1) comparing yourself to other IT departments or 2) developing a system of measurement that proves you are effective and getting better.
m /
I prefer the second approach -- develop an internal system of performance measurement. This is not a trivial exercise but it has been done. I suggest your get you CIO a copy of Martin Curley's book, "Managing Information Technology for Business Value." Here is a link to Intel Press http://www.intel.com/intelpress/offers/bundle5.ht
"If all the American people want is security, let them live in prisons." Eisenhower
Do they keep your shit fixed?
Steve's Computer Service, Hobbs, NM
Sit down all the sysadmins and techs on one long bench. Measure the amount of, erm, 'parking room' needed. Do this at the beginning and end of the test period.
This 'parking room' should increase as they become more efficient, automate more processes, and get off their buns less often.
The same method can be used to downsize the department as efficiencies grow; after a while, they'll have less and less room on the bench. Kind of like musical chairs.*
Note: This could have been written in the '1. 2. 3. Profit!' form, but I was too lazy.
*Except the skinniest one would be squeezed out by the 'most efficient'. Evolution inaction.
PS: The captcha text for this post was 'largely'...
I can think of a lot of ways to do it. The best practice (yeah, guess what...I'm an IT consultant) is to look at the ITIL library, and ITSM in general. ITSM is essentially a set of practices around helping business units tell IT what they want, in useful terms, and measuring the definition of success in how IT fulfills those wants/needs. ITIL is a library of information that relates to ITSM.
But even if you get all that working at your end (and it's not a tiny effort), how are you going to possibly compare against other organizations? How are you going to get information like that about what I presume are competitors? That's the big problem that I see, and I can't think of a way around it.
For your security, this post has been encrypted with ROT-13, twice.
Mr. Ballmer, you forgot to sign in again.
First, all the posters who are saying "It's a trap. If they're delving into this, you're gonna get shafted down the road." are absolutely right. Pay attention to that.
Second, different tasks get measured different ways, so I'm going to concentrate on one aspect, front-line support. That's what I do.
When my upper management got on a kick about assessing skills and measuring performance, my local manager starting doing her best to protect us from the crapola and just let us do our jobs. She's great. Still, I felt the need to pinch-hit for her so at a recent meeting with the big executive who's far enough up the org chart that we almost never actually see her in the flesh, I gave a little speech that went, approximately, like this:
"You want to measure our performance? Fine. For the last several years, you've made a big deal about how we're a support organization and our job is to make sure other folks can do their jobs. So measure that. Assign me a hundred employees (The target user-to-support tech ratio in my organization is currently 113-to-1.) with roughly similar jobs or, for smaller posts of duty, just assign me the entire office. Tell me my job is to keep those people happy. Then randomly and continuously survey those people and ask them if I'm keeping them happy. If I am, I'm doing my job. Nothing else matters. I don't care how many numbers you aggregate off of how many closed trouble tickets and how you compile them and present them in fancy charts and feed them into service delivery models. None of that means anything. All the numbers you're seeing are presented to you by analysts who've never actually been in the field fixing things and dealing with users; you can't trust a thing they say because they don't know what they're talking about. If my customers and my manager think I'm doing a good job, then I'm doing a good job. What I want to know is: Do you have the testicular fortitude to abandon all those meaningless performance metrics and actually start measuring performance by, you know, actual performance? "
Two side comments: My manager is about to give me a perfect performance rating for the year. I know she's going to do that because she had me write the thing. I'm getting it less than a month after that little speech. Also, for you young whippersnappers who think everything in government is bad and anyone who spends more than 6 months on a single job is a loser, I'd like to point out that the speech cited above is a perfect example of why it's great to work under civil service protections and even better to work there long enough that even if management tried to fire you, you could just tell 'me to FO, retire, and take your pension.
I love my job and I do it well; that's why I'm willing to stand up for it. Stand up for yours, if you've got the balls. NOW is the time to start; once management starts measuring the wrong stuff in the wrong ways, you're screwed a dozen ways, none of them involving a happy ending.
Everyone talks about benchmarking or comparing themselves to others. The first step in this scenario is being able to communicate what your organization does, then how well it does it. If you can't do that... forget benchmarking, and forget being able to validate your' IT organization's value.
It is most important to first look at the work being done. Even if your processes/activities aren't formalized, every organization has elements of problem management, change management, monitoring, inventory management. However, it is critical to recognize that many organizations think of the work very differently. Do you do release management? If so, are you in development or operations? Because release management may mean packaging a newly developed application for release into the environment (development views it this way), or it may mean performing all the functions necessary (e.g., reviewing support capabilities, evaluating risk, performing load balancing) to actually drop the new application into the environment (operations views it this way). Does your organization think of asset management in conjunction with the configuration of the environment and IT finances or does it simply view asset management as inventory management?
Once you identify the work, start to look who are responsible for those functions and what tools they are using. That will tell you how the work is being performed. Are there too many work overlaps? Are work hand-offs not covered? Do we have too many tools performing similar functions, or are we missing some automation opportunities? This information can be easily captured at the task level, as long as you provide those in your organization with a high-level example of each of the work areas.
Now that you have the work being performed identified and the mix of people and automation necessary to make it function, you are ready to consistently capture the performance necessary for benchmarking. It is true, there are no two 'identical' companies and even within similar industries, organizations are quite different. As such, apples-to-apples comparisons are rare. However, many valuable comparisons can be made. A number of the previous posts made some good suggestions regarding performance measures and not creating SLAs unless they are necessary and unless you can validate the numbers promised. You don't need a lot of metrics. In fact, the metrics you capture will change as your organization matures. Start measuring the things you are capable of measuring. As you mature you will grow into identifying those measures that are relevant to the processes, the business, and your regulatory requirements.
Once you are capable of capturing and communicating what you do and how well you do it, then you are ready to compare against others that can do the same. The company we used for IT process benchmarking did it this way and it worked well for us.
The sign of an incompetent department/person is one who doesn't want to be benchmarked.
But the act of benchmarking takes TIME. And ENERGY.
And the last thing that people want to part with - people who are doing useful, productive, mission critical work - is time. Or energy.
Bean counters & suits have plenty of time and energy to devote to nonsense like this. People who actually do something productive with their days do not.
Consider classical computer coding.
Say that writing the code to satisfy some architectural requirement takes X units of time and Y units of energy.
Now suppose that someone has decided [perhaps correctly] that all code must be commented. Guess what? X just became 2X, and Y just became 2Y.
Now suppose that some PHB comes along and says that all work must be "benchmarked". Well guess what? That means that both the code writing and the code commenting must be "benchmarked", ergo the original X becomes 4X [and the original Y becomes 4Y].
And then suppose some little smart ass in the back of the room says, "Oh yeah, well if everything must be benchmarked, then that means that the benchmarking must be benchmarked!"
La voila - the original X units of time are now 8X, and the original Y units of energy are now 8X [and then when the benchmarking of the benchmarking must be benchmarked, we're at 16X & 16Y, & cetera ad infinitum...]
Ever watch "Office Space"? Rememeber the TPS Reports?
At some point, you just have to draw the line, and say, "No Más!"
the easy way out would be to take on a real gung-ho attitude about finding out how well you measure up to other companies. once you undertake this new initiative, start by disabling all spam filters and network appliances. when the CEO, and everyone else starts complaining about dirty emails and the 4 hours a day spent cleaning out their inboxes, explain to him/her that maintaining the technology that (presumably) supports your business is a full-time job, and that if everything is actually working well, he/she should get off your back/give you a raise. (recall the scene in old school where l.wilson tells off his boss, john locke because he's already gotten it done)
course it's not so much the easy way as the ass-hole way but you get the idea.
not only is time travel possible, it's irrelevant.
I work in a company, where I am the IT dept. All the servers(15), all the users(60), the entire network and even some old press consoles are all my responsibility. Now think about if my company decided to have a survey amongst our users about my performance, and their satisfaction. Either I'm fired, or damn it, i deserve a raise.. depending upon the outcome of the survey. The primary issue I have with keep downtime to a minimum, and keeping users happy and doing their jobs is money. I'm constantly applying bandaids to hemorrhaging systems. What do you do when management has a "if it still sorta works, we're not spending a cent to fix it" attitude. Or when it does break, "what's the cheapest, buy that", regardless of the inevitable issues of subpar equipment and software. So in my case I'm not sure downtime could be used as an adequate measure of a successful IT dept. Give me a bag full of cash to spend, and I'll make sure we have superior redundancy and mere seconds of downtime in any situation.
in these discussions. I'd look to the business functions themselves, that you support. Each area of the business has that has some IT function supporting it has some type of metric for what it needs. This could be, how long does it take to get a new user up, what is the uptime/response time of the ERP system, is there available space for the document store, etc. You can then define performance metrics, either arithmetic or qualitative, that the user of the service can understand as being meaningful metrics of whether you are serving them well. These can then be used to review the IT team's performance in a meaningful way, and used to quantify the need for additional staffing/equipment.
At the risk of sounding like an MBA type, this is fairly simple management by objective thinking. At the end of the day, what the boss cares about first is, is you business running well. Only after the business is running well, can you consider reducing cost. If you have gone through this process, and you have these metrics in place, then a degradation of service due to a staff reduciton will be very clear, and likely fought by the users.
I was taught to respect my elders. The trouble is, it's getting harder and harder to find some.
I'm currently involved with benchmarking a county government IT department. There has been a change in the leadership, and we now have a boss who cares about using technology effectively. Here are some things we are looking at:
Do employees have access to reasonably modern technology? Answer: probably not. We're still using a 10-year old version of lotus notes, and we don't have any standard procedure for deciding who gets the new desktops.
Can the different department computer systems work together? No. Because of state law, each of Public Health, Mental Health, and Social Services has to use a specific, outdated database for record keeping. We are lobbying for permission to combine them into a single database. One of our goals it that any nurse or social worker on a home visit will have a TabletPC they can use to access all information about that household's problems.
Is the helpdesk helpful? More or less. Most problems are eventually resolved. But there have been some rather obvious gaffes. "Ivan the Terrible" became a very nice and helpful person the moment he was moved out of helpdesk duty, and is now a benefit to the dapartment. He should never have been put on helpdesk duty.
We've also been looking for ways to make better use of our programming staff for developing custom apps. Not many managers know about that resource, and fewer still know how to use them well.
Despite my statements elsewhere under this topic, I agree with most of what you say. We measure all that stuff. For example, we answer priority 1 requests within an hour and every hour one goes unresolved results in a personal, automatic memo to the next higher level of management. Leave a priority 1 open and unresolved for 5 hours and it's literally on the immediate to-do list of the CIO. One more hour and the Commissioner gets involved. Priority 3, an individual work stoppage, is addressed in 4 hours. Our success in meeting those deadlines, at least in my work area, exceeds 98%. All those numbers are readily available and continuously updated. However, it's easy to measure things where you control the entire process and can define the beginning, severity, and conclusion of a problem.
Much work, though, falls outside those parameters.
For example, you want to measure "avoided downtimes." Good idea. But despite years of negotiations, in my large organization we've never even successfully defined terms. Currently, I'll help someone who deleted all their saved email. I get a restore done remotely and get back to the user and close the ticket. The app was available the whole time and they could process new mail. They had plenty of other work to do so it wasn't a work stoppage. I'll report my "component not available" time as zero.
The user, OTOH, will be frustrated because he couldn't get to that old email telling him his responsibilities for planning the big boss' retirement party next week and since that was his biggest priority for the day, he reports 8 hours of downtime for the day, even though he knows he actually did spend as much time as during a normal day doing productive work.
At the yearly SLA re-negotiation, both sides sit down with radically different lost productivity figures. In their way, both sides are right and wrong simultaneously. Both sides definitely don't trust each other. Nobody every feels their expectations are being met.
How do you get over a hump like that? The problem really seems intractable. Can you recommend any reference material for me to read?
Thanks for your thoughts.
My customer base includes diverse clients and verticals, from Fortune 50 to SMB. Most if not all larger companies are currently wrestling with how to gauge overall IT performance. This is, in fact, one of the newest trends that is going to overtake much of the IT industry over the next few years, in my opinion.
However, there are other factors which should be looked at and different models for applying and measuring performance that should be considered before you simply go out and compare yourself to other companies.
One of the industry drivers that I have seen is the move to start treating IT as a real business within a business, and creating Key Performance Indicators and Benchmarks to measure the actual value of IT to the overall business. Historically, IT has been the dumping ground of the company. Every company has a product, whether that is a physical item or information or time. IT has been asked to supply some (and in many cases all) of the infrastructure to support these products, yet they are often not involved with much of the decision making process in developing the solutions. A COO or CSO or CEO makes a decision to sell something. They go to the CIO and ask, "How much will your portion of delivering X cost?", develop a budget, and then at some point equipment shows up and IT installs it and maintains it.
The key issue, however, is far more complicated. No IT department, even for a small company, is delivering a single service. Even a 1-man IT shop in a 10 person company is being asked to develop, purchase, install, and maintain multiple products. For this theoretical small company, at the minimum the IT guy is doing email, networking, web site maintenance, and supporting information stored on computers and servers.
The key is to begin treating these services as products, looking at the bigger picture, and then analyzing many well-established benchmarks against these services.
Right now, Service Delivery is really what IT is about.
Many companies and vendors that I work with are now starting to look at IT in this manner, and have come to the proper conclusion that a single corporate IT infrastructure is a business with in the business, and that it is possible to measure value and performance, as well as establish best-practices, benchmarks, and even future budgets against these benchmarks.
For example, one of my customers is in the information business. They sell access to up-to-the minutes (and in some cases, up-to-the-second) access to information. They currently sell hundreds of products, and until this point, each products was essentially its own entity, with its own supporting groups, and global IT infrastructure supporting these products.
Up until now, IT was simply told, "Here's a product we have developed. Our Global Product Manager is going to deliver this data to these customers. Our Development Team (with no common management short of the CIO) has used this architecture, this platform, and expects we will need X number of Y products (Servers, storage, network gear, etc) to deliver this service. Make it happen."
But think about that. They have at least 15 hardware platforms that I know of. They support at least 7 different OSes. The network group has no visibility into this process, and ends up adding blades and wiring and routers and everything else on virtually an ad-hoc basis.
So, what is the solution?
SOA. Service Oriented Architecture. This is, of course, the newest buzzword, but to me, it makes so much sense I wonder why no one thought of it sooner.
The basic gist is that you treat your IT infrastructure as a single architecture. Then you analyze each product on that architecture, and also begin analyzing what you really need to support Service Delivery of those products. In this case, you have the hundreds of client products. You also have the internal networks for the company, which deliver email, instant messaging, Internet access, database access (generally Oracle applications for C
Excellent points. Really excellent points. You made me stop for a minute and consider just how fortunate I am. And, also, how much I need to add a little more detail so that people don't get the wrong idea.
No, I can be fired for doing something outrageous. I can be escorted right out the door and lose my pension. But I can't be fired for pissing off some exec by saying things I believe to be true. That's a big deal and a huge advantage of public service over private industry. (After all, I need something to compensate me for the lower pay rate - but that's a whole 'nother can of worms.)
It's nice to have an emergency exit strategy.
I didn't tell the exec to FO. I told her to put her measurement money where her customer service mouth was.
I'm an old man. My attitude has bitten me in the ass more times than I can count. The saying around here is that my mouth is "career-limiting." I accept that and have for a long time.
That being said, I'm also straightforward, honest, and excruciatingly polite (except when I make a conscious decision to be otherwise). I am always the first to publicly admit when I make a mistake. About a number of topics, I'm the go-to guy for tech help and when I don't know the answer, I know where to go to get them. I can quickly think of the names of two dozen customers who will literally specify in their trouble tickets when they call the help desk that they want me and only me to come work on their computers because they know I really care if they get back to work.
To me, this isn't a job; it's serving my fellow man and a greater good. I'm willing to shoot off my mouth if that's what it takes to make this a better place. Why? Because I love it and I want to stick around for quite a while longer.
That was the point. We spent over an hour behind closed doors afterwards.
This was an exec high enough up that she could do something about it. For example, I knew that she was the primary architect of one of our support systems that I openly disparaged during the meeting. Later, she demanded to know what problem I had with it. You see what I gained there? I got her, an exec so far up the ladder it would give me a nosebleed to climb that high, to ask lowly little me a real question. I was prepared with a list of reasons why the old solution hadn't scaled with the problems and presented her with specific proposals for doing things a better way. By the end of our time together, she understood what my coworkers and customers already knew: I'm a nice, polite guy who knows my stuff but is willing to sacrifice personal goodwill if that's what it takes to bring attention to real problems that prevent my customers from doing their jobs. By the end of our time behind closed doors, SHE was the one taking notes. By the end of that time, we weren't enemies; we were mutually respectful coworkers who could talk out our shared customer service problems and actually make progress on them. I'll take that over a promotion or a new hangin'-out buddy any day.
Since it's a tactical decision to turn it on and off a
Complaints per analysis-calendar cycle/number of call center tickets per 100 users/customers supported.
I killed da wabbit -Elmer Fudd
Sounds like someone wants to outsource you. I've heard this question before. Next week you come in to the office and your CIO introduces you to Mr. Smith from IBM/Perot/EDS/etc. who'll be managing the transition.
My tounge-in-cheek response is that you need to look at how fast your hardware is compared to how fast it should be.
Specifically, I've been in many situations where my company-provided hardware was too slow due to the company-provided security software. McAffee is the biggest offender, it'll needlessly make me wait for minutes while it trashes my hard drive.
You know that utility that automatically updates everyone's desktop? Guess what, making everyone reboot in the middle of the afternoon cuts away half an hour of productivity!
No, I will not work for your startup
The key metric here is "loss in quality". There are always going to be cheaper people - heck, go recruit the wino in the alley out back - but quality is a strange beast to measure. If you're cheapening your customer service, how are your customers going to react? If you're cheapening your in-house troubleshooting/repair systems, how much time, money and patience are your own people going to lose? If you're cheapening your systems knowledge and documentation, what will be lost when it needs to be decyphered in the future? And yes, I am using that verb deliberately. There are certain tasks which can be effectively outsourced. Unfortunately, there are a heck of a lot more which can't be without suffering extensive side-effects.