System Admin's Unit of Production?

RailGunSally writes "I am a (strictly technical) member of a large *nix systems admin team at a Fortune 150. Our new IT Management Overlord is a hardcore bean-counter from hell. We in the trenches have been tasked with providing 'metrics' on absolutely everything from system utilization to paper clip recycling. Of course, measuring productivity is right up there at the top of the list. We're stumped as to a definition of the basic unit of productivity for a *nix admin. There is a school of thought in our group that holds that if the PHBs are simple enough to want to operate purely from pie charts and spreadsheets, then we should just graph some output from /dev/random and have done with it. I personally love the idea, but I feel the need for due diligence, so I put the question to the Slashdot community: How does one reasonably quantify admin productivity?"

Comments

Measuring productivity?

[haluness]

How many tickets answered per day? Completed per day? /dev/random is probably the most elegant though

Re:Measuring productivity?

[DudeTheMath]

For Fortune 150, the company is probably large, so account maintenance is probably a non-negligible portion of time. To "trouble tickets"/problems, I'd add "help desk" items (adding/deleting accounts, rescuing email, re-issuing passwords, etc.) with both qty and response time reported. Otherwise, metlin has a great list.

Re:Measuring productivity?

[ldspartan]

Commenting to undo my accidental positive moderation. You're an idiot. That is all.

Re:Measuring productivity?

[harakh]

Take your task seriously. KPI (Key Performance Indexes) are a great way to prove your worth to the company. But instead of just concentrating on the obvious, how fast is service X, what is the service level/uptime on service Y etc try to create another metric that captures the other side of the coin. What happens if you fail? You now have an opportunity to provide data on how important your contribution to the company is - sure, as others have mentioned, you aren't making teapots or cars but if you fail - how many teapots or cars aren't made? How much does the company loose? What is the probability of failure currently and how could you improve it? These are tough things to figure out but if you can - the beancounters will love you.

For instance, begin from just simply calculating the average income per hour per user, this is the absolute minimum the company looses if they are unable to work. How long will they be without computer services if something catastrophic happens, do you have plans for this catastrophic event? How probably is the event?

Begin from there and tell a story about how you are a productive part of the company by enabling them to do their job.

Uptime

[GuyverDH]

Keep track of uptime. Are the systems only down for scheduled maintenance? If they are down outside of scheduled maintenance windows, what is the percentage? Was it hardware or software or a mix (old firmware with updated driver requiring newer firmware), etc...

Was the outage extended due to vendor timing? if so, maybe stock of typical spare components should be maintained to shorten the window.

Typical maintenance like adding/deleting/unlocking user accounts, resetting passwords, printer maintenance, disk admin should be a small part of an admins day. The rest should be keeping an eye out on the real world looking for potential problems like security vulnerabilities, patches, planning the next updates / upgrades.

Tell the bean counters that their demands to quantify everything will only reduce uptime and complicate matters to where you spend more time doing paperwork than you do managing systems. If they can't understand that, it's time to go elsewhere. Be sure to tell the bean counter that they'll be lucky to find anyone talented to work under their regime.

I've seen systems that went from 20% loaded to always overloaded because of the number of *accounting* applications, programs and monitoring solutions that were *demanded* by the bean counters. After a user and business unit rebellion, the *fluff* was removed, as was the bean counter. This left the systems running in a state where the end users could do their work, and the business units had satisfied customers.

Re:Well...

[SplatMan_DK]

You can't quantify SA productivity. I respectfully disagree.

You can evaluate how many users the SA's systems serve, how many systems the SA maintains, and how much data throughput all these users/systems generate.

A confused Microsoft-SA running in circles around an Exchange server all day in order to serve 200 users is not "efficient" compared to a Linux-SA running an MTA which services 25.000 users (with better response times).

On the other hand, a non-skilled Linux-SA who is fiddling with a SAMBA server in order to maintain 200 users with Windows clients is not very "efficient" compared to a skilled Microsoft-SA with a well configured AD.

Off course you can measure SA efficiency. And there is nothing bad about it. In most cases it is even a benefit for the *nix admins.

:-)

- Jesper

I have a formula

[KermodeBear]

Hours Worked Fixing Problems divided by Hours Worked Doing Routine Work

The lower the number, the more efficient the sys admin is. A good sys admin doesn't have to do anything, because everything is already set up and working. If the admin is constantly fixing servers, bringing them up, restoring data from backups, etc., etc., etc., then he isn't doing his job. If the majority of his day is spent sleeping in his chair and responding to the occasional email and things are running smoothly, then you can't ask for anything more.

Measure value, not productivity

[cjonslashdot]

This is a thoughtful and intelligent response. I would add to it though in the following way: What is the VALUE of each of these things? That is, instead of trying to measure efficiency, or "productivity", it would be more useful to the CIO to measure the actual value of what IT does. For example, if security is increased by X amount as a result of investing Y dollars, then the ROI of that investment is easily calculated and it can be compared with other investments. Note that to measure security, one must estimate (somehow) the expected future loss per unit of time. Thus, an increase in security means a lower level of expected loss per future time. The same can be done for other aspects of IT operations, including those you mention above (reliability, response time, etc.). In the end it must be measured against the value to the organization. Productivity is the wrong approach. Value is the right approach. - Cliff

Sounds simple to me

[mrami]

1) Ask him what he wants to hear
2) Tell him what he wants to hear.

If you can't reasonably tell him what he wants to hear, tell him how much it will cost to produce what he wants to hear.

This is not a technical consideration. This is a political consideration. He already has an idea of how to cover his ass. Give him the asbestos he wants.

This is not your responsibility

[LargeWu]

...no matter what your boss says. Just don't do it. It is management's responsibility to come up with metrics. If they can't do that, they're not qualified to hold their position, and frankly, I would tell them to their face. It might get you fired. But I've taken the "this is not my responsibility" tack before with some success. The reason this stuff happens is because workers allow it to happen, and if you don't stand your ground once in a while they will just keep shoving this type of crap at you.

The hammer priciple.

[infonography]

I remember an old joke about a furnace repairman coming to a home and after looking at the furnace for about a minute and a half, listening to the rumbles and gurgles. He takes his hammer out and at once precise place he hits the furnace. The furnace starts up and runs fine as if it was brand new.

The bill was $200.

The homeowner asks why so much when all he did was hit it once with a hammer?

The repairman takes back the bill, and itemizes the bill still totaling $200.

                      Cost of hammering, $1
                      Knowing where to Hammer $199

Any idiot can muck about on a UNIX box, I worked at one Fortune 500 company where everybody in the dept had Double E's. Still their main Solaris server crashed ever 3-5 hours daily and had been for months.

Took me a week to unscrew it and put everything back in order.

Me, I am high school dropout with no GED and some non-technical college courses. Still most of what I was doing was letting them do their work and not have to bother about broken systems. My value was on par with theirs as it was time they didn't lose on their work.

Nevertheless beancounters are stupid (also Beancounters are not accountants), they know the cost of everything and the value of nothing. If you really want to send their head swirling take the entire labor budget for each dept expressed as an hourly unit. Every time you work for a dept internally charge the company that much for each hour you work on a project or ticket for them or better still your company and tell them thats how much it costs. Without Sysadmins nobody does anything but fight technical fires and gets no work done.

Likely this joker found out that Auto Mechanics have a book to calculate how much to charge for each service and repair with details on how long each job should take. This doesn't work because Sysadmins are closer to being chefs or doctors then low end auto mechanics.

Even so, people who own Jaguars, Ferrari, and Maserati don't take them to Jiffy Lube.

If they complain tell them the story about about that hammer. (or better yet use on on them)

Re:The hammer priciple.

[TapeCutter]

"God, slashdot is full of you guys and frankly I think you're all full of shit. No offense to you personally, I just hate seeing people kicking on college degrees like they don't mean anything."

Ditto, only financial success or inexperience permits the arrogance of thinking EE == Computer Science == Software developer == SysAdmin == Help Desk slave. It's total bullshit to infer dropping out of HS won't make a "difference" and that financially it is a detrimental step for all but a tiny minority of naturally gifted (or extremely lucky) entrepenuers and assorted geniuses.

Me, I'm a high school dropout who then went on to be a member of the "working poor" and/or "time poor" until I obtained a degree in my early 30's. I quit the factory, got a lisence and invested $1K in a brand new Acer XT with a "paper white" 16 colour CGA monitor to replace my $80 second-hand IIE at the start of my course. It was not a small investment for us, but freeing up the family TV was a diplomatic coup over the wife/kids and it was also the best finacial one I ever made.

My last year at uni the computer directly earned me $9K writing small "search and sort" examples for a text book the dept. head was writing. Indirectly, I am now quite comfortable in my late 40's and have a decent chance of being very comfortable in my 60's. Barring a lotto win, my alternate future would probably have seen me as a factory foreman (I was already a leading hand -death by rotating shiftwork- shudderrr). And when the factory job "disappeared", as so many did in the last couple of decades, I would probably have spent my severence pay on a franchise gardener/handyman/taxi-truck "bussiness", OTOH: I would probably be 10-15Kg lighter and have an endless supply of home grown pot... :)

Today I live by the beach with a 15min drive (or a broadband connection) to the office, I like to take pictures of storms on the bay but the trick is finding a profitable passion to pay for the time and material needed for your other passions (that may also one day be profitable). Sure doing a degree and finding out just how smart others before you have been can dull the "passion" as one by one you find your "original insights" are not original and/or not instghtfull. For me it didn't completely kill the passion but it did open up the enormity of what I don't know and how hard it can be to "find out". If the "passion" were to die completely I have made enough profit to walk away and do something else besides mowing lawns for the middle class simply because a little passion for one's job and a decent "fuck-you" fund equates to power when dealing with PHB's.

If someone doesn't think a degree will help with any of that then to them it isn't going to. They are either already "happy" or belive like I once did that the only reason for "working" is a paypack.

Re:The hammer priciple.

[Iron Condor]

Please correct me where I have misread, but in all your verbosity you seem to have made two statements:

- One cannot quantify the sysadmin job

- You're better at it than some

To my (non-MBA) eyes, these two appear in contradiction. If one cannot measure/quantify how well a sysadmin is doing his/her job, then one cannot claim that one is doing a better job than the other. Thus one might as well hire the cheapest guy. If, on the other hand it is supposed to be possible to say "you're doing a better job than joe" then there must be something measurable, observable, something that can be put into a number and that number differs for you and joe (in such a way as to make yours the better number).

I admit I have no idea how to measure the quality of a sysadmin. If my stakeholders forced me to (I.e. my boss said "quantify what your IT dudes are doing") I would go to my sysadmins and say "please give me numbers that show how well you're doing". Because the alternative is pulling something out off my ass -- and neither my superiors nor the people who work for me would like that.

Re:The hammer priciple.

[fkicker]

Management will always get what it measures, but not always what it wants.

In the town I grew up in, they used to measure the fire department by the number of runs they did per day. It wasn't long until the fire chief started rolling a 30 foot truck for every fender bender in the city. A 30 foot truck did wonders for traffic already snarled from the fender bender.

Poorly chosen metrics are worse than no metrics at all and poorly chosen managers love poorly chosen metrics. It gives them comfort in managing an organization that they don't really understand. It also keeps them from having to "be a manager" and making hard decisions. The metrics can decide who gets a raise and who gets the additional headcount.

How would the world be different if the Pope had measured Michelangelo's progress in the Sistine Chapel by the square feet painted per day? Maybe the number of feet painted per day?

Maybe a manager could measure the productivity of pregnant women by the number of months to make a baby?

I'm not saying that you shouldn't have metrics. Have tons of metrics! Some things should be measured all the time (like system uptime) and some should be sampled occasionally (like user satisfaction). But, all metrics need to be interpreted by a competent manager instead of an excel formula.

Management by metrics only works for people too stupid or too honest to not manipulate the metrics.

Re:The hammer priciple.

[SavvyPlayer]

I don't mean to respond to your post per-se as there are dozens of such posts to which this reply would be appropriate.

There is an intractable problem here that noone seems to be addressing but that everyone here contends with: accurate estimation. Every area of a publicly-traded company in this post-Sarbox/Enron era is required to justify its effort not only to management but to the public. IT, along with every other area of an organization is now required to show its efforts are not only practical but necessary. How exactly does an IT practice deliver quality estimates to both the business and its shareholders? Shareholders and governments demand these metrics, our PHBs simply carry out these orders.

Measure QUALYs, like doctors do

[Anonymous Coward]

You can measure your productivity in the same way that doctors do: by measuring how much better you make life for your users/patients.

The medical unit is the QUALY, or "Quality Adjusted Life Year". So If you're a paramedic and you save the life of a 25-year-old who will live to 85, that that's 60 QUALYs. However, if their "quality of life" will be degraded, e.g. due to a disability, then you quantify that.

QUALYs are normally used to determine where to spend money, e.g. if this cancer drug costs $X and adds 3 weeks of semi-unconcious life, is that better than spending $X on drug Y.

How do you apply this? For each thing that you do, count how many people are affected. Then try to judge the "quality of (work) life" effect on them. E.g. being without email for a day = 30% impairment to quality of work life. So if an email outage affectng 100 people lasts half a day that's 100 * 30% * 0.5 = 15 QUALdays. To measure your productivity you then need to judge what would have happened if you weren't there (i.e. outage lasts all week or no outage at all).

Presumably you already have some sort of issue-tracking system. I think you just need to extend this to send out an additional message: "now that this ticket is closed, please tell us how much your QUAL was impaired during the period of the problem". And the rest is a perl script.....

Re:Easy

[Alsee]

productivity = (SystemUsers) * (TimeReadingSlashdot) / (SystemDowntimeRate)

And while that is obviously a joke, it also happens to be pretty much correct.

Obviously if you increase SystemUsers while keeping all else constant, productivity goes up. If you reduce SystemDowntimeRate while keeping all else constant, productivity goes up. If you spend less time on general maintenance and random panic fixes (increase TimeReadingSlashdot) while keeping all else constant, productivity goes up.

-

The Barbrady Effect

[infonography]

Nevertheless beancounters are stupid (also Beancounters are not accountants), they know the cost of everything and the value of nothing.

Beancounters who try to impose their metrics are stupid, as they really don't understand what to measure. Any worker should understand their job enough to put together metrics that demonstrate their value. In the original post the beancounter went about things the right way, ask the admin how they would measure their job. If you've ever watched South Park you may have seen the seemingly inept cop who somehow seems to keep order. As demonstrated in one episode when he was removed from office. Chaos ensued and was stabilized by his return.

When a IT dept is feeling underfunded they merely threaten to quit. The reaction is akin to a three year old being told mommy is leaving. Good IT people are worth their cost, and most are not so greedy. They stay because they want to not because of the money. Beancounter bosses tend to pay for this sort of thing later as nobody wants to work for them when word gets around. Soon they have ex-MickyD employees running the Desktop support and the smell of French Fries just won't leave the server room.

Has was stated earlier, You can not derive a metric for a negative event. You can only measure a sysadmin's stability factor in terms of (well) stability. Other metrics could be created by how long new services appear after they are requested, but there is no way to gauge the factors relating to them.

Management:

                    We would like to replace Oracle with MySQL how long is it going to take?

                    All our servers are running Solaris 8, the current version is Solaris 10, please upgrade asap.

                    We want to move off linux and on to HP-UX before next month.

As to the opinions of others....

                  Hi, welcome to IT you must be new. Per hour performance metrics is about the stupidest thing I have heard. We are not factory workers we are information workers. One problem can generate 2000 helpdesk requests in 10 minutes and take less then that to fix. I am not going to write out two thousand emails to tell everybody that we had a fiber cut from the ISP and our mail server will be down. Sounds like you've been reading the latest management fad like Sigma 7 or some other such tripe.

      Actually thats not quite how it works. If the folks in accounting (non-beancounters) usually call up and ask if there is a problem and you give them a update on whats up and they go back to what ever they where doing. If the dept needs more money next time around they just explain what its for in a meeting. If it's not ludicrous then the usually get it. If the manager gets the numbers wrong then that manager has to fix it some way. A good manager will pad it and use Departmental luncheons which tend to eat up the slack.

Re:The sysadmin's job ...

[Anonymous Coward]

I have never been a SysAdmin but I have done IMR. In IMR if you are really doing your job, then no one knows your there but the bean counters, unless of course someone wants to add something new to the mix or operator error. Of course it was more obvious when you regularly had to change the machine setups to run different products. Pretty much everywhere I did that there was a ticket system to make the bean counters happy and the better you did your job the more creative you had to be to account for your time. If you didn't make the bean counters happy they would start working to get your job eliminated and if you worked directly for production in this respect the departmental manager might try making you do production work when not doing maintenance or repairs.

Fortunately the at the place I worked the longest in that field the plant manager was a former IMR technician and he would give a dressing down to any bean counter or ignorant lower management who messed with us, as he put it "this plant is running very smoothly because of this IMR crew and they all know that the smoother things are running the more time they have to relax and plan future required interventions to keep themselves relaxed with free time to think." Course we were not always thinking of such things while relaxing and he knew that, but he knew as long as we got rewarded with relaxing time at work, the smoother we would keep things running to preserve the relaxed atmosphere.

Personally, I think the SysAdmin who contributed the question here should check the obviously clueless "new IT Management Overlord"'s computer to make sure they not violating copyright via downloading recordings; make sure they are no trojan infesting the system; and make sure they are not downloading large amounts of porn, it would be a shame if this "bean counter" had to account for anything like that.

Re:Number of Cases

[mvdwege]

[...]the MANAGER -> the LEADER who should be providing these metrics to you and not making YOU do HIS work!

Welcome to the real world. In corporate management, it appears that it is the job of the manager to order his subordinates to find out what his job is. Actual productivity is irrelevant. It is all a shell game to provide the semblance of useful work for the new feudal overlords.

Which is why I quit. SMEs have their problems, and they are not entirely free of the MBA Morons, but it's a damn sight better than any Fortune x00 company.

Mart