Slashdot Mirror
Are Sysadmins Really that Bad?
tgbrittai asks: "According to Paul Boutin they are merely an obstacle to be manipulated or outmaneuvered. According to Steve Wozniak they are pimps. I've known my share of good and bad sysadmins, programmers and every other professional role out there, and I have to wonder: are sysadmins really THAT bad?"
Most times sys-admins are overworked and underpaid and have to deal with users who take advantage of their local IT person, tasking them to fix systems that they callously break. Others are truly worth the name "Bastard Operators from Hell". How would you rate your sys-admin and what things did you have to do to make things run smoothly (or not)?
Can I really be trusted to tell you how good or bad I am?
.. which begs the quesion.. do we really have SysOps anymore?
Frankly, I've always loved the name SysOp.. it just sounds better.. even though it's not an accurate title anymore.
----- The internet has given everyone the ability to have their voice heard equally as loud.. even if they shouldn't be
I currently work as a Network Admin for a large retail company. I started with this company as a store clerk, then moved to the helpdesk, while i was in college. This helped me learn patience and how to be polite with everyone no matter how annoying, wrong, irate the "customer"/"employee" is. I look at some coworkers who have no clue how to handle talking to a customer or a user needing help and give them lip every chance they get. I understand the frustrations of having petty work assigned to you by a VP level person that interrupts your day and workflow. All the time i have important time constrained projects interrupted by those "important" people who have to have some blackberry/treo/etc problem fixed asap. I have to drop whatever important task I am on and concentrate soley on the happiness of this one person. Ultimately thats what it comes down to i have found. Although i get my work done and i am thorough on all mky projects, I am not known for that, I am known for always being the nice guy who helps out the Execs and their exec assistants, and honestly that puts me in better light than anything else. Sometimes the interruptions are extremely frustrating, but when the execs are happy, everyone is happy.
I will admit, however, that I always hear circus music when I'm standing near one...
Its not that there bad people. Most of them are pretty nice if you talk to them over lunch. The problem is they are so constrained by what they can do that they are very frustrating to work with. I must say they also hate working with me for the most part.
The problem is that I'm somewhat tech savy. The sys admin don't like anybody trying new things. Their management likes it even less. Do one little thing or install one little app and if you have problems your on your own. Doesn't matter if your laptop explodes, they'll blame it on VMWARE or whatever you happen to be running.
It wasn't always like this. In days of old the Sys Admin were local and reported into the same groups they supported. As such they knew what we were working on and would help out. Management would support this because it often lead to increased productivity or reliability. But at some point a bean counter decided we needed a corporate IT organization.
Once you decouple the support from the groups they support you end up with apathy and endless rules. Also to get the groups to try anything new you have to weave your way through a bureaucracy. You also end up with smaller and smaller IT groups because their contributions to the end product become harder and harder to trace. If a business unit needs to cut costs the first thing they look at is horizontal organizations outside their own structure. Its a lot easier to cut an outside IT guy than a developer working on a product.
Things look to be taking a turn for the worse. Some of our IT is now going to be out sourced. To me this is equivalent to saying I now fully support myself. I can just imagine trying to convince some contracted person in India that I really do need to have VMPlayer installed on my Windows laptop....
One should not theorize before one has data. -Sherlock Holmes-
We will have SysOps as long as we have people that don't wish to know anything about the mainframe / mini computer / etc. . Their titles may change somewhat, but ultimately the role will remain.
;)
Need that report, but don't know how to login to the mainframe? Call an operator.
Need this report to print in front of the 30 jobs in the queue? Call an operator.
Need to cancel a scheduled batch process? Call an operator.
Alternatively, we could just add those tasks onto the shoulders of the sysadmin... it's not like they don't have free cycles
Once, we were narrow of scope with a deep understanding of the subject matter. Now, we are a mile wide and an inch deep. Less focus, more distractions... ooh - something shiny...
I strongly agree with everything you're saying. One of the 20 unofficial roles I have at a large bank is Unix System Administrator. I really only spend ~100 hours/year doing system administration, and that's only to deal with something breaking. We have enough work for a full time sysadmin, but we have management who aim to consistently do less than the minimum. I believe the fundamental problem of system administration in any business environment is that you never see the benefit of good results. You only see costs of failures and people running around putting out fires all of the time. A good system administrator tends to work himself out of a justification for a job because there's no compelling business reason to keep employing someone expensive whose benefits to the organization are invisible. Coming in on the weekend to replace hardware, fixing things that break before people notice they are down and recovering files for people who will never admit that they deleted something important are all common sysadmin tasks that are rarely acknowledged.
Micromanagement and imaginary, perceived cost savings create unsustainable environments. Here in a non-technology group of a large bank, we've got a handful of Sun servers attached to an EMC. There are numerous persistent memory errors on the Sun's that could be fixed with a service call and a small scheduled downtime. Well, in a normal environment that is all it takes. However, we don't currently have a maintenance contract. We did have a service contract years ago when the problems started, but maintaining systems is an anti-goal for management - apparently there is no profit in keeping things running. The EMC has been performing well, with the occaisional disk failure that is completely invisible thanks to RAID and automatic call home to get a replacement disk sent out. That's been our key saving grace since we don't backup anything(including production servers).
Unfortunately, this kind of short sighted, unprofessional approach to IT is common in business driven organizations. When everything comes crashing down, as it always will given sufficient time, someone will look at what happened and try to prevent it from happening again. This is the kind of sabatage through mismanagement that leads to the creation of company policies that make it hard for anyone to do their job. Our company has policies that require that system, network, security and database administrators all be separate people. The developers have to be separate as well and can't have access to production systems. There's some very good reasons for all of these policies, but business people can't resist the temptation of hiring one person to do all of these jobs. After all, who better to get things working and fix problems than a developer with root access to everything. It sure cuts down on time wasted in getting authorizations and having meetings.
The first IT job I took after getting out of the army was answering phones on a help desk for a retail company. We had to support about 1,500 users.
Not long before I started that job, the company had hired a new "Director of End User Technology" and this guy was sharp. His primary goal at the time was to straighten out the cobbled together mess of a network that had haphazardly grown department by department. The place was a real mess and the network ran like mud.
Over a period of about four years, we standardized our PCs and laptops, physically consolidated the servers that were spread all over the HQ building, corrected the messed up cabling, centralized administration, built a training room and implemented a number of classes, etc. It was a truly exciting and fun place to work and virtually everyone who I started out with on the help desk eventually learned, got certifications and moved into administration and/or engineering. When I had started there we had a real mom & pop shop type feel and very little oversight. All we had to go on were some clearly defined goals and a directive to "get things fixed."
We consistently accomplished our goals. Within the first couple years we had fixed the network and made it into something useful. The consequence was more use by upper management and as you might expect, more management from upper management. Every time we met another goal, the more visibility we received. The more visibility we received, the more layers of management they installed above us. Every layer of management installed made it harder and harder to actually get anything done, basically because each new layer of management knew less about IT but more about "managing.".
I guess mostly I'm just whining here, but eventually most of us who had built the network quit. They 'managed' us right out the door.
Politicians are like diapers - they should be changed frequently and for the same reasons.
But they lack the power to change it, and their bosses don't want to.
That brings to mind my first rule of systems administration: Give me the authority and the resources to prevent the problem and if it breaks anyway I'll work 20 hour days to fix it. Get in my way and stop me from preventing the problem and I'm headed home at 5:00 whether you're in a frothing panic or not.
Most places I've worked liked the display of initiative and steped back to let me do my thing. They liked the results too: 20 hour days were very very rare.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
That's not entirely untrue, but he's also the kindest-hearted guy on the planet and would never say anything genuinely nasty about anyone, not even sysadmins. Not even the sysadmins who tell you that Lotus Notes is fantastic, "it's just a terrible e-mail client".
What I'm listening to now on Pandora...
Yeah, I got my back all up and was ready to write a diatribe (I did RTFA first, however) and found out that this story is a simple case of a lack of reading comprehension. Every time something like this happens, it makes me sad. Especially on a geek news site, where probably a lot of the people have dabbled in programming - they'll be as careful as they need to be to get their code to validate, but when it comes to understanding a natural language, they won't even put in the effort. Then we end up with crap like this. Half the time I'm explaining something to someone, it seems like they just don't get it. (the other half of the time they're raising on-topic objections) :) Maybe I need to dial back my vocabulary for the average person, but I think there's something about logic missing there too.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
The solution is so painfully obvious that no one will ever implement it.
Business people need to look at more than one little line item on a budget. There are a lot of jobs that pay $50-80k for a sysadmin. The vast majority of day to day things can be done by one of these people. When they get stumped on legacy stuff or something really weird, they end up spending a lot of time spinning their wheels and have a hard time getting the problem solved.
The other option is to hire the $150k sysadmin who has tons of experience and makes the hard problems look easy. These are the kinds of people who you can give 3 months to solve a problem, or you can hire a team of 5 people to work for 20 years on the same problem. If you put it in that perspective, the money is well spent.
Smart business people look at numbers and know that $150k is more than $50k, and also know that if they yell loud enough about the $100k they saved, some of it will end up in their bonus.
The thing that seems obvious to me is that you hire a bunch of the cheaper people who can do all of the normal day to day stuff, and you also hire a guru who gets all of the impossible tasks. The less experienced guys learn from the guru and the guru doesn't spend 99% of his time doing tasks that would be better suited to a college student or a shell script.
Of course, companies don't like this idea because HR people don't want to believe that one person can be worth several times as much as another person who is referred to with the same type of job title. In HR there are no gurus, so the concept is completely foreign. After all, if someone was inclined to be a guru in any field, how would they end up in HR? =)
After 3 years in support for a fax solutions provider, I encountered what I would estimate to be a 5% level of competence. Meaning 95% of SA's I talked to don't know their ass from their elbow. I talked to this one admin of a windows network that didn't know what Active Directory is. Let me tell some more anecdotes.
Now, you might say that they're probably underpaid and overtasked... the underpaid and overtasked guys were actually usually the competent ones. I've had Windows admins call me, support for their FAX solution, for EXCHANGE support. Now I mostly handled Linux calls, My MSEXCH knowledge is very cursory... this guy got pissed and I basically referred him to a MS KB article on what he needed to do. Incompetence is so prevalent in the SA job role it's really just insane. It's gotta start somewhere above them though, cause management made the incompetent decision to hire this incompetent admin. So it's sort of a chicken/egg.
I also loved the admins who were all like "we HAVE to get a linux solution! linux is so rock solid! I don't really know how to use it or even log in to it, but it'll be the proverbial rock in the data center!" Then when it breaks here I am phonetically spelling out commands like cd and ls, while they enter the wrong kind of slashes (if they figure out how to log in first). Then when they enter the proper command finally they're like "I don't think it worked, nothing happened."
Another herd of sa's that were awesome were the ones who either had a language barrier or thought it might be impolite to say "no" when you ask "Is the XXX service running?" It makes for a really fun game where you just have to guess whether they really mean yes or no.
Well I won't keep boring you with my mundane tales of the greatest SA's of all time cause I think you get the picture.
"If you plant ice, you're gonna harvest wind."
Very true.
I'm a sysadmin in a DoD Classified network on a USAF Base ( LOTS of red tape) and the first rule is security (which it should be). That pretty much means lock everything down. Some examples include: lock the USB ports, prevent writes to CD-RW Drives, prevent writes to DVD-RW Drives, audit everyhthing (PL2), prevent printer installs, software installs/removal, lock down screen savers (executable code), password changes every 62 days, approved software installs only which usually means we are lagged on releasees, etc, etc, etc. Some of these are silly, yes, but I don't make the rules. The "red tape" is mandated to us by the Air Force.
All this red tape creates a very unfriendly user environment where the users frequently are annoyed with the admins because they can't do something as simple as copy data from a classified PC to a classified laptop for a presentation. They have to track down an admin to do the copy for them. Paperwork must be filled out and whitnesses present. They may not have access to files due to security permissions. Won't delve into the requirements here but it has to do with employees from different companies all working the same program who potentially have access to each company's proprietary information. I can go on and on, but the bottom line is red tape creates a very unfriendly user environment where the users frequently claim the sysadmins "don't know what they are doing", which isn't the case at all. The users are deliberatly not allowed to do what they are trying to do. However a majority of the user community thinks us admins make the system painful to use on purpose. Not the case. and they frequently take out their frustrations on us.
I wrote a blog post about this not too long ago. It's funny that this thread came up so soon afterwards.
r ception.html#links
I got my start in sysadmin work, and then moved over to programming. I've run across my fair share of BOFH, and they make everyone else look bad and probably even work harder. I've found that if I treat my department like a separate company and other people are the customers, that things go more smoothly. That's not to say that you let people run all over you. I suppose I've only worked in small businesses, so I don't know if that would be appropriate for large companies or not.
http://cmunkey.blogspot.com/2007/05/its-public-pe
I literally have the worst of both worlds -- all the aggravations of a small company (not enough resources) and all the aggravations of a big company (too much red tape). I work for a small, moderately nimble software group in a very large company as a DBA, so in many ways I am basically a sysadmin. I need root access to my servers, at least during the build phase. However, we have sysadmins who refuse on a regular basis. They'd rather wait around all day for me, running the commands I give them via IM, then complain how they have no time to get anything else done.
Our operations team has no process by which to engage them for work. They don't track anything. They cannot and will not give you a firm completion date on any work. When I ask for specs on some of my production hardware, they say "We're too flexible and dynamic, we don't bother with keeping those records." But when I need something like vim installed on a server, they claim "that's not a standard build software package." They all go to lunch together, leaving no coverage for 1-2 hours in the middle of the day. They sit in their cubicles and deride everyone else they think is stupid. They got a project manager -- thank god, we almost got a few things done -- then they drove the new PM off to another group with their utter disdain for being organized and following direction. They horde servers and SAN disk space like it's platinum. They've got 70+ unused servers, but they can't be bothered to inventory all of them and let us use them for things that we need. They don't have simple things set up properly, things like time synchronization between servers (you try running clusters of servers where the clocks are off by 20 minutes from each other!), backups, etc. They provide NO monitoring whatsoever -- they don't know it's down until you tell them.
I have a service mentality. I work with developers and make software. I understand that if we weren't doing that, we'd have no jobs. Not these guys. They expend extraordinary amounts of time and effort telling us what they will not do and why they won't do it, but if they put even half that energy into just doing the fucking work, we'd all be much, much happier.
So yes, for me, sysadmins really are that bad. And ours aren't nearly talented enough to be as arrogant as they are. *
*If you're super talented, you can have a level of arrogance.
While we're sharing anecdotes:
... and left. As everyone knows, the admin. assistants are the real power brokers in large companies. This one was no different.
A long, long time ago I was the primary SA for a building that housed one of the top projects at a well known software company. As part of my routine, I would walk the hallways, talking to the admin. assistants and a handful of Important Users to get an idea of what was happening in their world. [After all, if I knew what they were doing, I could make sure that I didn't schedule an outage right in the middle of them doing major work. So, unlike a lot of my brethren, I don't mind mingling with the masses.] One day on this walk, a person called me into his office and said:
"I need for you to do task XYZ. This is for project ABC so it has the highest priority."
I reminded said user that he needs to file a ticket ASAP, but I'll get to it as soon as I can.
I continue my stroll down the hallway (my office was in another building, so I couldn't run to my desk if I wanted to) and was stopped by another user. He said:
"I need for you to do task LMN. This is for project ABC so it has the highest priority."
Same thing: reminded said user that he needs to file a ticket ASAP, but I'll get to it as soon as I can....
As a bit of an epilogue: Months and months of walking this hallway had taught me one thing--this set of users in that hallway was particularly abusive of the SA team. Anything and everything was an emergency and they were highly abusive to junior SAs. I had had it. So one day when nearly the same scenario happened, I responded a bit differently. I went down the hall, grabbed user 123 and took him to the last user's (let's say 567) office.
User 123 has task XYZ for project ABC. User 567 has task LMN for project ABC. You are both on project ABC. You both say that your tasks have the highest priorities. I'm going to request that you sort this out and let me know which one is the actual highest priority. If it makes any difference, I'll be having a chat with the big kahuna for this project on my way out.... and I plan on asking her opinion as well. Give kahuna's admin a call to let her know your decision.
By the time I got to the head kahuna's admin. assistant, she told me that both users didn't actually need those things done after all. I also noticed that the users were a bit more realistic in setting priorities for tasks as well....
I'm sure they must do something important, but as far as helping the employees do work they are completely useless as far as I'm concerned. They come up with policies like "only use IE 5" at a computer security company and withhold privileges from people who know way more then they do, and block all security patches. Oh, and block all the dangerous hacker websites like redhat.com and secureinfo.com.
I once asked the sysadmins why we have 100 people on one printer that always breaks down, and could I be added to one of the many idle printers, and was basically told to go hack the servers and reconfigure the network myself. This was at a huge defense contractor. These were the same guys who backed up gigs of work on the F-22 onto obsolete tapes and then deleted the network drives and then threw away all of the tape readers. Yes, all of them. I was later assigned to reverse engineer what were basically sealed black boxes and re-do the VHDL. That begs the question: "why didn't the buy used readers on ebay?" Good question. Oh, and we had a 5 MB share drive to store all of our work, so obviously we stored almost everything locally. If we reported a problem with our PC they would reformat the machine. Once I told them I had a problem and to not format my machine unless they back up all the data. They said fine. Later I returned to find my machine formatted and my work gone. When I called they told me "we never back up data, it should be on your share drive." They decided something was wrong with my cubemate's computer and snuck in when he was at lunch and reformatted it without ever informing him. He lost years of email and all of his work. After that everyone put signs on their cases that said "do not format this machine." Oh, and our net of 50-100 people was on token ring and none of our apps were installed locally. If one person kicked the cable wrong we were all out of business.
At another defense contractor I reported problems where my machine would lock up for 300 seconds at specific intervals, and was told my problem was impossible and it didn't exist. I reported it many times before finding some lower guy, telling him about it, and he fixed the DNS server 5 minutes later. They never did fix the feature where if I set my clock to the correct time the server would change it to be off by 17 minutes. They also insisted that was impossible and never looked into it. It's not a bug, it's not a feature, it's a hallucination apparently.
Later at that company I requested a laptop with admin privs before leaving on 1-2 week trip, only to told after I got there that they don't give admin privs so that I basically lugged a boat anchor across the country for no reason as I couldn't install the compiler. You should have heard him whine when I said "So in order to use it I guess I just have to reformat it. No problem."
My bosses have recently taken to calling the workers human capital. It rather smacks of slavery, doesn't it?
... ouch!
Now I just hope they don't leverage me again