Slashdot Mirror
Are Sysadmins Really that Bad?
tgbrittai asks: "According to Paul Boutin they are merely an obstacle to be manipulated or outmaneuvered. According to Steve Wozniak they are pimps. I've known my share of good and bad sysadmins, programmers and every other professional role out there, and I have to wonder: are sysadmins really THAT bad?"
Most times sys-admins are overworked and underpaid and have to deal with users who take advantage of their local IT person, tasking them to fix systems that they callously break. Others are truly worth the name "Bastard Operators from Hell". How would you rate your sys-admin and what things did you have to do to make things run smoothly (or not)?
....when he suggests "Treat everything he does as a favor. ". Actually, that's not a bad life strategy - when the waitress refills your coffee right away, treat it as if she didn't really have to - because, really, she didn't! She could have just ignored your empty cup, or waited a few minutes, or whatever.
Same with a sysadmin. When he adds a rewrite rule (done!) 20 seconds after you ask for it, act appreciative and say thanks, even though that's his job. Because he could have put it off until tomorrow and probably would have reasonable excuses for doing so. (Incidentally, I hosed up this rewrite rule the first time by leaving off the trailing $. Doh!)
The Army reading list
Don't cry me a river about that...with the exception of upper echelons of management, I'd say most people do more for the company than they get back as a reward for their work.
I've been on both sides of the fence, I've seen users that put every piece of software they can find on their machine, then come calling when they break. I've been blamed for doing something to break a printer, about two weeks after I was there to swap a monitor.
On the flip side, I've worked in places with a tiny server share to store important data and an IT staff that doesn't really guarantee it'll be backed up. So we ended up having to work around the IT staff in a lot of things. It was easier to cobble together something that we can guarantee is backed up AND that has enough space for us than to go through the reams of paperwork to get more space and justify some sort of improved SLA.
In fairness to the IT folks though, a lot of the people working IT are just trying to feel their way through the system that was put in place before they started, and they think it's just as stupid as the end users. But they lack the power to change it, and their bosses don't want to.
"It is a miracle that curiosity survives formal education." -Albert Einstein
I think the new "fleet" (if I could call them that) of sysadmins are too inexperienced and are often thrown into a wild west of "our infrastructure works like this!... With an infrastructure that many times hasn't been planned out too well, is highly misconfigured, is a nightmare in progress. Often those sysadmins will have to adjust to someone else's tailored system and will fail miserably... I've seen it for years on end, horribly designed systems with no documentation, horribly managed systems butchered to perform a task. No two systems will be alike and I believe its this same scenario which makes or breaks an admin... However with the newer sysadmins coming around, and I've seen plenty in the past 3-4 years, they're inexperienced... Running Linux @ home or your own personal webserver does not make you a bonafide sysadmin. At least not in my little space... I know admins who strictly know perl... Good for you. Now go fix this legacy system which by the way doesn't have perl on it, and you're not allowed to install perl... Would you know how to do so in say awk and sed? To me a sysadmin knows things from the core up, not from a yum install *something*, apt-get *make-me-look-nice*, or whatever other command. Just my two centavos
Infiltrated dot Net
Replace "X" with any profession, and the answer is the same: some are and some aren't. The professions with high barriers to entry (i.e. medicine) tend to root out some if not most of the incompetents or otherwise poorly qualified, but some will still slip through. The same is true of sysadmins. They obviously exist for reason -- maybe the article writer should ask, "What would a world without sysadmins look like?" For large organization, the answer is "chaos," and they would quickly re-implement the same positions now being mocked.
I used to be a sys admin for a medium sized company. Some people thought I was great, some people thought I was a jerk. If someone was nice to me and was willing to learn how to do the simple things them selves I was more than happy to help them. People who I had to show how to attach a file to an email seven times saw a less friendly side of me.
That being said, some admins are just jerks no matter how nice you are to them, and some users are unreasonable and demanding no matter how hard you work for them.
MG
Randomly distributing Karma whenever possible.
Pay no attention to the systems administrator part of my job title...it's just a standard honorific. >_>
Before I launch into this, it really seems like they define good and bad by their customer service skills, so that's what I'm addressing by "good" and "bad", not so much their technical knowledge.
In my experience, the problems with sysadmins tends to be that with the ones that lack the ability to understand the user. This is what people refer to as the "IT mindset" where the user is the enemy and is doing whatever they can to make IT's life more difficult. In some cases, this is very true. There ARE abusive users out there. However, most people simply want to do their job, and their job is NOT getting these machines to work right. Getting back to the "understanding the user" thing, I find a great many sysadmins have no empathy for how a user feels when their machine has gone down, and why would they? When has a sysadmin ever really felt the panic and/or frustration of having a machine crash and not having the first clue of how to fix it? We KNOW what we should do, and while we'll be annoyed at the extra work, we're (hopefully) never flailing around blindly...or if we are we're careful never to show signs of it. A user's machine goes down and they have no idea what to do. They panic, they worry, they don't think logically...they immediately run to the nearest person who they think can help them and oftentimes get the look of "Why should I?" or "Can't you see I'm busy right now?"
Again, that doesn't mean there aren't people who don't actively try to bypass what they SHOULD be doing to get the problem they caused looked at immediately because they think they're more important. However, I think the sysadmins that most people complain about are the ones who let the handful of lazy/abusive users jade their dealings with the ones who simply want to do their job and go home.
However, I find that the "bad" sysadmins are about as common as the truly abusive users. They stand out in your memory so it seems like there's a lot of them, but they're actually far from the rule. YMMV, of course. After all, in the course of a day three or four people might stop to hold open a door for you, but the one you remember at the end of the day is the idiot that cut you off on the highway. Human memory is a funny thing...
This is spot on. I am that sysadmin you talk about. i work for a large software company - however, I was hired specifically to support one smaller office, as the IT group couldn't (or wouldn't) provide adequate support. Things were great for a while - I learned the quirks of this particular office, setup new systems and generally made the work environment better for those that work out of this office.
Thats when a new manager, and IT overlords stepped in. Now I have to do everything 'by the book', even when 'the book' doesn't mesh with what we are doing here at this satellites office. My life is now a hell of process, procedure, and meetings - and very little actual work is getting done.
What does this lead to? Developers going 'out of band' to get stuff done - purchasing hardware on credit cards, not using authorized apps, copying large files around the WAN when stuff should be local, etc. All because they can't get a slice of my time to help them with a correct solution.
Everyone here is frustrated - myself the most. I *want* to provide the best support I can, but I'm now hamstrung by process and management, whereas before (when the developers/local managers were happy) I wasn't.
I think most sysadmin jobs are going this route now, excepting the startups (and they will, as they grow). Sysadmins are a commodity now, they aren't viewed as adding value.
Admins are literally wedged between workers and management.
... umm ... his quarter years report (yahu, sure), and if it isn't reactivated IMMEDIATELY, you're in deep dung.
You, as an admin, get orders from management how they envision the network security to be. You know it doesn't work that way and will only create an obstacle for the people you're to protect, but you will do it anyway. Because the guy you knew from the day shift one day took one such memo and trotted upstairs to the brass.
He hasn't been seen since.
So you do what you're ordered, block non-corporate mail accounts, block porn sites, block ebay, block... everything. This is usually when one of the middle managers complains that he can't go online anymore, which turns out as him being unable to access ebay anymore which he needs for
It escalates up to the top brass, you get said pile of manure onto your head for not cooperating with middle management and you now have to work out a plan how to block ebay without blocking it. Sounds impossible? I know that. You go upstairs and tell the brass. Can I have your stuff?
Then you head down to the cafeteria for some coffee. Coffee good. Coffee lifeblood. My precious. But you forgot your fake moustache and the noseglasses, so people immediately recognize you and start asking what's wrong and why they can't access gmail and gmx anymore. You explain the brass note. Which causes them to tell you in no uncertain terms what a weenie you are, because they need mails from a contractor that the corporate top security firewall won't let pass because they are deemed insecure attachments and how the hell they're now supposed to work.
Need I go on?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Several others have already said that SysAdmins are only as good as the rules and management that constrains them. Then again, there is the personality issue.
I am a SysAdmin myself like many on Slashdot. However, I do SysAdmin work on two different levels. At my day job, I manage gigantic enterprise class data systems with clustered servers for everything from distributed processing to my Oracle 10g RAC cluster. However, I also do work on the side in my spare time for small businesses and friends in the area. I do everything from some simple web development to distributed networks for file and application sharing. I've been given compliments and complaints but the compliments far outweigh the complains.
What I have heard most and that I like to hear is that people like to deal with me. They like to have me answer thier help desk calls because they know it will get fixed correctly and as fast as humanly possible. I like having that reputation and professional respect. Because of that, I don't have to fight with a user or management when I say I need time to figure out an issue or stand up a system. Does that make me a good SysAdmin? I dunno. I think it makes me a good employee. Then again, I get the same compliments from my small business customers and friends who would rather call me for help with their DSL account or a piece of troublesome software than any help line.
Given that, I think that a SysAdmin is an employee just like everyone else. Because of that, we shouldn't be venerated above others even though we are an employee with a special job. A SysAdmin allows other employees to be productive. If the SysAdmin isn't doing the job they have to do, then the company as a whole suffers. I suppose this is where the 'root is god' can get out of hand. When an entire company's infrastructure depends on the work of a few people, that's a high stress deal. Sometimes it gets to people. Bottom line though, we are all employees and just like the loud guy at the water cooler that nobody wants to hang around with, if we aren't profession and approachable like other employees, we are hurting ourselves. SysAdmins have to be computer geniuses, we have to be business oriented, we have to be people people and we have to be avaialable and approachable. It is not an easy task, believe me, I know! However, we all need to have a certain degree of professionalism when dealing with our customer base (users). We SysAdmins are our own downfall. The poor perception by the slobbering masses of users is our own fault. We can change it. While we do understand that our companies would not survive without us, it is not our place to make it so painfully obvious. The users don't care how great we think we are or even how great we are. They just want thier problems fixed quickly so they can get back to being how great they are. If we can just appease that desire from the users, I think that's what would make a good SysAdmin.
If you screw up your computer, you have no backlash, no negative repercussions, nothing to teach you not to do that again. Being a sysadmin now, and a power user before, I have to say this is the problem. People screw up, this is fine, problems arise, we fix them, that is our job. But when Joe user or Bob in management keeps doing the same stupid things, over and over w/ no negative repercussions, and it's your fault, and you have to fix it, and you can't do anything to them to teach them a lesson why NOT to do stupid shit, you stop caring.
Sysadmins are under appreciated, and expected to work miracles, w/ foolish users, you get the perception of a "Bad sysadmin". Want to fix the problems? Make Joe user who hosed his system for the 4th time this week, downloading a buncha crap and clicking on every virus he gets, do his job w/ out his computer. his deadlines are the same, but on our side, every time you screw up from the same mistake we've warned you about, our time to fix is going to double.
just my $.02
Yep, taht is the problem. That is right, you are the problem.
This is generally because most places have rules against users installing apps on their own.
They are constrained by management and good administration. If you are frustrated that you can't do something, either you need to take it up with the people who set up the rules or you need to rethink what you are doing, because it is going against policy.
It should not be hard if you do need it. You should be able to say "I can't do my job without it" and that should be that. If you can't do that, then you probably don't need it.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
This comes from hiring the cheapest person HR can find.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
Say hello. Even when you don't need something.
Don't question what he does all day
Fill out the stupid request form
Treat everything he does as a favor
Never forget he can read your email
What this boils down to is:
I guess people forget that SAs are people and employees too and that they work under constraints placed on them by upper management.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
this is often the beginning of the bad attitude seen coming from admins. A developer who has the option of blaming the admin for the failure ( while just finding a way to get it done ) can be endlessly frustrating. Obviously the problem with the app is a problem with the server, as the app is running on the server. This will lead to admins asking for the detailed instructions as an act of self preservation. If your admins are asking for these specific details on how you want your system and environment, and requesting a document for these details, the admin has likely been burned and wants to have something to point to when the developer needs a scapegoat at the deadline.
As a person who worked my way through school waiting tables and bartending, then dropping out in the *typical* SysAdmin fashion, I tend to liken my department's level of service to the level of service one would receive at a restaurant.
Heavily corporate restaurants make their customers sit through a whole song and dance about the restaurants offerings and their associated flair. Heavily corporate IT makes their customers (fellow employees or clients) wade through a song and dance about red tape and process.
Mom and Pop restaurants allow more freedom in day to day management of the customer experience, likewise startups do the same.
Greasy spoons with the head waitress who can run the floor and cook the food and do the dishes while balancing the books do it the head waitresses way...
You can draw the parallels anyway you like.
The real point is, as a SysAdmin, I try to keep in mind that me and my department are providing a service to our clients in whatever way, shape or form you want to define them. Without clients, while there may be considerably more time for Nethack and Slashdot posting, there would be no job.
I agree that 99% of the "I'm a sysadmin 'cause I run linux at home" crowd have gross delusions of competency.
But that final 1%....
The bottom line is somebody with a bit of skill and motivation can learn things at home that they could never dream of at work, precisely because nobody gives a damn if the network is down for a week. I would be laughed out of the office if I suggested a pilot project on the main network with Kerberos authentication and applications, or switching apps to use LDAP authentication, or running a VPN on the internal network as a precaution against internal compromise. But I've done all of them at home and learned a lot of the pros and cons. It's not the same as anyone who's used these tools at work, but there are a lot of well-experienced sysadmins with even less experience out there. And even the work-seasoned sysadmins might have only used one or two tools instead of trying every server supported by their distro.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
I think Woz's post tells us what a lot of us already know: Just because you're "technical" doesn't mean you can be a "high-end" administrator or understand the difficulties/nuances of "scaling up".
It reminds me of many, many, many conversations I've had with programmers, qa, etc, over the years where they tell me what they perceive to be the solution to the problem without really understanding either the long term impact or other factors. [I'm sure we've all heard the "disks are cheap" line when someone has filled their home directory with crap.]
You've hit on a few of the key points in your post. However, let me address your complaint about sysadmins not liking people trying new things. In general, we admins LOVE new things! Take a look at all the gadgets around me or software installed on my workstations, and you'll see the truth in that.
However, there's also a supportability issue. If I have five users I'm responsible for, then I'll happily accept five different machines. If I have 30 users, then I don't want 30 different builds and application bundles. If I have 500 users (or even 100), then I cannot AFFORD to have variance between machines, if I'm expected to support them.
You want a program installed? If I'm going to install it, then I will have to make sure it won't interfere with the existing software, and then I have to keep track of the fact that your machine is different than anyone else's. If someone else wants a different program installed, same problem, squared. Alternatively, I can give you admin access to your workstation or laptop, but then I can't guarantee anything about that machine anymore, and can't support it.
The third alternative is to put in a formal request to have the software added to the official bundle, or at least put on an 'allowed/approved' list. That's the best solution, but also the most onerous, bureaucracy-laden, time-intensive one, as you well know.
Mostly, it's a matter of (a) scale, (b) supportability, and (c) accountability. If your system is strange and nonstandard then when it breaks it's easier to say, "it's " than explain the reasoning behind, "because you have installed, I can't help you."
I feel your pain, but there is some valid reason behind it.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
I work as a System Support Engineer. My job is to sit behind a phone and computer, visually try to hold multiple plant or company infastructures in my mind with out ever viewing anything and step through Sys-Admins, DBAs, Engineers, and others through finding their root problem and providing a solution.
I've noticed a signficant break in those born after 1981 in terms of 'how' people understand computers. Not so co-incidently, those born after 1981 are also called Gen-Y or Millinials.
I have noticed that the following generations really don't understand computers, electronics, or any of the like. They know how to use them, they might learn some rules to follow. But to them, all this technology is simply like the microwave. I've had 'techs' try to 'clarify' with me that you know when your moving a file you can see the paper moving over from on folder to the other as well as all other sorts of rediculus ways of understanding the device they are working on.
How to these people can you possibly explain to them the source of the problem is a bad router that is tossing out a few bad packets once and a while at plant A, while their software is giving them a garbled message at plant B. Let alone stepping through the root problem.
These people can multitask like no other, text message 20 people 20 different messages at once; however, get them to think? That's another story. It's not about inexperience completely, it's about not studying and not understanding. I rather take the guy who has never heard of Solaris, and put them incharge of a Solaris machine who really understands computers then a person who has worked with Solaris for a year or two who ultimately doesn't understand the machine they are using.
Well, a good admin should have plenty of time to read /. because they have a smooth running system, with scripts set up to do repetative tasks, and are really only there to put out fires and work on ongoing projects.
Don't blame me, I voted for Kodos
You said it. I was one of two Unix SAs supporting a few dozen servers for which several hundred users depended for their jobs. If something went wrong, they called and, just like magic, things were fixed. They loved us and they loved the application. The worst thing that could happen would be a server death and when that happened, we'd call up the manageer of the affected group, ask them to have their people save their work locally and sit tight. Out of the closet would come a pre-configured replacement server. We'd plug it in, restore data from one of our three redundant back-up systems, and have those users up and running again in two hours, max.
I loved the work. Absolutely loved it. Because this was a government job with generous paid leave when one of us would be gone, having two of us meant there was always coverage and no downtime. Given that our users brought in 10s of millions of dollars a month, we were a paltry and perfectly justifiable expense.
Our problem was that nothing ever went wrong. Our big 'ol rack of servers hummed along with no drama and whenever the boss dropped by, he'd likely see us plodding through something routine like adding a user or checking system capacity reports. Every few days, we'd get bored and actually walk around the cube farm of the users, stick in our heads, and ask if everything was ok, can we do anything to make things work better? Our users loved us; our bosses didn't even seem to know what to write on our evaluations.
The Windows servers on the other side of the datacenter? Holy Cow, did those guys have the drama! Things were crashing all the time (We're back in the early NT days, mind you.) Whole populations of users suffered critical amounts of downtime. The admins put everything back together, of course, and were lauded as heroes because they had fixed the big, bad problems that had killed so many people's productivity for so long. They were HIGHLY visible to management. They got awards for fixing things. They were heroes.
Us Unix admins were those two people who sat over in the corner and never seemed to actually, visibly do anything.
You can see where this is leading, right? The Windows server side and the Windows front-line support side needed warm bodies, so I got thrown off Unix and into a GUI world I neither wanted nor understood. (Don't get me wrong, I've done the Windows work for years and I love helping people, but I'm not in love with the OS I now use and support.) Later, the other SA was tossed and our servers virtualized on mainframes. The number of SAs was cut to the bone and beyond. Virtualization was a nice concept and it works fine, but getting something fixed when it breaks is now a major red tape experience for our poor (former) users.
Fires to put out mean that firemen get chances to become heroes. Safety engineers who inspect your business and show you how clean the grease traps so nothing actually catches on fire are just needless expenses to be cut as soon as possible.
The moral is: Be a fireman. I figure they get more women, anyway.
The people that might find me antisocial are:
See, the above make me seem pretty grumpy, right? But the truth is that most days are fairly pleasant for both myself and (judging from feedback) for my "clients." However, there are always a few people that can magically manage to rain on a sunny day. Secretaries are often both the best and the worst. Some are obviously in their job because of wonderful PR skills, and manage to be extremely friendly, and, more refreshingly, honest (they can admit when they have messed something up, or don't know how something works). They also often have candies on their desks
But trust me, anyone can have a bad mood after being 2-3h late for lunch and when running a full day without breaks.
The real moral is that if you want to be a valuable geek, you have to learn enough people skills to make sure other people know. I've got a couple of decades of professional experience under my belt and am an expert in several areas, but the most valuable experience I've had professionally comes from working in a large company with a good number of untrained monkeys.
There are a lot of people who can't tell the difference between a seasoned professional and someone who would have bought a computers for dummies book if they were literate. Some of these people will be promoted into management to keep them out of the way of people doing the work. Being able to interact with people on their level is an incredibly valuable skill. It's nice to work with intelligent people who know what you do, but not everyone gets that kind of dream job. Basic communication skills are important, even if you feel like a retard when you're doing what is expected. If you don't feel like a retard, you're probably not going to effectively communicate with the business people. =)
For example, I'll send out emails to users, managers and the VP to let them know that a disk on the EMC failed, switchover to one of the hot spares occurred without incident, the failed disk was replaced and transitioned back into the array without issues and with no more than negligible performance degradation to the systems and users. No data was lost and we're back up and running. This happens once or twice a year.
If you know anything about EMC arrays, storage systems in general, or how to get your VCR to stop blinking 12:00, you probably realize that I didn't really have to do anything other than be aware that something happened and let the field service technician do his job. I've spent my whole career learning about technology so I am perfectly capable of doing all of the maintenance myself, but in this kind of case, I just need to let someone else do their job. This is not exactly rocket science here. However, people who don't get the technology see something like this and think "Huh, I guess something broke and now it's fixed and everything's good. Good thing he knows what to do because I wouldn't even know who to call or what to say to them." Most of the people whose opinions matter have no idea what you do.
There are a lot of arsonist-firefighter types in IT. You can be just as valuable as them without losing any shred of decency as a human being. Just let people you help know to let your boss, your boss's boss, their boss, and anyone else they know how incredibly helpful you were. Chances are that they asked for your help because they needed you to do 10 minutes of work so they could avoid trying to spend weeks trying to figure it out themselves and making it much worse before it got to you. Most people will be willing to spend 60 seconds to send a quick email to help you out.
There are a lot of bad sysadmins out there and there's no good way to tell them apart, from a manangment prospective.
A bad sysadmin looks very busy all of the time so management and co-workers think that they are busy and important.
Things are always breaking and they come to the rescue. Things are down for days and through their heroic efforts, (cough, reinstall Windows, cough) things are back working.
While good sysadmins are proactive and very little breaks or goes wrong. They remain calm during user's crisises (because panic never fixed anything). They are seldomly seen by co-workers and management. They do things like scheduling down time when the system is working fine or nagging users to do thinks a "better" way when the old way worked fine for the users.
No one see them fixing much, and nothing ever breaks, and the network is never down, so they must no be very important or valuable to the company.
Competition Good, Monopoly Bad.
They come up with policies like "only use IE 5" at a computer security company and withhold privileges from people who know way more then they do, and block all security patches.
First, if they come up with policies like "use IE5", they're not really sysadmins. Sysadmins don't set policies on Windows machines, particularly desktops. Sysadmins administer servers, and "real" servers don't usually run Windows (even then, they don't have policies dictating the browser to be used). Workstation people are "help desk" or "Microsoft Certified Professionals" or something other than sysadmins.
Second, every time I've heard someone complain that they know more than the admin above them, it's always proven out to be false. "Well, I set up my Linksys wireless router at home to use a different network name than Linksys, I know as much as that stupid Cisco Engineer who can't even get all of my porn sites to behave at work". Or "I installed Linux From Scratch, I know way more about Unix in general than the people who designed the network running 10,000 HP-UX machines". Yeah, right. Welcome to the land of no credibility, populated by thousands of end-users just like you, who don't even know enough to realize what you don't know. The neighboring town is full of the users who always say "I didn't do anything, the file just disappeared", even if you watched them drag the file to a different folder / press the delete button / click "ok" without reading the message - many of them have weekend condos in the land of no credibility.
Sounds like you had some bad sysadmins, sure. Incompetent people might obtain/attain the job title of sysadmin, but they're not really more than "company computer guy" (much like I've had "engineer" in my title before, despite not finishing an Engineering program). Based on the tone of the post, though, I'm fairly confident that you would've eventually had bad experiences even with a good sysadmin.