Slashdot Mirror
Are Sysadmins Really that Bad?
tgbrittai asks: "According to Paul Boutin they are merely an obstacle to be manipulated or outmaneuvered. According to Steve Wozniak they are pimps. I've known my share of good and bad sysadmins, programmers and every other professional role out there, and I have to wonder: are sysadmins really THAT bad?"
Most times sys-admins are overworked and underpaid and have to deal with users who take advantage of their local IT person, tasking them to fix systems that they callously break. Others are truly worth the name "Bastard Operators from Hell". How would you rate your sys-admin and what things did you have to do to make things run smoothly (or not)?
....when he suggests "Treat everything he does as a favor. ". Actually, that's not a bad life strategy - when the waitress refills your coffee right away, treat it as if she didn't really have to - because, really, she didn't! She could have just ignored your empty cup, or waited a few minutes, or whatever.
Same with a sysadmin. When he adds a rewrite rule (done!) 20 seconds after you ask for it, act appreciative and say thanks, even though that's his job. Because he could have put it off until tomorrow and probably would have reasonable excuses for doing so. (Incidentally, I hosed up this rewrite rule the first time by leaving off the trailing $. Doh!)
The Army reading list
Don't cry me a river about that...with the exception of upper echelons of management, I'd say most people do more for the company than they get back as a reward for their work.
I've been on both sides of the fence, I've seen users that put every piece of software they can find on their machine, then come calling when they break. I've been blamed for doing something to break a printer, about two weeks after I was there to swap a monitor.
On the flip side, I've worked in places with a tiny server share to store important data and an IT staff that doesn't really guarantee it'll be backed up. So we ended up having to work around the IT staff in a lot of things. It was easier to cobble together something that we can guarantee is backed up AND that has enough space for us than to go through the reams of paperwork to get more space and justify some sort of improved SLA.
In fairness to the IT folks though, a lot of the people working IT are just trying to feel their way through the system that was put in place before they started, and they think it's just as stupid as the end users. But they lack the power to change it, and their bosses don't want to.
"It is a miracle that curiosity survives formal education." -Albert Einstein
Replace "X" with any profession, and the answer is the same: some are and some aren't. The professions with high barriers to entry (i.e. medicine) tend to root out some if not most of the incompetents or otherwise poorly qualified, but some will still slip through. The same is true of sysadmins. They obviously exist for reason -- maybe the article writer should ask, "What would a world without sysadmins look like?" For large organization, the answer is "chaos," and they would quickly re-implement the same positions now being mocked.
I dont disagree with your statement here, necessarily, but it does sound to me like an issue of pointing out a problem, without really offering any notions for improvement. You say that new sysadmins are too "inexperienced" and dont necessarily know enough about Legacy systems. Well... how would they? I mean, if you've worked on any mainframe systems, you'll know that knowing one set of commands doesnt do you ANY good on the very next mainframe you might be forced to work on. And how exactly does one become an "experienced" sysadmin? Go to sysadmin school? Really? Sign me up!
It seems to me that things are the way they are because... well, they have to be. When an old sysadmin leaves, you're not going to be able to replace him with someone who knows everything about your current infrastructure, and happens to have niche knowledge of all of your various legacy machines. If such a person exist, chances are very high that they're currently still employed somewhere else, or are about to retire. Employees dont stay in circulation forever. Eventually mass amounts of experience starts falling out of the market, and has to be replaced with "noobs".
Im not saying huge companies should necessarily be hiring inexperienced sysadmins. But someone has to, or inexperienced sysadmins can NEVER become experienced sysadmins. Im fortunate, in that I was hired on as a sysadmin at a University, during a complete infrastructure rebuild. So while Ive been forced to learn a passing familiarity with the mainframe systems, it's mostly been to help usher them out entirely. And Ive been, for the most part, at liberty to build the new infrastructure around her to my own personal standards and benefits, meaning Ive got a pretty good grip on things. Gradually, I run into problems that I cant solve with a simple script, so Ive been forced to learn things like sed and awk, as you mentioned, more and more over time. And even those, btw, arent a universal solution, especially if old IBM-era mainframes are involved.
Even if what you're saying is the problem, if sysadmins with "not enough" experience for a particular job are being thrown into them... there's no real solution for that. I mean, if you draw up a requirement for all of the systems you want a sysadmin to know, chances are NOT good that you're going to find someone who A. Meets all the requirements, including experience with all of your legacy systems at your company B. Lives nearby or is willing to relocate to where you are and C. Is looking for a salary exactly where you're offering it.
Pay no attention to the systems administrator part of my job title...it's just a standard honorific. >_>
Before I launch into this, it really seems like they define good and bad by their customer service skills, so that's what I'm addressing by "good" and "bad", not so much their technical knowledge.
In my experience, the problems with sysadmins tends to be that with the ones that lack the ability to understand the user. This is what people refer to as the "IT mindset" where the user is the enemy and is doing whatever they can to make IT's life more difficult. In some cases, this is very true. There ARE abusive users out there. However, most people simply want to do their job, and their job is NOT getting these machines to work right. Getting back to the "understanding the user" thing, I find a great many sysadmins have no empathy for how a user feels when their machine has gone down, and why would they? When has a sysadmin ever really felt the panic and/or frustration of having a machine crash and not having the first clue of how to fix it? We KNOW what we should do, and while we'll be annoyed at the extra work, we're (hopefully) never flailing around blindly...or if we are we're careful never to show signs of it. A user's machine goes down and they have no idea what to do. They panic, they worry, they don't think logically...they immediately run to the nearest person who they think can help them and oftentimes get the look of "Why should I?" or "Can't you see I'm busy right now?"
Again, that doesn't mean there aren't people who don't actively try to bypass what they SHOULD be doing to get the problem they caused looked at immediately because they think they're more important. However, I think the sysadmins that most people complain about are the ones who let the handful of lazy/abusive users jade their dealings with the ones who simply want to do their job and go home.
However, I find that the "bad" sysadmins are about as common as the truly abusive users. They stand out in your memory so it seems like there's a lot of them, but they're actually far from the rule. YMMV, of course. After all, in the course of a day three or four people might stop to hold open a door for you, but the one you remember at the end of the day is the idiot that cut you off on the highway. Human memory is a funny thing...
This is spot on. I am that sysadmin you talk about. i work for a large software company - however, I was hired specifically to support one smaller office, as the IT group couldn't (or wouldn't) provide adequate support. Things were great for a while - I learned the quirks of this particular office, setup new systems and generally made the work environment better for those that work out of this office.
Thats when a new manager, and IT overlords stepped in. Now I have to do everything 'by the book', even when 'the book' doesn't mesh with what we are doing here at this satellites office. My life is now a hell of process, procedure, and meetings - and very little actual work is getting done.
What does this lead to? Developers going 'out of band' to get stuff done - purchasing hardware on credit cards, not using authorized apps, copying large files around the WAN when stuff should be local, etc. All because they can't get a slice of my time to help them with a correct solution.
Everyone here is frustrated - myself the most. I *want* to provide the best support I can, but I'm now hamstrung by process and management, whereas before (when the developers/local managers were happy) I wasn't.
I think most sysadmin jobs are going this route now, excepting the startups (and they will, as they grow). Sysadmins are a commodity now, they aren't viewed as adding value.
Admins are literally wedged between workers and management.
... umm ... his quarter years report (yahu, sure), and if it isn't reactivated IMMEDIATELY, you're in deep dung.
You, as an admin, get orders from management how they envision the network security to be. You know it doesn't work that way and will only create an obstacle for the people you're to protect, but you will do it anyway. Because the guy you knew from the day shift one day took one such memo and trotted upstairs to the brass.
He hasn't been seen since.
So you do what you're ordered, block non-corporate mail accounts, block porn sites, block ebay, block... everything. This is usually when one of the middle managers complains that he can't go online anymore, which turns out as him being unable to access ebay anymore which he needs for
It escalates up to the top brass, you get said pile of manure onto your head for not cooperating with middle management and you now have to work out a plan how to block ebay without blocking it. Sounds impossible? I know that. You go upstairs and tell the brass. Can I have your stuff?
Then you head down to the cafeteria for some coffee. Coffee good. Coffee lifeblood. My precious. But you forgot your fake moustache and the noseglasses, so people immediately recognize you and start asking what's wrong and why they can't access gmail and gmx anymore. You explain the brass note. Which causes them to tell you in no uncertain terms what a weenie you are, because they need mails from a contractor that the corporate top security firewall won't let pass because they are deemed insecure attachments and how the hell they're now supposed to work.
Need I go on?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Several others have already said that SysAdmins are only as good as the rules and management that constrains them. Then again, there is the personality issue.
I am a SysAdmin myself like many on Slashdot. However, I do SysAdmin work on two different levels. At my day job, I manage gigantic enterprise class data systems with clustered servers for everything from distributed processing to my Oracle 10g RAC cluster. However, I also do work on the side in my spare time for small businesses and friends in the area. I do everything from some simple web development to distributed networks for file and application sharing. I've been given compliments and complaints but the compliments far outweigh the complains.
What I have heard most and that I like to hear is that people like to deal with me. They like to have me answer thier help desk calls because they know it will get fixed correctly and as fast as humanly possible. I like having that reputation and professional respect. Because of that, I don't have to fight with a user or management when I say I need time to figure out an issue or stand up a system. Does that make me a good SysAdmin? I dunno. I think it makes me a good employee. Then again, I get the same compliments from my small business customers and friends who would rather call me for help with their DSL account or a piece of troublesome software than any help line.
Given that, I think that a SysAdmin is an employee just like everyone else. Because of that, we shouldn't be venerated above others even though we are an employee with a special job. A SysAdmin allows other employees to be productive. If the SysAdmin isn't doing the job they have to do, then the company as a whole suffers. I suppose this is where the 'root is god' can get out of hand. When an entire company's infrastructure depends on the work of a few people, that's a high stress deal. Sometimes it gets to people. Bottom line though, we are all employees and just like the loud guy at the water cooler that nobody wants to hang around with, if we aren't profession and approachable like other employees, we are hurting ourselves. SysAdmins have to be computer geniuses, we have to be business oriented, we have to be people people and we have to be avaialable and approachable. It is not an easy task, believe me, I know! However, we all need to have a certain degree of professionalism when dealing with our customer base (users). We SysAdmins are our own downfall. The poor perception by the slobbering masses of users is our own fault. We can change it. While we do understand that our companies would not survive without us, it is not our place to make it so painfully obvious. The users don't care how great we think we are or even how great we are. They just want thier problems fixed quickly so they can get back to being how great they are. If we can just appease that desire from the users, I think that's what would make a good SysAdmin.
Yep, taht is the problem. That is right, you are the problem.
This is generally because most places have rules against users installing apps on their own.
They are constrained by management and good administration. If you are frustrated that you can't do something, either you need to take it up with the people who set up the rules or you need to rethink what you are doing, because it is going against policy.
It should not be hard if you do need it. You should be able to say "I can't do my job without it" and that should be that. If you can't do that, then you probably don't need it.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
This comes from hiring the cheapest person HR can find.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
As a person who worked my way through school waiting tables and bartending, then dropping out in the *typical* SysAdmin fashion, I tend to liken my department's level of service to the level of service one would receive at a restaurant.
Heavily corporate restaurants make their customers sit through a whole song and dance about the restaurants offerings and their associated flair. Heavily corporate IT makes their customers (fellow employees or clients) wade through a song and dance about red tape and process.
Mom and Pop restaurants allow more freedom in day to day management of the customer experience, likewise startups do the same.
Greasy spoons with the head waitress who can run the floor and cook the food and do the dishes while balancing the books do it the head waitresses way...
You can draw the parallels anyway you like.
The real point is, as a SysAdmin, I try to keep in mind that me and my department are providing a service to our clients in whatever way, shape or form you want to define them. Without clients, while there may be considerably more time for Nethack and Slashdot posting, there would be no job.
I think Woz's post tells us what a lot of us already know: Just because you're "technical" doesn't mean you can be a "high-end" administrator or understand the difficulties/nuances of "scaling up".
It reminds me of many, many, many conversations I've had with programmers, qa, etc, over the years where they tell me what they perceive to be the solution to the problem without really understanding either the long term impact or other factors. [I'm sure we've all heard the "disks are cheap" line when someone has filled their home directory with crap.]
You've hit on a few of the key points in your post. However, let me address your complaint about sysadmins not liking people trying new things. In general, we admins LOVE new things! Take a look at all the gadgets around me or software installed on my workstations, and you'll see the truth in that.
However, there's also a supportability issue. If I have five users I'm responsible for, then I'll happily accept five different machines. If I have 30 users, then I don't want 30 different builds and application bundles. If I have 500 users (or even 100), then I cannot AFFORD to have variance between machines, if I'm expected to support them.
You want a program installed? If I'm going to install it, then I will have to make sure it won't interfere with the existing software, and then I have to keep track of the fact that your machine is different than anyone else's. If someone else wants a different program installed, same problem, squared. Alternatively, I can give you admin access to your workstation or laptop, but then I can't guarantee anything about that machine anymore, and can't support it.
The third alternative is to put in a formal request to have the software added to the official bundle, or at least put on an 'allowed/approved' list. That's the best solution, but also the most onerous, bureaucracy-laden, time-intensive one, as you well know.
Mostly, it's a matter of (a) scale, (b) supportability, and (c) accountability. If your system is strange and nonstandard then when it breaks it's easier to say, "it's " than explain the reasoning behind, "because you have installed, I can't help you."
I feel your pain, but there is some valid reason behind it.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
You said it. I was one of two Unix SAs supporting a few dozen servers for which several hundred users depended for their jobs. If something went wrong, they called and, just like magic, things were fixed. They loved us and they loved the application. The worst thing that could happen would be a server death and when that happened, we'd call up the manageer of the affected group, ask them to have their people save their work locally and sit tight. Out of the closet would come a pre-configured replacement server. We'd plug it in, restore data from one of our three redundant back-up systems, and have those users up and running again in two hours, max.
I loved the work. Absolutely loved it. Because this was a government job with generous paid leave when one of us would be gone, having two of us meant there was always coverage and no downtime. Given that our users brought in 10s of millions of dollars a month, we were a paltry and perfectly justifiable expense.
Our problem was that nothing ever went wrong. Our big 'ol rack of servers hummed along with no drama and whenever the boss dropped by, he'd likely see us plodding through something routine like adding a user or checking system capacity reports. Every few days, we'd get bored and actually walk around the cube farm of the users, stick in our heads, and ask if everything was ok, can we do anything to make things work better? Our users loved us; our bosses didn't even seem to know what to write on our evaluations.
The Windows servers on the other side of the datacenter? Holy Cow, did those guys have the drama! Things were crashing all the time (We're back in the early NT days, mind you.) Whole populations of users suffered critical amounts of downtime. The admins put everything back together, of course, and were lauded as heroes because they had fixed the big, bad problems that had killed so many people's productivity for so long. They were HIGHLY visible to management. They got awards for fixing things. They were heroes.
Us Unix admins were those two people who sat over in the corner and never seemed to actually, visibly do anything.
You can see where this is leading, right? The Windows server side and the Windows front-line support side needed warm bodies, so I got thrown off Unix and into a GUI world I neither wanted nor understood. (Don't get me wrong, I've done the Windows work for years and I love helping people, but I'm not in love with the OS I now use and support.) Later, the other SA was tossed and our servers virtualized on mainframes. The number of SAs was cut to the bone and beyond. Virtualization was a nice concept and it works fine, but getting something fixed when it breaks is now a major red tape experience for our poor (former) users.
Fires to put out mean that firemen get chances to become heroes. Safety engineers who inspect your business and show you how clean the grease traps so nothing actually catches on fire are just needless expenses to be cut as soon as possible.
The moral is: Be a fireman. I figure they get more women, anyway.
The real moral is that if you want to be a valuable geek, you have to learn enough people skills to make sure other people know. I've got a couple of decades of professional experience under my belt and am an expert in several areas, but the most valuable experience I've had professionally comes from working in a large company with a good number of untrained monkeys.
There are a lot of people who can't tell the difference between a seasoned professional and someone who would have bought a computers for dummies book if they were literate. Some of these people will be promoted into management to keep them out of the way of people doing the work. Being able to interact with people on their level is an incredibly valuable skill. It's nice to work with intelligent people who know what you do, but not everyone gets that kind of dream job. Basic communication skills are important, even if you feel like a retard when you're doing what is expected. If you don't feel like a retard, you're probably not going to effectively communicate with the business people. =)
For example, I'll send out emails to users, managers and the VP to let them know that a disk on the EMC failed, switchover to one of the hot spares occurred without incident, the failed disk was replaced and transitioned back into the array without issues and with no more than negligible performance degradation to the systems and users. No data was lost and we're back up and running. This happens once or twice a year.
If you know anything about EMC arrays, storage systems in general, or how to get your VCR to stop blinking 12:00, you probably realize that I didn't really have to do anything other than be aware that something happened and let the field service technician do his job. I've spent my whole career learning about technology so I am perfectly capable of doing all of the maintenance myself, but in this kind of case, I just need to let someone else do their job. This is not exactly rocket science here. However, people who don't get the technology see something like this and think "Huh, I guess something broke and now it's fixed and everything's good. Good thing he knows what to do because I wouldn't even know who to call or what to say to them." Most of the people whose opinions matter have no idea what you do.
There are a lot of arsonist-firefighter types in IT. You can be just as valuable as them without losing any shred of decency as a human being. Just let people you help know to let your boss, your boss's boss, their boss, and anyone else they know how incredibly helpful you were. Chances are that they asked for your help because they needed you to do 10 minutes of work so they could avoid trying to spend weeks trying to figure it out themselves and making it much worse before it got to you. Most people will be willing to spend 60 seconds to send a quick email to help you out.
And that seems to me to be the key thing: inexperienced people need to work along side experienced people. That's how people get experience.
Why is this such a hard concept for people to understand these days? It's like there are two camps: either you think companies should hire all-knowing experienced geniuses or you think companies are better off hiring a small army of inexperienced guys.
Throughout history, in pretty much every trade, there's been this idea of apprentices. There's been the idea of "working your way up the ladder". The idea is pretty simple: you put the new guy in with the experienced guy, so that the experienced guy can pass on his knowledge and the new guy can get up-to-speed. Over time, the new guy learns enough to take over the small/easy portions of the experienced guy's work. The experienced guy gets to avoid the crap-work, and the new guy gets experience. Over more time, the new guy starts becoming an experienced guy, can take on more complicated problems, the experienced guy can keep focusing on higher and higher-level problems, and it keeps building on that model until either the old experienced-guy or the new experienced-guy move on to something better.