Slashdot Mirror
Are Sysadmins Really that Bad?
tgbrittai asks: "According to Paul Boutin they are merely an obstacle to be manipulated or outmaneuvered. According to Steve Wozniak they are pimps. I've known my share of good and bad sysadmins, programmers and every other professional role out there, and I have to wonder: are sysadmins really THAT bad?"
Most times sys-admins are overworked and underpaid and have to deal with users who take advantage of their local IT person, tasking them to fix systems that they callously break. Others are truly worth the name "Bastard Operators from Hell". How would you rate your sys-admin and what things did you have to do to make things run smoothly (or not)?
Are you trying to get us in trouble?! It's a damn good thing I'm a subscriber, I managed to block slashdot in our squid cache and drop it in a dns blackhole just before this story went live.
Trolling is a art,
End of Thread.
From woz.org:
"If my son wants to be a pimp when he grows up, that's fine with me. I hope he's a good one and enjoys it and doesn't get caught. I'll support him in this. But if he wants to be a network administrator, he's out of the house and not part of my family. I tell this joke a lot. Once, a teacher told me that she tells the same one but for a 'teacher'."
....when he suggests "Treat everything he does as a favor. ". Actually, that's not a bad life strategy - when the waitress refills your coffee right away, treat it as if she didn't really have to - because, really, she didn't! She could have just ignored your empty cup, or waited a few minutes, or whatever.
Same with a sysadmin. When he adds a rewrite rule (done!) 20 seconds after you ask for it, act appreciative and say thanks, even though that's his job. Because he could have put it off until tomorrow and probably would have reasonable excuses for doing so. (Incidentally, I hosed up this rewrite rule the first time by leaving off the trailing $. Doh!)
The Army reading list
Quality of sys admin is inversely proportional to the number of rules they have to work under.
The more red tape an admin has the worse the actual results they will provide
When you take a good sys admin, tell them what you want, give them a sensible budget and ask them to delvier it you will frequently get a great system.
When management try to micro manage, heap them with rules, specify particular components because they read an artical that described it as good or the vendor took them out to lunch you will get problems - lots of them.
Right now I work in a very large bank and some days I think the admins could not find their rear end with both hands and a man page - I've never met them persoanlly so no idea what they are actually like. From otehr friends I have working in banking I know how much red tape they have to work under and I suspect half of the problems the user end sees (bear in mind I'm an ex admin myself and now developer) are caused by the red tape, not by the admin.
Virtual break glass on the root password? 2 weeks aproval before changing anything, even if its trivial? These are the kind of things that can drive an admin insane.
Last company I worked for was a start up - a great place to work for a short period, the admin their was very competant on solaris, windows and linux, had a great system implemented. It didn't start going downhill til a new CEO came in that started to micromanage him (and everyone else). Thats why I got out, same for a few others, the sys admin is leaving when he can find something else he wants to do. Still even with all the hassel he had I still got great results from him, mainly because I respected his limitations, didn't break things, knew what I was doing and helped him out when he needed it. Sales staff on the other hand? With them if it wasn't explicitly on his supported list he;d tell them to take a running jump - because of all the hassel they caused breaking things (the same way repeatedly), ignoring instructions, using unsupported devices or software and then wanting it fixed - and they wondered why he didn't want to help them?
Sys admins are human like the rest of us - overly managed they are stiffled, pissed off they are unhelpful - what else would you expect?
$_="Slashdotter";$syn="OTT";s;..;;;sub _{print shift||$_};s!ash!Perl !;s=$syn=ack=i;tr+LLEd+BLAH+;_"Just Another ";_
Don't cry me a river about that...with the exception of upper echelons of management, I'd say most people do more for the company than they get back as a reward for their work.
I've been on both sides of the fence, I've seen users that put every piece of software they can find on their machine, then come calling when they break. I've been blamed for doing something to break a printer, about two weeks after I was there to swap a monitor.
On the flip side, I've worked in places with a tiny server share to store important data and an IT staff that doesn't really guarantee it'll be backed up. So we ended up having to work around the IT staff in a lot of things. It was easier to cobble together something that we can guarantee is backed up AND that has enough space for us than to go through the reams of paperwork to get more space and justify some sort of improved SLA.
In fairness to the IT folks though, a lot of the people working IT are just trying to feel their way through the system that was put in place before they started, and they think it's just as stupid as the end users. But they lack the power to change it, and their bosses don't want to.
"It is a miracle that curiosity survives formal education." -Albert Einstein
And he is the best Boss ever. He even reads /., which should demonstrate how cool he is. He certainly wouldn't do anything bad, like access my computer, log on to Slashdot, post an article telling the world how awesome he is, and then give me a warning to secure my computer (and change my /. password).
You are reading a copy of my copyrighted post.
I currently work as a Network Admin for a large retail company. I started with this company as a store clerk, then moved to the helpdesk, while i was in college. This helped me learn patience and how to be polite with everyone no matter how annoying, wrong, irate the "customer"/"employee" is. I look at some coworkers who have no clue how to handle talking to a customer or a user needing help and give them lip every chance they get. I understand the frustrations of having petty work assigned to you by a VP level person that interrupts your day and workflow. All the time i have important time constrained projects interrupted by those "important" people who have to have some blackberry/treo/etc problem fixed asap. I have to drop whatever important task I am on and concentrate soley on the happiness of this one person. Ultimately thats what it comes down to i have found. Although i get my work done and i am thorough on all mky projects, I am not known for that, I am known for always being the nice guy who helps out the Execs and their exec assistants, and honestly that puts me in better light than anything else. Sometimes the interruptions are extremely frustrating, but when the execs are happy, everyone is happy.
Its not that there bad people. Most of them are pretty nice if you talk to them over lunch. The problem is they are so constrained by what they can do that they are very frustrating to work with. I must say they also hate working with me for the most part.
The problem is that I'm somewhat tech savy. The sys admin don't like anybody trying new things. Their management likes it even less. Do one little thing or install one little app and if you have problems your on your own. Doesn't matter if your laptop explodes, they'll blame it on VMWARE or whatever you happen to be running.
It wasn't always like this. In days of old the Sys Admin were local and reported into the same groups they supported. As such they knew what we were working on and would help out. Management would support this because it often lead to increased productivity or reliability. But at some point a bean counter decided we needed a corporate IT organization.
Once you decouple the support from the groups they support you end up with apathy and endless rules. Also to get the groups to try anything new you have to weave your way through a bureaucracy. You also end up with smaller and smaller IT groups because their contributions to the end product become harder and harder to trace. If a business unit needs to cut costs the first thing they look at is horizontal organizations outside their own structure. Its a lot easier to cut an outside IT guy than a developer working on a product.
Things look to be taking a turn for the worse. Some of our IT is now going to be out sourced. To me this is equivalent to saying I now fully support myself. I can just imagine trying to convince some contracted person in India that I really do need to have VMPlayer installed on my Windows laptop....
One should not theorize before one has data. -Sherlock Holmes-
I dont disagree with your statement here, necessarily, but it does sound to me like an issue of pointing out a problem, without really offering any notions for improvement. You say that new sysadmins are too "inexperienced" and dont necessarily know enough about Legacy systems. Well... how would they? I mean, if you've worked on any mainframe systems, you'll know that knowing one set of commands doesnt do you ANY good on the very next mainframe you might be forced to work on. And how exactly does one become an "experienced" sysadmin? Go to sysadmin school? Really? Sign me up!
It seems to me that things are the way they are because... well, they have to be. When an old sysadmin leaves, you're not going to be able to replace him with someone who knows everything about your current infrastructure, and happens to have niche knowledge of all of your various legacy machines. If such a person exist, chances are very high that they're currently still employed somewhere else, or are about to retire. Employees dont stay in circulation forever. Eventually mass amounts of experience starts falling out of the market, and has to be replaced with "noobs".
Im not saying huge companies should necessarily be hiring inexperienced sysadmins. But someone has to, or inexperienced sysadmins can NEVER become experienced sysadmins. Im fortunate, in that I was hired on as a sysadmin at a University, during a complete infrastructure rebuild. So while Ive been forced to learn a passing familiarity with the mainframe systems, it's mostly been to help usher them out entirely. And Ive been, for the most part, at liberty to build the new infrastructure around her to my own personal standards and benefits, meaning Ive got a pretty good grip on things. Gradually, I run into problems that I cant solve with a simple script, so Ive been forced to learn things like sed and awk, as you mentioned, more and more over time. And even those, btw, arent a universal solution, especially if old IBM-era mainframes are involved.
Even if what you're saying is the problem, if sysadmins with "not enough" experience for a particular job are being thrown into them... there's no real solution for that. I mean, if you draw up a requirement for all of the systems you want a sysadmin to know, chances are NOT good that you're going to find someone who A. Meets all the requirements, including experience with all of your legacy systems at your company B. Lives nearby or is willing to relocate to where you are and C. Is looking for a salary exactly where you're offering it.
Pay no attention to the systems administrator part of my job title...it's just a standard honorific. >_>
Before I launch into this, it really seems like they define good and bad by their customer service skills, so that's what I'm addressing by "good" and "bad", not so much their technical knowledge.
In my experience, the problems with sysadmins tends to be that with the ones that lack the ability to understand the user. This is what people refer to as the "IT mindset" where the user is the enemy and is doing whatever they can to make IT's life more difficult. In some cases, this is very true. There ARE abusive users out there. However, most people simply want to do their job, and their job is NOT getting these machines to work right. Getting back to the "understanding the user" thing, I find a great many sysadmins have no empathy for how a user feels when their machine has gone down, and why would they? When has a sysadmin ever really felt the panic and/or frustration of having a machine crash and not having the first clue of how to fix it? We KNOW what we should do, and while we'll be annoyed at the extra work, we're (hopefully) never flailing around blindly...or if we are we're careful never to show signs of it. A user's machine goes down and they have no idea what to do. They panic, they worry, they don't think logically...they immediately run to the nearest person who they think can help them and oftentimes get the look of "Why should I?" or "Can't you see I'm busy right now?"
Again, that doesn't mean there aren't people who don't actively try to bypass what they SHOULD be doing to get the problem they caused looked at immediately because they think they're more important. However, I think the sysadmins that most people complain about are the ones who let the handful of lazy/abusive users jade their dealings with the ones who simply want to do their job and go home.
However, I find that the "bad" sysadmins are about as common as the truly abusive users. They stand out in your memory so it seems like there's a lot of them, but they're actually far from the rule. YMMV, of course. After all, in the course of a day three or four people might stop to hold open a door for you, but the one you remember at the end of the day is the idiot that cut you off on the highway. Human memory is a funny thing...
I strongly agree with everything you're saying. One of the 20 unofficial roles I have at a large bank is Unix System Administrator. I really only spend ~100 hours/year doing system administration, and that's only to deal with something breaking. We have enough work for a full time sysadmin, but we have management who aim to consistently do less than the minimum. I believe the fundamental problem of system administration in any business environment is that you never see the benefit of good results. You only see costs of failures and people running around putting out fires all of the time. A good system administrator tends to work himself out of a justification for a job because there's no compelling business reason to keep employing someone expensive whose benefits to the organization are invisible. Coming in on the weekend to replace hardware, fixing things that break before people notice they are down and recovering files for people who will never admit that they deleted something important are all common sysadmin tasks that are rarely acknowledged.
Micromanagement and imaginary, perceived cost savings create unsustainable environments. Here in a non-technology group of a large bank, we've got a handful of Sun servers attached to an EMC. There are numerous persistent memory errors on the Sun's that could be fixed with a service call and a small scheduled downtime. Well, in a normal environment that is all it takes. However, we don't currently have a maintenance contract. We did have a service contract years ago when the problems started, but maintaining systems is an anti-goal for management - apparently there is no profit in keeping things running. The EMC has been performing well, with the occaisional disk failure that is completely invisible thanks to RAID and automatic call home to get a replacement disk sent out. That's been our key saving grace since we don't backup anything(including production servers).
Unfortunately, this kind of short sighted, unprofessional approach to IT is common in business driven organizations. When everything comes crashing down, as it always will given sufficient time, someone will look at what happened and try to prevent it from happening again. This is the kind of sabatage through mismanagement that leads to the creation of company policies that make it hard for anyone to do their job. Our company has policies that require that system, network, security and database administrators all be separate people. The developers have to be separate as well and can't have access to production systems. There's some very good reasons for all of these policies, but business people can't resist the temptation of hiring one person to do all of these jobs. After all, who better to get things working and fix problems than a developer with root access to everything. It sure cuts down on time wasted in getting authorizations and having meetings.
Several others have already said that SysAdmins are only as good as the rules and management that constrains them. Then again, there is the personality issue.
I am a SysAdmin myself like many on Slashdot. However, I do SysAdmin work on two different levels. At my day job, I manage gigantic enterprise class data systems with clustered servers for everything from distributed processing to my Oracle 10g RAC cluster. However, I also do work on the side in my spare time for small businesses and friends in the area. I do everything from some simple web development to distributed networks for file and application sharing. I've been given compliments and complaints but the compliments far outweigh the complains.
What I have heard most and that I like to hear is that people like to deal with me. They like to have me answer thier help desk calls because they know it will get fixed correctly and as fast as humanly possible. I like having that reputation and professional respect. Because of that, I don't have to fight with a user or management when I say I need time to figure out an issue or stand up a system. Does that make me a good SysAdmin? I dunno. I think it makes me a good employee. Then again, I get the same compliments from my small business customers and friends who would rather call me for help with their DSL account or a piece of troublesome software than any help line.
Given that, I think that a SysAdmin is an employee just like everyone else. Because of that, we shouldn't be venerated above others even though we are an employee with a special job. A SysAdmin allows other employees to be productive. If the SysAdmin isn't doing the job they have to do, then the company as a whole suffers. I suppose this is where the 'root is god' can get out of hand. When an entire company's infrastructure depends on the work of a few people, that's a high stress deal. Sometimes it gets to people. Bottom line though, we are all employees and just like the loud guy at the water cooler that nobody wants to hang around with, if we aren't profession and approachable like other employees, we are hurting ourselves. SysAdmins have to be computer geniuses, we have to be business oriented, we have to be people people and we have to be avaialable and approachable. It is not an easy task, believe me, I know! However, we all need to have a certain degree of professionalism when dealing with our customer base (users). We SysAdmins are our own downfall. The poor perception by the slobbering masses of users is our own fault. We can change it. While we do understand that our companies would not survive without us, it is not our place to make it so painfully obvious. The users don't care how great we think we are or even how great we are. They just want thier problems fixed quickly so they can get back to being how great they are. If we can just appease that desire from the users, I think that's what would make a good SysAdmin.
The first IT job I took after getting out of the army was answering phones on a help desk for a retail company. We had to support about 1,500 users.
Not long before I started that job, the company had hired a new "Director of End User Technology" and this guy was sharp. His primary goal at the time was to straighten out the cobbled together mess of a network that had haphazardly grown department by department. The place was a real mess and the network ran like mud.
Over a period of about four years, we standardized our PCs and laptops, physically consolidated the servers that were spread all over the HQ building, corrected the messed up cabling, centralized administration, built a training room and implemented a number of classes, etc. It was a truly exciting and fun place to work and virtually everyone who I started out with on the help desk eventually learned, got certifications and moved into administration and/or engineering. When I had started there we had a real mom & pop shop type feel and very little oversight. All we had to go on were some clearly defined goals and a directive to "get things fixed."
We consistently accomplished our goals. Within the first couple years we had fixed the network and made it into something useful. The consequence was more use by upper management and as you might expect, more management from upper management. Every time we met another goal, the more visibility we received. The more visibility we received, the more layers of management they installed above us. Every layer of management installed made it harder and harder to actually get anything done, basically because each new layer of management knew less about IT but more about "managing.".
I guess mostly I'm just whining here, but eventually most of us who had built the network quit. They 'managed' us right out the door.
Politicians are like diapers - they should be changed frequently and for the same reasons.
This comes from hiring the cheapest person HR can find.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
The solution is so painfully obvious that no one will ever implement it.
Business people need to look at more than one little line item on a budget. There are a lot of jobs that pay $50-80k for a sysadmin. The vast majority of day to day things can be done by one of these people. When they get stumped on legacy stuff or something really weird, they end up spending a lot of time spinning their wheels and have a hard time getting the problem solved.
The other option is to hire the $150k sysadmin who has tons of experience and makes the hard problems look easy. These are the kinds of people who you can give 3 months to solve a problem, or you can hire a team of 5 people to work for 20 years on the same problem. If you put it in that perspective, the money is well spent.
Smart business people look at numbers and know that $150k is more than $50k, and also know that if they yell loud enough about the $100k they saved, some of it will end up in their bonus.
The thing that seems obvious to me is that you hire a bunch of the cheaper people who can do all of the normal day to day stuff, and you also hire a guru who gets all of the impossible tasks. The less experienced guys learn from the guru and the guru doesn't spend 99% of his time doing tasks that would be better suited to a college student or a shell script.
Of course, companies don't like this idea because HR people don't want to believe that one person can be worth several times as much as another person who is referred to with the same type of job title. In HR there are no gurus, so the concept is completely foreign. After all, if someone was inclined to be a guru in any field, how would they end up in HR? =)
We are not "bad" at all. We are doing our jobs. What are our jobs? Our role is to keep our systems operational and secure.
Think of it in terms of roles.
Sys Admin: Protects and defends the infrastructure of your company. Prevents people from shooting themselves in the foot. Enforces good security policies. Identifies poorly performing software and forces its developers to improve it (or get shut down). Keeps your systems patched and ready. An iron fist in a velvet glove. An enigma wrapped in a mystery. A big, sexy man!
Programmer Type 1: Cooperative with sysadmin. Tries to write solid code. Doesn't break stuff. Often has a good rapport with sysadmin and finds, mysteriously, that his jobs get run on time, every time. Filled with the Tao.
Programmer Type 2: Bastard child of Peter Lorre and Marty Feldman (with the voice and the eyes). Doesn't care about correct practice, only what he can bang out in an hour. Takes ridiculous shortcuts, risks crashing servers and services. Source of all memory leaks. Tries to be clever and fails. Mortal enemy of all sysadmins and Type 1 programmers!
User: Whirling dervish of chaos in an otherwise orderly world. Between downloading P2P apps, questionable freeware, and trojan and adware corrupted hacks of popular programs, spends time inviting the wrath of the RIAA and MPAA by sharing his entire music collection from the main file server. Browses pr0n instead of working. Plays solitaire instead of working. Cries like a little girl every time he's forced to comply with official policy. Complains bitterly about those nasty sysadmins. Secretly wishes he was a pr0n star and has been stalking Shelly down in accounting. She'll mace him in the cafeteria later on in the week.
NO CARRIER
We're watching you and we also happen to know that a 3 phase 415V busbar runs dreadfully close to the RJ11 cable which terminates at your office desk telephone. I also recall this cable being reported as damaged (stripped bare by *accidental* friction). However the repair job is in our very long backlog queue at the moment, just after "Halon release trigger malfunctioning in Lockejaw's office".
Please call us if you have any further queries on what we do down here in the basement.
You said it. I was one of two Unix SAs supporting a few dozen servers for which several hundred users depended for their jobs. If something went wrong, they called and, just like magic, things were fixed. They loved us and they loved the application. The worst thing that could happen would be a server death and when that happened, we'd call up the manageer of the affected group, ask them to have their people save their work locally and sit tight. Out of the closet would come a pre-configured replacement server. We'd plug it in, restore data from one of our three redundant back-up systems, and have those users up and running again in two hours, max.
I loved the work. Absolutely loved it. Because this was a government job with generous paid leave when one of us would be gone, having two of us meant there was always coverage and no downtime. Given that our users brought in 10s of millions of dollars a month, we were a paltry and perfectly justifiable expense.
Our problem was that nothing ever went wrong. Our big 'ol rack of servers hummed along with no drama and whenever the boss dropped by, he'd likely see us plodding through something routine like adding a user or checking system capacity reports. Every few days, we'd get bored and actually walk around the cube farm of the users, stick in our heads, and ask if everything was ok, can we do anything to make things work better? Our users loved us; our bosses didn't even seem to know what to write on our evaluations.
The Windows servers on the other side of the datacenter? Holy Cow, did those guys have the drama! Things were crashing all the time (We're back in the early NT days, mind you.) Whole populations of users suffered critical amounts of downtime. The admins put everything back together, of course, and were lauded as heroes because they had fixed the big, bad problems that had killed so many people's productivity for so long. They were HIGHLY visible to management. They got awards for fixing things. They were heroes.
Us Unix admins were those two people who sat over in the corner and never seemed to actually, visibly do anything.
You can see where this is leading, right? The Windows server side and the Windows front-line support side needed warm bodies, so I got thrown off Unix and into a GUI world I neither wanted nor understood. (Don't get me wrong, I've done the Windows work for years and I love helping people, but I'm not in love with the OS I now use and support.) Later, the other SA was tossed and our servers virtualized on mainframes. The number of SAs was cut to the bone and beyond. Virtualization was a nice concept and it works fine, but getting something fixed when it breaks is now a major red tape experience for our poor (former) users.
Fires to put out mean that firemen get chances to become heroes. Safety engineers who inspect your business and show you how clean the grease traps so nothing actually catches on fire are just needless expenses to be cut as soon as possible.
The moral is: Be a fireman. I figure they get more women, anyway.
From TFA:
How do I get my sysadmin to do anything?
Very simple.
Cookies, brownies, pizza, etc. I've worked as a sysadmin, so I know all sysadmins like the ones with the little dark chocolate chunks in them.
Register the editry.
Very true.
I'm a sysadmin in a DoD Classified network on a USAF Base ( LOTS of red tape) and the first rule is security (which it should be). That pretty much means lock everything down. Some examples include: lock the USB ports, prevent writes to CD-RW Drives, prevent writes to DVD-RW Drives, audit everyhthing (PL2), prevent printer installs, software installs/removal, lock down screen savers (executable code), password changes every 62 days, approved software installs only which usually means we are lagged on releasees, etc, etc, etc. Some of these are silly, yes, but I don't make the rules. The "red tape" is mandated to us by the Air Force.
All this red tape creates a very unfriendly user environment where the users frequently are annoyed with the admins because they can't do something as simple as copy data from a classified PC to a classified laptop for a presentation. They have to track down an admin to do the copy for them. Paperwork must be filled out and whitnesses present. They may not have access to files due to security permissions. Won't delve into the requirements here but it has to do with employees from different companies all working the same program who potentially have access to each company's proprietary information. I can go on and on, but the bottom line is red tape creates a very unfriendly user environment where the users frequently claim the sysadmins "don't know what they are doing", which isn't the case at all. The users are deliberatly not allowed to do what they are trying to do. However a majority of the user community thinks us admins make the system painful to use on purpose. Not the case. and they frequently take out their frustrations on us.
The real moral is that if you want to be a valuable geek, you have to learn enough people skills to make sure other people know. I've got a couple of decades of professional experience under my belt and am an expert in several areas, but the most valuable experience I've had professionally comes from working in a large company with a good number of untrained monkeys.
There are a lot of people who can't tell the difference between a seasoned professional and someone who would have bought a computers for dummies book if they were literate. Some of these people will be promoted into management to keep them out of the way of people doing the work. Being able to interact with people on their level is an incredibly valuable skill. It's nice to work with intelligent people who know what you do, but not everyone gets that kind of dream job. Basic communication skills are important, even if you feel like a retard when you're doing what is expected. If you don't feel like a retard, you're probably not going to effectively communicate with the business people. =)
For example, I'll send out emails to users, managers and the VP to let them know that a disk on the EMC failed, switchover to one of the hot spares occurred without incident, the failed disk was replaced and transitioned back into the array without issues and with no more than negligible performance degradation to the systems and users. No data was lost and we're back up and running. This happens once or twice a year.
If you know anything about EMC arrays, storage systems in general, or how to get your VCR to stop blinking 12:00, you probably realize that I didn't really have to do anything other than be aware that something happened and let the field service technician do his job. I've spent my whole career learning about technology so I am perfectly capable of doing all of the maintenance myself, but in this kind of case, I just need to let someone else do their job. This is not exactly rocket science here. However, people who don't get the technology see something like this and think "Huh, I guess something broke and now it's fixed and everything's good. Good thing he knows what to do because I wouldn't even know who to call or what to say to them." Most of the people whose opinions matter have no idea what you do.
There are a lot of arsonist-firefighter types in IT. You can be just as valuable as them without losing any shred of decency as a human being. Just let people you help know to let your boss, your boss's boss, their boss, and anyone else they know how incredibly helpful you were. Chances are that they asked for your help because they needed you to do 10 minutes of work so they could avoid trying to spend weeks trying to figure it out themselves and making it much worse before it got to you. Most people will be willing to spend 60 seconds to send a quick email to help you out.
Once upon a time, in a land far-far away, I ran the IT department of a medium sized company. We weren't so big that I was the CIO, but I did repot directly to our corporate president (who reported to noone else, since he owned the company). A few hundred seats total, offices in 14 states, most of which were home offices, that sort of thing. I had a staff of eight, including four first-line phone technicians, and on the whole the entire operation ran quite smoothly.
The only times I had major problems, or heard of major complains, about a system administrator was when they started making one major flaw in their perception: that IT was there to do things in a way that was best for IT.
When a system administrator begins to believe that their entire function is self-serving, that they are there to support their own operations, that's when I've seen things go bad. Regardless if you are an IT consultancy firm, or an internal IT department, the sole purpose of IT is to play a supportive role in the organization. It's important to recognize that if IT didn't show up for a week, things would probably be okay (backup tapes would *really* need to be changed), but if the sales and customer service departments didn't show up for a week, we'd be damn near out of business.
I always tell people to think about their product. What is it you produce? System administrators, software engineers, they produce things that let *other* people get their work done *more* efficiently. If it's difficult for the sysadmin or developer to do, who cares? They aren't there to make life easier for themselves, or to devise some system that's perfect on the whiteboard but impractical in the real world - they are there to bring practical, cost-effective efficiencies to their end users.
Now, if you have a guy going around unplugging peoples network cables at the switch because they pissed him, fire that guy and hire a professional.
My $0.02.
And that seems to me to be the key thing: inexperienced people need to work along side experienced people. That's how people get experience.
Why is this such a hard concept for people to understand these days? It's like there are two camps: either you think companies should hire all-knowing experienced geniuses or you think companies are better off hiring a small army of inexperienced guys.
Throughout history, in pretty much every trade, there's been this idea of apprentices. There's been the idea of "working your way up the ladder". The idea is pretty simple: you put the new guy in with the experienced guy, so that the experienced guy can pass on his knowledge and the new guy can get up-to-speed. Over time, the new guy learns enough to take over the small/easy portions of the experienced guy's work. The experienced guy gets to avoid the crap-work, and the new guy gets experience. Over more time, the new guy starts becoming an experienced guy, can take on more complicated problems, the experienced guy can keep focusing on higher and higher-level problems, and it keeps building on that model until either the old experienced-guy or the new experienced-guy move on to something better.