Slashdot Mirror
Life in the Trenches: a Sysadmin Speaks
Anonymous Coward writes "A senior systems administrator at a big ISP in Australia offers
a no-nonsense view about his line of work, the pros and the cons, ths ups and the downs."
← Back to Stories (view on slashdot.org)
Pros: Cheese Doodles
Cons: Users
I didn't see the "problem solving skills" as a requirement for being a sys admin, mentioned anywhere in the article.
I think problem solving skill are a must for the sys admin job, especially if you don't want to be a Jr. Sys Admin and perform backups all your life.
I worked for a relatively large institution, in the capacity of a Sys Admin, and I know for a fact that you need some serious problem solving skills.
Consensus is good, but informed dictatorship is better
Good read - I think its important to recieve an impression on what your future jobs might turn into once you have been on the line for a couple of years.
Of course, its important to try your dreamsjobs during during university, but you never know if your dream wont turn into a nightmare after a few years but just working a few weeks there...
+++ MELON MELON MELON +++ Out of Cheese Error +++ redo from start +++
oh i forgot to mention, in my last post:....
Being a dedicated sys admin is like being a hobo, and thus the saying: "I maybe a hobo (sys admin), but you are a nobo (users)..."
Consensus is good, but informed dictatorship is better
This guy basically has the "bastard operator from hell" mentality, he's just a little more polite about it.
Any sysadmin that has to log into a system while on holiday in *India* is a bad one. If you don't have enough redundancy built into your system that your junior admins/engineers can't hold down the fort for a week or two, something is wrong.
Second, "strong experienced based opinions" is crap. Open your eyes to new concepts and ideas. Like me trying to explain to two 10+ year network engineers that having a flat, layer 2 network across an entire Air Force base with 8000 users is a Bad Idea, and that adding layer 3 switching capability at the distribution points wouldn't slow down the network, and it would, in fact, be faster. Sure, hold on to your opinions, but understand things change, and if you don't change with them, you're a gorram dinosaur.
(This are a Melbournian's answer to your questions) No-one in Australia actually drinks Fosters, it's impossible to get in Pubs (where Tooheys or Carlton/Victoria Bitter generally own most of the pubs), and tastes crap. We're well aware that it's one of our most successful exports, however (even if it does taste like crap). V, Red Bull or Coffee suffice as drinks to keep you up at night. Melbourne city has at least four cafes and three 7-11s per square meter, so it's anything you want, really.
What qualities do you rate as essential for a good sysadmin?
In rough order of importance:
Aptitude.
...
Is it just me, or is that a somewhat circular choice for first on the list? What IS aptitude, but the qualities essential for the purpose?
Having been in the IT field for 10 years, of which I've been a UNIX sysadmin for about 5, I must say this is one of the better articles/interviews I've read on the subject (not that I've seen that many). Not to over emphasize the importance of the job, or to inflate my own ego, but in all honesty I believe the job of the system administrator in IT to be one of the most important, if not the most important. System administrators must design, implement, and maintain computer systems. This is obviously one gigantic chunk of what makes up the information technology field as a whole.
It has often been my experience that the sysadmin(s) for an organization is/are the best informed resources from an IT perspective (at least if you're a good one). Who else do you talk to when needing to discuss any significant change to an organization's computing infrastructure?
To the person who commented that there was no mention of good troubleshooting skills as qualification for a good sysadmin....I believe that fell under the comment that a component of the sysadmin's job was to keep the systems running. To be able to troubleshoot and solve problems is a prerequisite to keeping systems running.
Second, "strong experienced based opinions" is crap.
It's better than just 'strong opinions'. Anyone logical enough to realize that you should normally have opinions based on experiences is normally logical enough to be reasoned with regarding how those experiences may differ from other experiences, and how 'new' approaches may in fact be better.
In your Air Force situation, it sounds like the people you were dealing with had had little or no experience with the type of topology you were recommending.
creation science book
Systems Administration is the kind of job that nobody notices if you're doing it well. People only take notice of their systems when they're not working, And they tend to forget that a lot of work and expertise goes into making sure that they continue working.
You only ever talk about IT when things go wrong. In my mind, that's a thankless job. I am SO thankful that there are people that don't mind that... And this guy is a professional through and through:
But that's as it should be - computer networks are infrastructure that you should be able to rely on, to take for granted, just like telephones and electricity. If you can't do that, then there's something wrong, something that can and should be fixed.
I like how he takes responsibility. This is unbelievable. I want him as my IT guy now.
a sysadmin has to be _ethical_. They're in a position to witness alot of people's private information, especially in a place like an ISP - not even Echelon can monitor people online like the sysadmin can.
I also fully agree that when you're on vacation, if your underlings can't keep the ship together, you're not doing a very good job.
What he doesn't hit on very well in his preachy missive is the importance of diplomacy. I work in a big enough operation that I don't even deal with the end-customers, we have an application support team for that. This means that (a) the problems are reduced, since I only have to worry about a handful of real "users" who can damage the systems and (b) the problems are greater, because those guys are vastly better at really kicking the legs out from under my boxes! So it's mightily important to always touch base with the application support teams, and keep a continuous stream of communication up. It's easy to lose that, especially in a giant operation, especially when your specialty is copping an attitude.
And finally: Why do so many sysadmins dedicate their lives to looking like freaks? Find a shower, a razor, a comb, and use them, people!
He's dead-on with his observation that personality type and aptitude are the most important qualities in a sysadmin. I am fighting a battle with a boss who actually thinks you can train someone (anyone) to be a sysadmin. Unfortunately when these people fail miserably I get accused of poor training. Oh well, I can always work for a service provider in my next life.
I suppose it could seem like an "over dramatization" if you haven't been in the job or haven't been in the job long.
While reading the article, I found I was agreeing with almost everything written except, maybe, the MBTI bit.
The part I liked most was one of the last comments about knowing you've done a good job when nobody knows you you did anything at all.
I spent a weekend replacing the HDs in two Banyan servers (upgrading five 1.2 gig drives in each, with 9.1s in a RAID 5 array) then restoring and testing all services and data.
I walked in Monday morning and asked the users if everything was OK.
They said "everything's fine, why?"
"No reason." I said and walked away with a smile.
Like the admin in the interview, I also had a piece of furniture give way from underneath a server but I was in the room at the time and was able to stop its rapid decent to oblivion and eased it to the floor. It stayed on the floor until we got a proper rack unit.
So, there is "drama" but, I wouldn't call what was written an "over dramatization".
What's your biggest complaint about the profession?
I don't have much to complain about
HUH!? I'm gonna go out on a limb here using my expereince and the people I know and say this is the exception and not the norm... Is this guy for real? Every sysadmin professional I know complains about the users, the hours, the pay and their job security. And what's this Telecommuniting BS? 70% of the time he was able to stay at home? Am I missing something here? This does NOT sound like the average Sysadmin Job I've come to know. Most employers are too damn anal for that to occure, even if you could effectively...
Jeez... I must be missing something here... Talk about a raw deal...
You need a FREE iPod Nano
> hell" mentality, he's just a little more polite about it.
Too true and still an unfortunate stereotype of all too many self-annointed sysadmins, or at least those who can get away with this attitude. Unfortunately, many inexperienced management types still think that this is acceptable behavior - but that is changing.
He sounds like he works at a relatively small and fairly autonomous site without too much interaction with other groups/departments using the systems on a day-to-day basis. His management also doesn't appear to know what is going on - but it probably doesn't matter and they don't care given the circumstances of this particular site.
Any one involved in system admininstration or interested in this type of job should consider the recent book "The Practice of System and Network Administration" (by Thomas A. Limoncelli and Christine Hogan) a must read. This is a far more realistic description of contemporary practices in system administration than the comments made in this article.
troubleshooting often has nothing whatsover to do with the system at all.
The primary difference between a really good admin and a BOFH is the realization that "lusers" are *part of the system.* A really, *really* good admin has to be that apparently rarest of geeks, the person with outrageously good technical *and* people skills.
After all, the admin isn't just responsible for the machines, he is also the primary interface between the machines and the people.
How do you know if your company has a really talented admin? If he kills all of a user's processes and deletes all of his files, and the user is so greatful the treats the admin to lunch.
Now *that* is evidence of an admin who has figured out what his job is and how to do it. Which is, unfortunately, rare.
KFG
Yes, you are missing out. A good sysadmin at a decent company can have a very good life. I have had sysadmin positions with small, medium and very large companies where I telecommuted 90% of the time. In one job I telecommuted 100% of the time for a year, before I felt a bit lonely and started frequenting the coorporate campus for a few hours a week. It's amazing what a difference there is when people can put a face with the voice at the other end of the phone.
I was a good sysadmin and I have greater aspirations than this guy does so, I have moved up and beyond these older jobs but, they were very good jobs while I was there.
You're missing out. The question you must ask is, why? Are you really as good at your job as you think you are? Are you able to relate to management or are you constantly trying to win pissing contests with them? Do the users like you, or do the fear or view you with disdane? Honest answers to these questions are harder to get than you might think. You may want to ask a peer or higher-up engineer type for brutally honest answers to these questions. Engineer types will usually oblige, provided they aren't close friends or subordinates. Once you have these answers, accepting them and working to truely address potential shortcomings could completely turn things around for you. Good luck.
Because most people can do some of these things, they can end up doing sysadmin work. Does that make someone a sysadmin? I have interviewed for sysadmin roles before and always been amazed at the people who have used an application, or watched and install, and then applied for the sysadmin job. It's not enough.
The problem is, lots of people doing this kind of work without the training and experience (and often, no mentor either - nontechnical boss) give the profession a bad name - hence the whole BOFH subculture.
This link describes some of the issues related to this job that isn't very mature at all ...
Why did you buy the computer? To run programs. And so step forward the programmer...
Why did the programmer write the program? Because it performed the task needed. And so step forward the analyst...
Who needed the task performed? And so step forward the end-user...
I've always thought Syadmins to have an over-inflated importance in the world. As I show above, I put them third or fourth in the pecking order (depending on whether the end-user and the analyst are not the same people). Many admins forget that the point isn't to have lots of wonderfully run locked-down computers that don't do anything (damned users! get in the way of my policies...). A computer is a tool - a beautifully polished tool that doesn't do anything is worthless.
Cheers,
Ian
I will agree with someone else who posted that this guys comments about personality types are right on. You do not *have* to have a particular personality type to be a good sysadmin, but you need to at least have the self-awareness to know what your personality is and how it affects your job performance.
Of course people on slashdot are always looking for something to disagree with, so a few of you have already lashed out at the "strong experience-based opinions" quote. Experience is the number one most important part of being good at *any* job. If you don't agree, then you probably don't have enough experience.
I'll also say this: You don't have to agree with everything someone says to learn from them. (In fact, if you only listen to people who you are in complete agreement with, you will never learn much of anything.) There are a lot of good points in this article, and even if you are somehow offended by the experience-based opinions remark or something else, you can still gain something from it.
Here's a positive comment.
I thought that was an insightful article. System administration is the process of keeping together an organization's information infrastructure. People often find this job to be non-human oriented, but it is in fact completely human oriented. The good sys admin is constantly thinking of, and even torturing themselves over how the users will be affected by anything he/she ever does and how it can make their lives easier.
The really good sys admins will unfortunately be perceived as adversaries because they would rather disagree and cause a political stir than develop a system that they believe is going to harm the users more in a long run.
Most intelligent people can figure this out, and will respect their sys admin's position in the company. The sys admins who stay quiet during meetings when they see the company making a wrong move are the ones who don't care, and IMO better fit the profile of BOfH.
At the heart of the matter, our profession is to increase the quality of life through information technology. Anyone who doesn't see their IT profession this way is in the wrong career.
As long as all the SysAdmins seem to be making it up as they go along, we will continue to be marginalized and geek-ified by management. Try on for size:
- ITIL (IT Infrastructure Library)
- More ITIL
- IT Service Management Forum (the U.K. headquarters)
- CMM (Capability Maturity Model) Technical Engineering Practices
- The Open Group's Technical Architecture Framework.
Heck, even Microsoft is trying to get into the picture with its Microsoft Operational Framework, a kind of embrace-and-extend on ITIL, though I don't know of many places that are actually using it.It's not that the SysAdmin necessarily has to manage these processes - though in a small shop no one else will - but he/she/it needs at least to be able to talk the language and understand the processes that the IT Manager has set up. And if you are managing the shop, then this is your job. You must know this stuff as a matter of professional responsibility and "keeping up" in your field.
A 20 min. presentation to the other managers on Best Practices and Processes in IT Management will gain you a lot of credibility and help lift you out of the geek gutter. There are decades worth of lessons that have been learned the hard way and documented into these processes. When you can demonstrate to management that you are drawing on a substantial body of knowledge that is geared towards improving service and reducing total cost of ownership, you will gain their respect (assuming that you care about their respect).
Beyond this, I want to emphasize an excellent point that Sanders makes in the article. The SysAdmin job is one that is invisible if you're doing it right. A good day at work is a boring day. Excitement is a sign that something has gone wrong. You should structure your environment to be as boring and reliable as possible.
Too many SysAdmins live off the adrenaline rush of fixing a broken server while everyone else in the organization sits on their thumbs waiting. That's costly for the organization, but ironically is the easy way out for the SysAdmin - you don't need to be disciplined or structure your time or do any planning or thinking, just jump from crisis to crisis. It's much more challenging to turn it into a boring desk job where most of your work is pushing paper and the machines pretty much take care of themselves. But guess which option is better for the organization's mission?
Once you do get to that Nirvana state of boring life, you can strategize how to produce some measurables so you can blow your department's horn at the monthly managers meeting. Because if you do your job well, with the result that your work is invisible, they'll cut your funding unless you keep in their face on a regular basis.
.nosig
I think that having a good understanding of how something works is far more valuable than having a specific rote procedure to follow. If you understand it, you can deal with situations that haven't been pre-scripted i.e. you can deal with unplanned emergencies. If all you know is a set of rote procedures then you're in serious trouble when something crops up for which you don't have a set procedure.
As another poster mentioned here, his number one quality for the job is aptitude. If that's not problem solving, I'm not sure what is. So it seems that you and the article agree, except that the author expects his juniors to get it and would not keep them around long if they did not.
Friends don't help friends install M$ junk.
Ye gods, how true!
--K.
Sig: Bad people happen. Try to avoid being one of them.
The reason why it's not a thankless job for Craig Sanders is because he is in a worthwhile position within his company, able to control and hence take pride in the running of complete systems, not employed as a mere grease monkey without input yet always blamed when the systems are down.
I think many sysadmins on this forum will find that the following rings a bell. You begin with total control in a startup IT team, decide on and bring into operation all aspects of a solution and keep it all running perfectly for years, with near-zero downtime and great job satisfaction. Then the corporate machine takes over, basically overturns everything you've done and creates an absolute disaster, and despite ignoring utterly all your input, you are to blame since you're the sysadmin. Needless to say, job satisfaction is, let's just say, less. This ring a bell?
Craig Sanders has managed to avoid stage 2 so far. He deserves only praise, in my book.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
I have been administering systems for over a decade now. I do many of the technical interviews for the company I work for... or at least I did when we were hiring :( . Dismiss me if you want, I don't particularly care, but be aware it may be me or somebody much like me, on the other end of the phone the next time you try and get a job.
For everyone whining about the fact that he says a good sysadmin should have strong opinions based on experience... If you think that every problem is going to be so clear cut and so clean that you can just bang out an optimal solution and provide a clean and mathmatical defense for it, all you have done are home or academic excercises.
The problem domain for solutions is so incredibly broad, and so incredibly rich, that if you are not depending on collection of good solid abstract rules of thumb and effective practices, you will never get to a good solution. You have to use intuition to narrow down the problem domain to a few concrete approaches, and then apply logic and experience to decide which of them to implement and how.
These are not opinions like "NT Sucks, Linux rules", these are opinions like "I don't want to hinge my business case on an operating system controlled by a single vendor". I don't want an enterprise IT infrastructure that depends on technology that only runs on non-scalable hardware". "I don't want an operating system that I cannot remotely administer". "I want an operating system that allows me to update and maintian, stop, and start some subsystems without effecting other subsystems". "I want an operating system where I can apply security patches without being forced to install operating system updates". You get the idea.
Having an open mind is important, but at some point you have to get off your ass and decide something, and act upon that decision. The older I get, the more important I have realized this becomes.
A group of people with "strong opinions based on experience" can get together and hammer out a list of pro's and cons, and come up with an excellent solution to a problem, fully aware of what the solution does well and where it will be weak. It will be a stressfull meeting, and tempers may occasionally flare, but when you finally grind through it you will end up on solid ground, and everyone will likely be on board.
A bunch of people with "open minds and no strong opinions" are going to dither about endlessly and end up with an unfocused, innefective, designed by committe monstrosity.
Acedemia is all about exploration and investigation. Work is about getting things done. Note though that even the academia people typically won't get much "exploration" done if their home made router is down because it is an old Linux box built around a $20 commodity power supply that just went up in smoke, and the only guy that knew how to set up the IPTables to get the routing right left to go to grad school 3 months ago.
I am with this guy... a lack of a strong opinion and the ability to defend it, suggests to me a lack of experience. How on earth can you do something day in and day out, sweat over it, bleed over it, live and die by it, day by day and year by year, and not form an opinion?
Mathematically impossible requirements are technically not against policy.
hired out are a special case. Certainly an ISP is the most obvious example, and one where the indirection is so great most users don't even realize they're users.
I'd only point out that help desk people are themselves users of the system, and generally rank only a smidgeon above subscribers on the "luse-O-meter."
My point stands.
KFG
I kinda feel sorry for any company which lets unknowlegdable people make decisions for the experts to implement.
... Governments are instituted among Men, deriving their just Powers from the Consent of the Governed...
I've been a sysadmin for the past 5 years, two of them at a large department in a very big educational institution. I have to say that of all jobs I've had in the past, this is the most personally unrewarding.
Sure, the pay is good, and the benefits are nice, and you get to sit in your comfy chair most of the time punching buttons and not really doing anything in particular. However, this "bliss" comes with the following drawbacks:
I was an education major in college, and during one of the classes our professor told us: "when you start teaching, there will be rich schools and poor schools. If you work in one of the rich schools you will have a good salary, good budget, nice classrooms, and decent lunches. If you work in a poor school, you will have none of that, plus drugs, violence, and complete lack of parental involvement. Believe it or not, some people prefer to work in poor schools simply because if they are doing their job well, there will be people who will stop them every day in the hallway and tell them how much they admire their work. Not only that, but people working in poor schools are able to see with their own eyes how much difference they are making in the lives of the children they teach."
That seemed weird to me then, but now I think I understand. It all comes down to what one thinks to be a good reward for their work. If it is good pay, quick career path, and a Porshe by the time you're 30, then being a sysadmin is your dream job (granted, of course, that you're good at it). However, if you are looking for something that is personally rewarding, something you want to feel good about doing... You might want to pick a different carreer. Or at least do it only until you start feeling burnt-out.
Me? Oh, I'm quitting as soon as I can afford it. :)
If you open yourself to the foo, You and foo become one.
Problem solving ability would probably be included in "aptitude."
The end-user plays little to no role in information technology.
The individual end user may have little voice in matters, but if the end users are not doing what they are supposed to be doing, the system is pretty much worthless.
I don't know if the slashdot poll code can handle 16 options, but I suspect that the "fairly uncommon (less than five percent of the population)" people are probably more like 90 around here. Perhaps it is time to find out.
I know some imformal polls have been done, and indeed the normally rare INTJ and INTP types show up far more often among computer types than in the general population.
I did see one about 5 years ago;
Wanted: Web master. 10 years experience.
I drank what? -- Socrates
I guess this job was inevitable for me since I discovered computers at the age of 11.
He says that as if that's a particularly early age to screw around with computers!
</sarcasm>
mogorific carpentry experiments
I've heard Coopers is a much better brew than Fosters
-Stu
To make a long story short (and the flame war got ugly), Craig feels that a DNS server needs to support the legacy BIND zone file format. Dan, obviously, does not; he feels that it only matters that one can transfer the zone file over to the new format (losing all comments in the zone file in the process).
Now, I will side with Dan here. Keep in mind that my viewpoint is rather biased, being the person responsible for the MaraDNS server, a server which Craig uses but feels is "poorly written code". Now, the only specific that Craig went in to when pointing out that he did not like my DNS server is that fact that, like Dan's TinyDNS, MaraDNS has no support for BIND's zone file format.
Now, with all due respect for Dan, I think he should not knock a gift horse in the mouth. The fact of the matter is that the code for MaraDNS is open; if support for BIND-style zone files is important to Craig, I suggest that he start coding it himself. I will gladly accept code which can read BIND-style zone files and make it part of MaraDNS.
I am not saying that BIND style zone file support is unimportant. However, I think Craig should be a little more courtious in requesting this feature than badmouthing MaraDNS on the Debian ISP mailing list.
I am sure he is an excellent system administrator; I really wish that he would start up a serious open-source project so that he understands how we OSS coders feel. I think it would make him interact with us in a more mature fashion; and save both him and the developers he flames some grief.
- Sam
P.S. I know Craig already knows this, but there is a non-BIND DNS server which supports BIND style zone files called NSI. It is on the list of DNS servers on my web page.
The secret to enjoying Slashdot is to realize that it should not be taken too seriously.
When casual dress first started in the business world, every programmer jumped on it. Beards, long hair, unkeptness. I started wearing suits. When asked, I told them "I refuse to conform to the nonconformity".
If I ever got a job as a systems geek, I would go back to suits just to be different. I like different.
If you aren't part of the solution, there is good money to be made prolonging the problem
I agree with you, and I think it may be at least in part explained by something that occurred to me a while ago. Remember the old quip "Good, fast, cheap. Pick two."? Well, management can tell "cheap", at least in the sense of "Plan A will cost $2M up front, while Plan B will cost $1.5M...let's go with B." They can tell "fast": "Plan A will be done before the end of the fiscal year, but Plan B won't." But "good"? Your management, upper management especially, probably can't tell what a good IT infrastructure is. So the balance is always going to be tilted towards "fast and cheap", simply because that's what the people calling the shots understand. (Now, ideally, your management would trust you and ask for your opinion...but now we're living in a world of make-believe, with flowers and bells and leprechauns and magic frogs with funny little hats...)
Maybe I am just lucky to be at an oeganization which truly values what the IT department has to offer. Most often we are given the problem to find a solution for and present several solutions to TPTB. Ultimately it is TPTB who make the final decisions, but the decision is based upon choices which we already know that will fit into our infrastructure. I have seen what happens when a PHB makes a decision simply for the sake of making a decision and it is often never a pretty sight.
... Governments are instituted among Men, deriving their just Powers from the Consent of the Governed...
I kinda feel sorry for any company which lets unknowlegdable people make decisions for the experts to implement
... or the local McDonald's awaits!
So, you're an *expert* now? Been reading up on cryptography, TCP/IP, programming, and so on, have we? Because a few weeks ago, you were, as you put it "unknowlegdable" to an painful degree. It is to laugh.
Luckily your utter stupidity is now well-documented in my journal for all to see.
"WSH is the most powerful language ever" - William "woogieoogieboogie" Platt
Time to get that pesky degree, I think
Hey, look it is the self proclaimed genius. You are not even intelligent enough to worthy of my time discussing, debating or trolling. At first I thought you had some glimmer of intelligence worthy of a good troll, but that quickly turned out to be nothing more than reflection of light off of the debris of insanity locked inside your head.
... Governments are instituted among Men, deriving their just Powers from the Consent of the Governed...
Hey, look it is the self proclaimed genius
You were the one who said "I can learn *anything* in minutes". And "I should be CEO of a Fortune 500 company". God, it's hilarious.
You are not even intelligent enough to worthy of my time discussing, debating or trolling
No, you lost. I have smashed every attempt you came up with to "hide the web page source". You keep inventing ever more bizarre new ones, but they are all worthless. Every time one is destroyed, you move onto a new one, and never try to defend the old ones again. I'm pleased that you realize the limits of your intellectual capacity, finally.
Anyway, get back your web site in Nowheresville Florida, your Windows pointing-and-clicking, and Javascript "programming".
Read your journal
... Governments are instituted among Men, deriving their just Powers from the Consent of the Governed...