Slashdot Mirror
Life in the Trenches: a Sysadmin Speaks
Anonymous Coward writes "A senior systems administrator at a big ISP in Australia offers
a no-nonsense view about his line of work, the pros and the cons, ths ups and the downs."
← Back to Stories (view on slashdot.org)
This guy basically has the "bastard operator from hell" mentality, he's just a little more polite about it.
Any sysadmin that has to log into a system while on holiday in *India* is a bad one. If you don't have enough redundancy built into your system that your junior admins/engineers can't hold down the fort for a week or two, something is wrong.
Second, "strong experienced based opinions" is crap. Open your eyes to new concepts and ideas. Like me trying to explain to two 10+ year network engineers that having a flat, layer 2 network across an entire Air Force base with 8000 users is a Bad Idea, and that adding layer 3 switching capability at the distribution points wouldn't slow down the network, and it would, in fact, be faster. Sure, hold on to your opinions, but understand things change, and if you don't change with them, you're a gorram dinosaur.
Systems Administration is the kind of job that nobody notices if you're doing it well. People only take notice of their systems when they're not working, And they tend to forget that a lot of work and expertise goes into making sure that they continue working.
You only ever talk about IT when things go wrong. In my mind, that's a thankless job. I am SO thankful that there are people that don't mind that... And this guy is a professional through and through:
But that's as it should be - computer networks are infrastructure that you should be able to rely on, to take for granted, just like telephones and electricity. If you can't do that, then there's something wrong, something that can and should be fixed.
I like how he takes responsibility. This is unbelievable. I want him as my IT guy now.
a sysadmin has to be _ethical_. They're in a position to witness alot of people's private information, especially in a place like an ISP - not even Echelon can monitor people online like the sysadmin can.
troubleshooting often has nothing whatsover to do with the system at all.
The primary difference between a really good admin and a BOFH is the realization that "lusers" are *part of the system.* A really, *really* good admin has to be that apparently rarest of geeks, the person with outrageously good technical *and* people skills.
After all, the admin isn't just responsible for the machines, he is also the primary interface between the machines and the people.
How do you know if your company has a really talented admin? If he kills all of a user's processes and deletes all of his files, and the user is so greatful the treats the admin to lunch.
Now *that* is evidence of an admin who has figured out what his job is and how to do it. Which is, unfortunately, rare.
KFG
I don't see "hands" as a requirement for being a sys admin, mentioned anywhere in the article.
I think hands are a must for the sys admin job, especially if you don't want to be a Jr. Sys Admin and perform backups (with your teeth!) all your life.
I worked for a relatively large institution, in the capacity of a Sys Admin, and I know for a fact that you need some serious hands.
I agree. I think problem solving skills, and the ability to learn and adapt are really the two things that make a good technology admin (system or network). If you have those skills, which are almost impossable to measure in test form, you really have all you need. All the technology knowledge and such can be gained later.
I know that I personally would much rather work with someone who was an ace problem solver and a quick learner, but who had little technology knowledge, than someone who had memorised every certification book, but was unable to apply that knowledge to real-world problems.
I will agree with someone else who posted that this guys comments about personality types are right on. You do not *have* to have a particular personality type to be a good sysadmin, but you need to at least have the self-awareness to know what your personality is and how it affects your job performance.
Of course people on slashdot are always looking for something to disagree with, so a few of you have already lashed out at the "strong experience-based opinions" quote. Experience is the number one most important part of being good at *any* job. If you don't agree, then you probably don't have enough experience.
I'll also say this: You don't have to agree with everything someone says to learn from them. (In fact, if you only listen to people who you are in complete agreement with, you will never learn much of anything.) There are a lot of good points in this article, and even if you are somehow offended by the experience-based opinions remark or something else, you can still gain something from it.
As long as all the SysAdmins seem to be making it up as they go along, we will continue to be marginalized and geek-ified by management. Try on for size:
- ITIL (IT Infrastructure Library)
- More ITIL
- IT Service Management Forum (the U.K. headquarters)
- CMM (Capability Maturity Model) Technical Engineering Practices
- The Open Group's Technical Architecture Framework.
Heck, even Microsoft is trying to get into the picture with its Microsoft Operational Framework, a kind of embrace-and-extend on ITIL, though I don't know of many places that are actually using it.It's not that the SysAdmin necessarily has to manage these processes - though in a small shop no one else will - but he/she/it needs at least to be able to talk the language and understand the processes that the IT Manager has set up. And if you are managing the shop, then this is your job. You must know this stuff as a matter of professional responsibility and "keeping up" in your field.
A 20 min. presentation to the other managers on Best Practices and Processes in IT Management will gain you a lot of credibility and help lift you out of the geek gutter. There are decades worth of lessons that have been learned the hard way and documented into these processes. When you can demonstrate to management that you are drawing on a substantial body of knowledge that is geared towards improving service and reducing total cost of ownership, you will gain their respect (assuming that you care about their respect).
Beyond this, I want to emphasize an excellent point that Sanders makes in the article. The SysAdmin job is one that is invisible if you're doing it right. A good day at work is a boring day. Excitement is a sign that something has gone wrong. You should structure your environment to be as boring and reliable as possible.
Too many SysAdmins live off the adrenaline rush of fixing a broken server while everyone else in the organization sits on their thumbs waiting. That's costly for the organization, but ironically is the easy way out for the SysAdmin - you don't need to be disciplined or structure your time or do any planning or thinking, just jump from crisis to crisis. It's much more challenging to turn it into a boring desk job where most of your work is pushing paper and the machines pretty much take care of themselves. But guess which option is better for the organization's mission?
Once you do get to that Nirvana state of boring life, you can strategize how to produce some measurables so you can blow your department's horn at the monthly managers meeting. Because if you do your job well, with the result that your work is invisible, they'll cut your funding unless you keep in their face on a regular basis.
.nosig
>>I've always thought Syadmins to have an over-inflated importance in the world. [...] Many admins forget that the point isn't to have lots of wonderfully run locked-down computers that don't do anything (damned users! get in the way of my policies...). A computer is a tool - a beautifully polished tool that doesn't do anything is worthless.
Granted the job of a Sysadmin is to keep the machines running so that the user can do their job, but to say they are "unimportant" is absolutly stupid.
The job is more like a janitor, you "own" the house, you make sure that everything is clean, that the kids are not running in then hallways and that the bathrooms are clean.
Having said that, that also means that I am going to restrict of what a user can and cannot do, in order to make the system work for EVERYBODY.
The problem is mostly not the endusers, they are EASY to deal with, the problem in my own experience are all those wonderful programmers who think because they can write some code they should have all the rights, all the power and oh yeah, root because "Well, the program can only do what it is supposed to do when it is run as root." Right, permissions are for wimps.
I never had a real problem with an enduser that couldn't be solved after some facetime, on the other hand I had Programmers who activly tried to root production boxes because they NEEDED to testrun a program that had failed on the dev AND test box (he later claimed they were broken, yeah right), never heard of permissions, it sometimes amazes me how little of an understanding programmers have about System Architecture and security.
Sorry, but face it, if you ARE on my System *I* am the one who tells you what you can do and can't do. I AM the cop on that system and if you don't behave I make sure you can't do much damage.
Sounds "God like"? No, I never kill processes without first knowing what they are doing or why unless they jepardize the system.
Oh, and for the guy who tried to root the box: He got a warning from the manager and I am sure he thinks about me the same way you think about Sysadmins.
M.
If you want to e-mail me, use my PGP Key.
Ye gods, how true!
--K.
Sig: Bad people happen. Try to avoid being one of them.
The reason why it's not a thankless job for Craig Sanders is because he is in a worthwhile position within his company, able to control and hence take pride in the running of complete systems, not employed as a mere grease monkey without input yet always blamed when the systems are down.
I think many sysadmins on this forum will find that the following rings a bell. You begin with total control in a startup IT team, decide on and bring into operation all aspects of a solution and keep it all running perfectly for years, with near-zero downtime and great job satisfaction. Then the corporate machine takes over, basically overturns everything you've done and creates an absolute disaster, and despite ignoring utterly all your input, you are to blame since you're the sysadmin. Needless to say, job satisfaction is, let's just say, less. This ring a bell?
Craig Sanders has managed to avoid stage 2 so far. He deserves only praise, in my book.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
I have been administering systems for over a decade now. I do many of the technical interviews for the company I work for... or at least I did when we were hiring :( . Dismiss me if you want, I don't particularly care, but be aware it may be me or somebody much like me, on the other end of the phone the next time you try and get a job.
For everyone whining about the fact that he says a good sysadmin should have strong opinions based on experience... If you think that every problem is going to be so clear cut and so clean that you can just bang out an optimal solution and provide a clean and mathmatical defense for it, all you have done are home or academic excercises.
The problem domain for solutions is so incredibly broad, and so incredibly rich, that if you are not depending on collection of good solid abstract rules of thumb and effective practices, you will never get to a good solution. You have to use intuition to narrow down the problem domain to a few concrete approaches, and then apply logic and experience to decide which of them to implement and how.
These are not opinions like "NT Sucks, Linux rules", these are opinions like "I don't want to hinge my business case on an operating system controlled by a single vendor". I don't want an enterprise IT infrastructure that depends on technology that only runs on non-scalable hardware". "I don't want an operating system that I cannot remotely administer". "I want an operating system that allows me to update and maintian, stop, and start some subsystems without effecting other subsystems". "I want an operating system where I can apply security patches without being forced to install operating system updates". You get the idea.
Having an open mind is important, but at some point you have to get off your ass and decide something, and act upon that decision. The older I get, the more important I have realized this becomes.
A group of people with "strong opinions based on experience" can get together and hammer out a list of pro's and cons, and come up with an excellent solution to a problem, fully aware of what the solution does well and where it will be weak. It will be a stressfull meeting, and tempers may occasionally flare, but when you finally grind through it you will end up on solid ground, and everyone will likely be on board.
A bunch of people with "open minds and no strong opinions" are going to dither about endlessly and end up with an unfocused, innefective, designed by committe monstrosity.
Acedemia is all about exploration and investigation. Work is about getting things done. Note though that even the academia people typically won't get much "exploration" done if their home made router is down because it is an old Linux box built around a $20 commodity power supply that just went up in smoke, and the only guy that knew how to set up the IPTables to get the routing right left to go to grad school 3 months ago.
I am with this guy... a lack of a strong opinion and the ability to defend it, suggests to me a lack of experience. How on earth can you do something day in and day out, sweat over it, bleed over it, live and die by it, day by day and year by year, and not form an opinion?
Mathematically impossible requirements are technically not against policy.
I've been a sysadmin for the past 5 years, two of them at a large department in a very big educational institution. I have to say that of all jobs I've had in the past, this is the most personally unrewarding.
Sure, the pay is good, and the benefits are nice, and you get to sit in your comfy chair most of the time punching buttons and not really doing anything in particular. However, this "bliss" comes with the following drawbacks:
I was an education major in college, and during one of the classes our professor told us: "when you start teaching, there will be rich schools and poor schools. If you work in one of the rich schools you will have a good salary, good budget, nice classrooms, and decent lunches. If you work in a poor school, you will have none of that, plus drugs, violence, and complete lack of parental involvement. Believe it or not, some people prefer to work in poor schools simply because if they are doing their job well, there will be people who will stop them every day in the hallway and tell them how much they admire their work. Not only that, but people working in poor schools are able to see with their own eyes how much difference they are making in the lives of the children they teach."
That seemed weird to me then, but now I think I understand. It all comes down to what one thinks to be a good reward for their work. If it is good pay, quick career path, and a Porshe by the time you're 30, then being a sysadmin is your dream job (granted, of course, that you're good at it). However, if you are looking for something that is personally rewarding, something you want to feel good about doing... You might want to pick a different carreer. Or at least do it only until you start feeling burnt-out.
Me? Oh, I'm quitting as soon as I can afford it. :)
If you open yourself to the foo, You and foo become one.