System Admins Should Know How To Code

snydeq writes "You don't need to be a programmer, but you'll solve harder problems faster if you can write your own code, writes Paul Venezia. 'The fact is, while we may know several programming languages to varying degrees, most IT ninjas aren't developers, per se. I've put in weeks and months of work on various large coding projects, but that's certainly not how I spend most of my time. Frankly, I don't think I could just write code day in and day out, but when I need to develop a tool to deal with a random problem, I dive right in. ... It's not a vocation, and it's not a clear focus of the job, but it's a substantial weapon when tackling many problems. I'm fairly certain that if all I did was write Perl, I'd go insane.'"

Comments

Very true, for many reasons.

[uCallHimDrJ0NES]

People who've never coded tend to have many "magic boxes" in their thinking about systems. I find it hard to fully trust an administrator who can't at least parse through other people's code.

Re:Very true, for many reasons.

[bolthole]

Depends what they're administering.
There are plenty of systems that are more closed, and the sysadmin should be spending their time living in the pre-provided frameworks, rather than coding their own.

Many times, it's a matter of learning what is already available for the system, rather than coding your own, lesser quality replacement.

Re:Very true, for many reasons.

[pinfall]

I don't code a lick after 15 years of sysadmin. I do solve incredibly interesting problems with hardware and that ability alone provides a serious interface to all those 'needs software' perspectives. All of my best friends are coders, we just solve problems differently.

Re:Very true, for many reasons.

[hairyfeet]

The problem is the way corps shit all over IT they'll just go "Hey, one more job we can dump on them without raising their pay!" and that will be that.

Lets face it folks, we are gonna end up with critical shortages of IT and infrastructure workers because between the offshoring, the H1-Bs, and the PHBs treating IT as this money pit that doesn't give them any profits? IT has been shat upon for the good part of the last decade.

I know myself and most of the old guard guys I knew ended up getting out of corporate IT for just this reason, piling more and more work upon us while expecting everything to be done with less help and a shrinking budget...now you want to add coding to the requirements? You gonna add a pay raise and pay for the classes? Yeah, thought not.

Re:Very true, for many reasons.

[countach74]

I think you missed the point. The coding helps save you time in the long run. Not the other way around.

Re:Very true, for many reasons.

[Mr2cents]

You don't or you can't? Because even when I'm not coding, just in day-to-day computer use, I often write scripts to batch-process or automate various things. I have found this highly beneficial. I don't want to break you down, I just want to know your reasoning. Personally, I can highly recommend learning scripting, be it bash/awk/php (yes, I wrote php scripts)/powershell/whatever, I'm sure you'll benefit from it.

Re:Very true, for many reasons.

[uCallHimDrJ0NES]

I agree with what you're saying, but you're missing my point about magic boxes. There are sometimes feasibility matters that are obvious to coder admins that are not obvious to non-coder admins. How often is this important? Rarely. When admins are also designers, it becomes more important. It depends on what you expect of your person with the title "admin".

Re:Very true, for many reasons.

[fsck1nhippies]

2Cents, you are absolutely right. Even in windows systems, a basic understanding of what can be done with code can stop 5 people from running around to a couple hundred machines each.

Should the sysadmin be a programmer? Not in the conventional sense, but they should be able to programmatically attack the problems placed before them before they just brute force their way through them.

Re:Very true, for many reasons.

[karnal]

I have coded in the past using Perl to give us some reporting ability from some non network-connected (think no email/notification ability) phone switches. They had relay outputs that would trigger on an alarm; so we used a box to generate a trap that would fire up my perl script to send out emails to whomever was in the config. I don't know a lot - I mostly use bits and pieces of code found around the web and customized to purpose. I initially thought I wanted to code for a living, but found in my first job that politics trump the technical, and boy were the politics thick. I do more security/administration nowadays, but still draw on the basic coding ability to create automated notifications for anything non-standardized. Really what I'm trying to say here is sometimes the basic experience of coding as well as a need to fulfill a solution that doesn't have a canned solution readily available can push you to do some interesting things.

Re:Very true, for many reasons.

[karnal]

And - sorry about the lack of line breaks there.

Re:Very true, for many reasons.

[scubamage]

I think you have another step in your development as a sysadmin my friend. You sound like you're still in a bit of the I have a hammer, so these problems must be nails. Sometimes a mix of some carefully crafted scripts as well as hardware is the most elegant solution (especially when cost becomes an issue).

Re:Very true, for many reasons.

[scubamage]

Exactly - it comes down to a matter of cost very often. If you can spend a half hour throwing together a script to make a configuration change across, say, 2000 devices (or in our case, several million), it is far cheaper than trying to find a vendor solution, or having folks go out and do the work themselves. The vendor will often have maintenance fees and high initial costs for a tool that "sort of" does what you want. You can have people go out and pound pavement, but if you have 2000 devices and send out 50 people, and it takes each person 20 minutes to do the work on the device plus 30 minutes trave time, times 17$ an hour... well, that adds up fast too. Not to mention the opportunity cost and backlog of tickets that that would generate.

Re:Very true, for many reasons.

[fsck1nhippies]

I see a lot of admins in very large companies throw labor at a problem as their first course of action. It is typically a face palm moment for me as I often see the problems as fixable in minutes. I believe that all sysadmins should be able to program, but think that making a programmer a sysadmin is generally a bad idea.

Re:Very true, for many reasons.

[swflint]

Pragmatically would be a better term; they ought also read "The Pragmatic Programmer: From Journeyman to Master."

Re:Very true, for many reasons.

[hairyfeet]

Yeah only unlike the janitors when IT goes wrong they are gonna be a lot of businesses that find out the hard way they are fucked, may even go under. I've seen first hand what happens when you let some guy lowball and outsource IT, the place ended up with a bunch of hand built "TigerDirect specials" with ZERO related drivers or parts so support was a nightmare, we opened up the networking closet and found the guy had sold the Sonicwalls we had put in and replaced EVERYTHING with those cheap shitty Dlink home routers, the network itself was a bunch of shitty home lines from a half a dozen different companies because the bozo didn't know shit about dealing with a T-Line, it was a fucking mess.

Of course the sad part is the retard that approved this moron had already made his bonuses for "saving money on the budget" and moved on to another company, leaving the next poor bastard to get stuck with the mess. but it really don't take more than one or two "WTF?" moments to bring a company to its knees and the costs to fix the problems this short term thinking causes may end up bankrupting the corp, again I've seen it happen, lots of places where no net means no work getting done and it don't take too many days of your entire workforce sitting on ass because they can't do their jobs thanks to IT issues to kill a corp.

Re:Very true, for many reasons.

[Anonymous Coward]

I write tons of scripts, but the problem tends to be management. Management will often see a task that takes say 10 minutes per computer, and does the mental math into man hours, and determines whether or not that task is necessary, if an admin spends 30-45 minutes writing a script to do everything in 10 mor minutes the manager will often see that 45 minutes as a waste and that he could have done 4 computers in the time to do "one". It's often an old school mentality, they did it that way when they were in the role so everyone should be able to do it that way. I've often have to write scripts on my own time, to have them available to me for when I need to do it.

To be honest, most of my managers have been very nervous about my suggestions to even use tools which I'm knowledgable in because they don't understand the concepts. Ad the bigger the company the more likely they are to want a vendor solution rather than use internal resources.

Re:Very true, for many reasons.

[roc97007]

The problem is the way corps shit all over IT they'll just go "Hey, one more job we can dump on them without raising their pay!" and that will be that.

Lets face it folks, we are gonna end up with critical shortages of IT and infrastructure workers because between the offshoring, the H1-Bs, and the PHBs treating IT as this money pit that doesn't give them any profits? IT has been shat upon for the good part of the last decade.

I know myself and most of the old guard guys I knew ended up getting out of corporate IT for just this reason, piling more and more work upon us while expecting everything to be done with less help and a shrinking budget...now you want to add coding to the requirements? You gonna add a pay raise and pay for the classes? Yeah, thought not.

You're right, with caveats, which I will get to in a moment.

I'm not entirely sure, but I strongly suspect that this situation you describe will turn around, at least a little. There's been some high profile fails in the news associated with outsourcing, either personnel or relying on cloud services, and as more and more outsourcing is done, I think these fails will get more spectacular and more common. You can already see a few companies swimming against the current. I read I think yesterday that GM is insourcing from HP, and trying to poach 3K of HPs admins.

But on coding, I'd really recommend that you rise to the challenge, even if you intend to stay in the dwindling (at least for now) pool of onshore admins. For a couple reasons: (a) Some moderate ability to code makes you more valuable, and valuable people are more likely to be retained or rebadged, not just laid off. (b) It possibly gives you some place to go (app development, app maintenance, system programming) should the admin field dry up completely. Or, I dunno, you could go into sales. A friend of mine, an admin with 20 years experience, never did learn to code, and moved into sales when his admin job was made redundant. On the other hand, not everyone is cut out for that, and it's a pretty cutthroat field.

Parenthetically, those of us who are still working, should keep in mind that if we find ourselves looking for a job, we will have as competition all those admins that were laid off in the last few years, and are perhaps hungrier than us. The safe bet, if you're going to stay in this line of work, is to hang onto your job tooth and nail, which could mean learning the programming thing.

Re:Very true, for many reasons.

[hawguy]

To be honest, most of my managers have been very nervous about my suggestions to even use tools which I'm knowledgable in because they don't understand the concepts. Ad the bigger the company the more likely they are to want a vendor solution rather than use internal resources.

As a manager, sometimes I see the tool to automate and script changes to be a risk of taking down the network faster than anyone can do anything about it.

Like the time a senior network admin accidentally took down our network by setting the IP address of all of our network switches to the same set of IP address. He tested the script and it worked perfectly on one network closet - then he proceeded to apply the remaining set of IP addresses the the remaining hundred network switches in our organization.

He knew as soon as the first alarm went off what had happened, but by then it was too late to stop it. Fortunately, he had the good sense to not write the config to memory, so recovery just walking to each network closet to power cycle the switches -- much faster than if we had to walk around with a console cable to back out the changes on each switch

That said, he still uses the same tool to push out changes, but he tests his scripts on 2 IDF's before pushing out changes and has the junior network admin double check his work.

But I can see why a manager would be cautious about a new and untested automation tool.

Re:Very true, for many reasons.

[guruevi]

Not to rain on your parade, but the SonicWall's are really just D-Link routers running a re-badged version of Linux (you can even drop to the command line and inspect iptables) while most of the shit Dell and HP churns out is not much better than the TigerDirect special.

I myself buy Netgear's ProSafe stuff for the network because it does the same stuff and has the same capabilities as Cisco etc. but for about 1/10th the price and most of my servers are SuperMicro because the first time I bought a COTS server, it was a rebadged SuperMicro server with a 50% premium. Off course, I don't use Windows so I have no problems with drivers and my desktops are Mac's so I barely have to touch the machines after they've been imaged, if it breaks, I troubleshoot and after 5 minutes on the phone Apple sends a tech out which is way better than the Dell Gold contract I inherited during the "we're not responsible for those blown caps" and the "call nVidia if your GPU blows up" debacles where I had to replace 10's of motherboards myself or have all my people without computer as I send them all in and wait 3 weeks for repairs.

Re:Very true, for many reasons.

[Rennt]

OP made a distinction between coding and scripting. It's assumed that EVERY sysadmin does scripting, but very few can code.

Re:Very true, for many reasons.

[Mr2cents]

Citation needed

Re:Very true, for many reasons.

[Crosshair84]

This is why so many small businesses,like the one I work for, can run rings around larger firms..The boss/owner started the company. He was the one running cable for many years, while he doesn't understand a lot of the newer stuff as well as me or my co-worker does, he understands it more than enough to understand what we are saying.and why we are saying it. He owns the business outright, he is in it for the long haul and wants things built properly.

Several years ago I developed a method to resolder the capacitors on a particular motherboard that we HAD to have, but had the caps with the bad electrolyte. (Nothing fancy, but the board has a few issues that made replacement difficult.) The Mark 1 variant was very ugly, but functional and proved the concept, the Mark 2 variant is what we still do today. When looking at the invoice the boss asked me why I was using capacitors that cost over 75 cents each, I simply pointed to the spec sheet and told him that the ones I bought were rated for 8,000 hours at 105C while the ones that were 10 cents each were rated for 1,000 hours at 80C. He immediately understood and saw how my decision was correct. No need to try to explain cost-benefit to some bean counter. No need to write up some fancy report, just point out the specs and he understands that by going with the expensive caps we will never have to re-solder those caps again and saving us gobs of money in the long term.

Hard drives in our recorders? WD Blacks all the way. Greens are so much cheaper, especially during the shortage, but he never even considered telling us to use cheaper drives and the increased cost was easily factored into our budgets. Contrast that with my brother, who works for a national company. When hard drives skyrocketed, getting them to authorize the funds they needed for hard drives was a monumental task and took several weeks of daily emails and phone calls.

There are many inefficiencies in small companies, however there are many inefficiencies as well that give them an advantage. Despite the government giving the large firms heaps of advantages,many small firms can still compete, though many simply leave the country, some better than others, as we have seen over the last couple decades.

Re:Very true, for many reasons.

[smellotron]

Here, have
some
of mine.

Re:Very true, for many reasons.

[smellotron]

Look at large buildings where management outsources janitorial care. Guess where IT is going.

Toilets in a large building are fungible. If one bathroom is out because of a mega-deuce, I can just go to the next one. Computer networks are frequently unique snowflakes tightly bound to the business.

Re:Very true, for many reasons.

[CAIMLAS]

... as opposed to most programmers, who seem to think that a system is an open sandbox in which they can piss and shit to their hearts content, expecting it to magically clean itself. :)

At least the simpleton sysadmins understand the cause and effect of various behaviors.

(That said, balance is really best. You're much more versatile that way and look like a miracle worker to everyone.)

Re:Very true, for many reasons.

[zaphod777]

While I agree with you, most things like that can be done with Group Policy and the things that can't can then be done with simple scripts managed by group policy. Macs and Linux machines are a slightly different ball game though.

Re:Very true, for many reasons.

[CAIMLAS]

There are certain things you can not learn - such as things which do not yet exist.

Case in point, from experience: complex mail migration tools. Take the use case of "Migrate mailboxes from x mail system to y mail system." Yes, there are dozens if not nearly a hundred different tools for doing so. Many vendors provide tools for their products which work with other mail systems to a limited degree, and there are a handful of 3rd party commercial and open source tools available for the job.

None of them work well except in the most simple of use cases: only a few users, simple mailbox utilization, not a lot of actual data, lengthy downtime is acceptable, shared mailboxes aren't important, etc.

Or sometimes it's not simply what needs to be done, but the timeframe in which it needs to be done. A tool might work perfectly, but at a 10th the performance you need to meet your deliverables. That's where sysadmin monkey coding comes into play - hack that tool up, throw it in a wrapper, and hope you put your bounds at the right place to not get into a race condition or fork bomb. It's not elegant but it's fast and gets the job done when nothing else seems to be able to.

Honestly, if you're not hitting situations where hacking together custom code, packaging your own tools, etc. aren't coming into play, you're not really trying that hard to find the best solutions. You're taking the 90% effectiveness solution and running with it, to the detriment and irritation of everyone who uses the systems you manage. Eventually, decisions like that will catch up with you: no, they may not require 'ongoing maintenance' like a custom internal process might, but they're going to be unmaintainable in the long run, anyway.

Re:Very true, for many reasons.

[CAIMLAS]

Experience has told me something about automated batching of config changes which your 'senior' admin should've known as well. This is crucial:

DEPLOY IN (SEQUENTIAL) PHASES. I don't care if you tested it in one situation or 5: if you've got 50 different situations, you need to 'test' it in each of those unless you know those different situations are in fact the same situation. This means a longer deployment, but you need to test ad verify everything, especially something infrastructural. It will take longer, but it's inerrantly necessary.

Not only does it reduce the outage potential, but it allows you to more quickly back out of it. Sure, it's not as lazy as deploying everywhere at once after a couple quick tests, but it's more foolproof than having half a dozen people look at your changes. Yes, having a second set of eyes is important, but its mostly pointless if they don't fully understand what they're looking - that's why they're the junior, after all.

Re:Very true, for many reasons.

[CAIMLAS]

Not only that: the truly gifted, good IT people go to managed services and then take the jobs away from the common IT person. It's good for managed services companies, because they make a lot of money by only employing the exceptional types, but in the long term it's bad because it sets the expectation that a few super bright people can do the job of a half dozen normal sysadmin types.

Guess what, those bright people burn out too.

Re:Very true, for many reasons.

[hairyfeet]

Dude don't be fucking silly. yes I know the sonicwalls aren't Cisco but you sure as fuck can't say that those shitty $30 blue dlink specials are the same as a 24 port sonicwall, there is NO comparison. Like I said it wasn't because this moron thought the Dlink was just as good, it was because he had never done jack shit but home installs and didn't know his ass from a 7/11 when it came to pro gear.

And look I got NO problem with "Tigerdirect specials" as i make my bread and butter off the boxes..but I would NEVER EVER try to set up a business on those, not unless they gave me enough to set up the whole office AND have some spares in case one goes down...why? Its the drivers dude, trying to support a mish mashed mess of hardware where no damned driver matches is a royal PITA to admin, and this guy had dribbed and drabbed the place. The worst part? He built them fucking OCed from the factory "gamer specials!", my guess is because he wanted one for himself and just added it to the bill, but fuck! You wanna talk about some seriously fucked up issues! Why don't YOU try dealing with over 30 OCed from the factory unstable messes, where even the RAM timings are aggressively OCed! And this was for a fricking law firm, not some gaming place, yet every damned system had gamer cards!

Sure even if you don't like the sonicwalls you can see that situation was a clusterfuck from hell. Frankly they only gave a shit about his lowball price and never once stopped to ask "Why is he cheaper than the other guys?" and the reason he was cheaper was he didn't know shit, he was some "gamer guy" pretending like he knew enough to do corporate. I mean picture trying to track down a networking issue only to open up the closet and find more than a dozen of those damned blue dlink 4 port jobs with half a dozen ISPs and not a damned thing labeled? We had to shitcan the whole place and start over, it was just too big a mess to untangle and fix.

And not to rain on YOUR parade but unless you live in one of the major metro areas Apple sucks big hairy balls. I get to deal with a doctor tomorrow who has been to Apple's genius bar more than a dozen times with networking issues, all they do is reload the OS and apply the patches and hand it back, needless to say that don't fix shit. At least if he had Dell or HP under contract he could get an actual tech out there to work on the problem on site, but since Apple don't do that I'm gonna have to try to fumble my way around an OS I've avoided like an STD because frankly nobody else will even try, he says Apple and they say "fuck that proprietary crap, we won't touch it" and then they try to sell him a new system. I'm at least willing to try, which is more than he's gotten from Apple.

Re:Very true, for many reasons.

[hairyfeet]

Man...that must be a hell of a nice place to work Crosshair, congrats.

The reason i got out of corporate IT, even though it paid better and wasn't the "feast or famine" that is running a small shop is my two nephews had a kind of intervention on me. they just kept piling on the work, never giving me the resources that I needed, frankly i was looking like shit from being run so hard. One day I came in and with tears practically in their eyes they said "We done lost our mom last year and our dad abandoned us. All we have is you and grandma and you look like you are a ghost, you're so pale. We can do without the fancy stuff, we can't do without you" and needless to say I tried to give my notice the next day. Of course being the ungrateful pricks that they were they said "Don't bother, just get your stuff and pick up your check"..yeah, fuck you too.

Now as I said the pay ain't as good and its feast or famine but GOD, what a difference! When I say things they actually LISTEN TO ME! When I say they need this or that I'm not treated like I'm fucking trying to rob them, in fact I'm often asked "Are you sure that's all we need? better to spend it now than need it down the road and not have it" needless to say the first time that happened I damned near choked, talk about a difference!

Frankly if you want to know what is wrong with this country the way the corps treat IT is a perfect "canary in the coalmine" as far as I'm concerned. All they give a shit about is the quarter, never think ahead, they will pinch the fuck out of that penny while blowing pounds later on when the chickens come home to roost, but since all those PHB and MBAs...which I call Masters of Being Assholes play musical chairs they'll be gone before the shit really hits the fan so they don't give a fuck. Corp America is just one big pile of uber paychecks and golden parachutes at the top and crumbling infrastructure and braindead decisions below. Its no wonder this country is in bad shape Crosshair, its run by Masters of Being Assholes!

Re:Very true, for many reasons.

[White Shade]

I feel like even if you're not going to be doing any hardcore programming, at least having the mindset to be able to reason out the functional blocks that make something in the computer happen is a huge benefit.

If you know that to do X, you need to do something along the lines of A,B, and C, even if you don't know exactly what A, B and C are , or you don't know how to actually implement it, you're still miles ahead of someone who just doesn't get it.

That was my experience in programming classes in college too... some people just got it and could make things work relatively efficiently within the boundaries of their knowledge, but others just had no idea at all and were always so far of the mark that it was almost a waste of time for them to even try anymore.

Re:Very true, for many reasons.

[mjwx]

You don't or you can't? Because even when I'm not coding, just in day-to-day computer use, I often write scripts to batch-process or automate various things. I have found this highly beneficial. I don't want to break you down, I just want to know your reasoning. Personally, I can highly recommend learning scripting, be it bash/awk/php (yes, I wrote php scripts)/powershell/whatever, I'm sure you'll benefit from it.

I'll fall firmly into the "don't" category.

I don't consider scripting, SQL or writing out a text based conf file coding at all. I do far too many practical and useful things to consider myself a coder.

Re:Very true, for many reasons.

[LongearedBat]

make a configuration change across, say, 2000 devices (or in our case, several million)

Running a bot net, are ya?

Re:Very true, for many reasons.

[Rennt]

A wiki'tard, eh? Try the summary at the top of the page.

Re:Very true, for many reasons.

[mverwijs]

The title reads 'should know how to code'. Not 'should code'.

The difference is, imho, that if one understands code it stops being a black box. Even if it *is* a black box. That understanding will help you debug many problems faster. It's another tool in the toolbox.

But if you're happy with your hammer, then by all means, hammer away!

Re:Very true, for many reasons.

[dropadrop]

I work in a multinational company with an IT department of about 120 people (about 5500 total employees). We have maybe 10 people that write code. Why on earth would anyone labeled as a system administrator have to write code? What am I doing maintaining multiple SANs, ESX farms, MS clusters, Cisco switches, routers, VPN, and firewalls that I should know how to code?

The 10 people have that write code do exactly that. Write code and build applications. I would not expect them to have a working knowledge of setup routing our WAN, they should not expect me to code an application.

Sounds like a multinational corporation :)

I've done everything you are doing, but still occasionally needed to write code. Maybe you have a database and instead of purchasing an expensive license to be able to get clean backups you write a script which puts it in a clean state, connects to the SAN, takes a snapshot of the LUN, mounts that snapshot so you have a clean backup. Maybe you are using SNMP to monitor servers and need to get information on some service (active dhcp leases for example). Maybe you need to periodically clean old files from somewhere that fill certain requirements etc...

Most of these examples can also be solved by throwing money at it, but if I was hiring somebody I'd rather take a person who also has the ability to solve problems with sense and in a hurry. It could be the difference between a team having 5 people to maintain 100 servers or 5 people to maintain 5000 servers.

Re:Very true, for many reasons.

[Anonymous Coward]

2Cents, you are absolutely right. Even in windows systems, a basic understanding of what can be done with code can stop 5 people from running around to a couple hundred machines each.

Should the sysadmin be a programmer? Not in the conventional sense, but they should be able to programmatically attack the problems placed before them before they just brute force their way through them.

The "sysadmin" can be anything from a guy who runs a single mail server, to someone who has to deal with thousands of remote workstations. There's no hard and fast definition for what exactly a "system admin" actually is. This entire discussion is just a collection of people assuming that every other "sysadmin" has the exact same job environment, the same duties, and the same problems. Nothing could be further from the truth. So the answer to the submission is "Define what you mean and you might get a decent answer".

I've seen sysadmins who simply have no need to ever code or script anything, and others who need to do it all the time. I've seen sysadmins who will spend an hour coding and save a week's worth of work, and I've seen sysadmins who will spend a week coding to save 5 minutes work.
Do you need to know how to code? It ALL DEPENDS on the job you're given. Usually it will be of benefit, but it's not always required.

Re:Very true, for many reasons.

[hairyfish]

I never trust a plumber who can't wire up a fuse box either. Same goes for a doctor that can't practice law. Fuck specialist skills, everyone should know everything

Re:Very true, for many reasons.

[hawkinspeter]

In an ideal situation, you'd have a test network. However, it tends to be expensive to duplicate network equipment and it's difficult to justify the expense when network switches don't have their config changed very often.

Virtual networks are useful to test some aspects, but the chances are that it won't necessarily duplicate everything on the production network and testing might not highlight a problem that occurs on production.

Also, network equipment doesn't tend to be as programmable as computers and a typical switch config won't end up being many thousands of lines long, so there's a huge difference in complexity between software programming and network configuration.

Re:Very true, for many reasons.

[Chris Mattern]

You missed the OP's point. The point isn't that you'll necessarily be coded for the system you have. The point is that knowing how to code provides critical understanding of how a computer operates that a sysadmin needs regardless of whether or not he's actually coding.

Re:Very true, for many reasons.

[MS]

If not a botnet, maybe he's running Google. :-)

Re:Very true, for many reasons.

[Wolfraider]

Programmers code programs Administrators code scripts The difference is that programs are used but lots of people and generally have nice GUI interfaces. Scripts are used by 1 or 2 people (sometimes more) and are used to make your life easier.

Re:Very true, for many reasons.

[Krneki]

I think you missed the point. The coding helps save you time in the long run. Not the other way around.

No it doesn't. I'm overbooked and I refuse to work more then 40 hours a week, because you know you work to live not the other way around. I can code in Perl and every time I change job (because it's the fastest way to earn more money) I end up automating up to 2 person day in 1 year. And this is all done thanks to my spirit for initiative, most of the IT bosses don't even know what I'm doing or what could all be achieved with scripting.

But even if I work for 3 or 4 "standard IT" guy in all my life scripting has never saved my time if anything it eat my time, since once they realized that I can solve most of the problem they outsource I end up with more work.

right now I'm freaking coding Perl script to handle financial files between one task and another when I have 5 free minutes (imagine the code quality) and if the partner doesn't get the right amount of money, they don't ask the guy who is responsible for the file flow where is the error, they come to me, since I made a little script to translate the file content for different systems like a fucking care for the whole file flow across 6 different servers.

Sorry for the rent, but again I'm burned out and I need to change my job.

Re:Very true, for many reasons.

[TheLink]

Using Cisco stuff? Juniper allows you to specify an automatic rollback after X minutes if not confirmed.

With Cisco you can kludge it with "reload in 5" (which tells it to reboot in 5 minutes) and then reload cancel if stuff seems to work (followed by a real commit if stuff really seems to work ;) ), but rebooting a router is not the same as rolling back a configuration - since state is lost (route tables etc). But still better than having to manually power cycle the stuff.

Re:Very true, for many reasons.

[bolthole]

The appropriate, time-honored solution to your problem is:
stop asking permission to do so.
Just write the script (very, VERY carefully, making sure it has an insane amount of safety checks), then give it a limited test run... then if it works out okay, start using it fully.

Even a cautious manager will usually be impressed once they find out you've been using some script you wrote, in production, for a year, and they havent heard of any problems with the system.

Re:Very true, for many reasons.

[bolthole]

Darn, no edit button.

I suppose I should add the disclaimer to the above, that it presumes you are a competent, experienced script coder.

If you are not... then please disreguard my above post.

Re:Very true, for many reasons.

[TheLink]

Guessing your likely labour cost and the downtime costs, I'd definitely not want you to have to solder it again, even if it's 1 buck per capacitor. But he might have been checking just in case you were doing something silly/dubious.

As for WD Blacks how do they compare with the WD AV drives for your purposes? http://wdc.com/en/products/products.aspx?id=150

Re:Very true, for many reasons.

[Synerg1y]

how is knowing how to code beneficial to administrating an exchange server???

Knowing how to script on the other hand... that would make a lot more sense.

Re:Very true, for many reasons.

[thetoadwarrior]

But windows server has a GUI for everything. There's no need to program when you've picked what is obivously the best server in the whole universe.

Re:Very true, for many reasons.

[countach74]

I agree, we work to live. But I think perhaps you've been making mistakes, such as informing your employer that you can automate tasks to save time. Just use it to your advantage so you can be lazy. http://www.thegeekstuff.com/2011/07/lazy-sysadmin/ :-D

Re:Very true, for many reasons.

[RatherBeAnonymous]

Here's the problem I have with management.

First, I spend a couple hours writing and testing a script, then I deploy that script out to our 300 or so computers in about 10 minutes. Then, the next time something comes up, they think it should only take me 20 minutes, soup to nuts, and by the way, it has to be done in half an hour. They forget the lead time it took me to come up with that 10 minute fix.

I do have to agree somewhat with management wanting vendor solutions. If an admin develops involved scripts to do everything and then changes jobs or gets hit by a bus there may be no one to support those scripts. It's even worse if an Admin uses some off the wall language.

Re:Very true, for many reasons.

[scubamage]

Lol, I'm not nearly wealthy enough. Just about 30 million cable modems though, which you talk to via SNMP (so I guess in a way yeah, it's kind of like a botnet?)

Re:Very true, for many reasons.

[hairyfeet]

Welcome to the club friend, coffee and donuts are in the back. your story is sadly what I've seen more and more the past decade, damned good IT guys that know their stuff that the PHBs just shit all over until they become so fucking stressed and burned out they look like walking corpses and hate their lives.

Like I said my family actually pulled an intervention on me, i was looking so bad from the constant stress and being told to constantly do more with less, and while I don't make near what I did I'm happy, I have color in my face and don't dread going to work, I even met a really nice gal who needed her PC fixed and we've been together more than 5 years now.

so while I can't advise you to take the path i took, I'd STRONGLY urge you to get out now while you can. remember you only get one shot at life and I know too many guys who have been worked to the point of failing health by PHBs that just don't give a shit. life is short, don't throw it away man. i just wish i could go back and tell myself that before i used up nearly a decade of mine in corporate, I probably wouldn't have a head of grey hair now.

Coders should know how to admin

[Anonymous Coward]

...and support for that matter.

Re:Coders should know how to admin

[scubamage]

Agreed. Several companies I have worked for have had coders who couldn't care less about the fineries of system administration. All they care about is having a nice development environment and a working svn/git where they can store their code. They couldn't care less how the infrastructure behind it works.

Re:Coders should know how to admin

[freeze128]

Coders should know how to code!

Sure thing

[Anonymous Coward]

Next you'll tell me my developers should know how to admin a server and do so at a drop of a hat.

Sure

[xaoslaad]

And if you're going to say I'm a programmer then pay me like one. I don't think most sysadmins get paid as much as programmers, and I don't think most companies want to pay sysadmins as much as developers.

Also, developers trying to write tools for sysadmins usually suck at it, unless they've been a sysadmin at some time in the past. I have used a few products lately which are trying to solve all our sysadmin problems, and the one that doesn't suck comes from a dev who is a former sysadmin. And when I talk to him and make suggestions he sees exactly where I'm coming from.

Developers just want to solve use cases that fit neat little scenarios without any corner cases, and it shows when their tool is so inflexible as to be useless.

Re:Sure

[Anonymous Coward]

Actually I do. I have written a feature or two. Surely not as cleanly as the full time developers could, but sufficiently to make it work so that they can take it, clean it up, and pull it in. I am also not afraid to look at source code, a luxury I can take because we use primarily open source, and I have pin-pointed faulty code more than once. I also help out some of our developers packaging their software even though it is also not my job, so they can spend more time doing what they do best, because ultimately I believe it will benefit us both. None of that changes the fact that I am not a competent developer.

I do however believe firmly that I am competent and dedicated sysadmin. That's more than a lot of people in this industry can say. But now the insinuation is that I should have two valuable skill sets, and one I don't enjoy doing at length even. Ok then, give me time on the job to learn, pay for the courses so I can relearn to code competently, and then triple my salary. I aced all my CS classes in college, whether in Java, C, or Assembly. One thing all those hours and hours and hours coding made me realize is that I didn't want to do that shit for a living. I still don't. I do what I like doing now.

Now don't you feel silly for assuming what I do and don't do along with what I did and do know?

Re:Sure

[Anonymous Coward]

"take it, clean it up, and pull it in"

I'll gladly bill you for cleaning your shitfight code and submitting it for review and test prior to even considering it in production. Thanks.

Re:Sure

[Lando]

Hmmm?
      Real sysadmins in general are paid more than the programmers. Sure the hotshot program leads get paid more, but a non-toy os sysadmin makes decent money.

Not only admins

[Mr2cents]

In general, everybody dealing with computers can benefit from a bit of programming knowledge, not only admins. The rule of thumb is: if you're doing a repetitive, braindead job, you're doing it wrong. Computers are built to do exactly that. A small script can automate a lot of work for you, if you have that skill it can help you tremendously.

Re:Not only admins

[Mr2cents]

PS: YOu don't really need to know how to program, if you can just identify tasks that are braindead and repetitive, that's already a plus. Then you can go and talk to someone who does know how to program, explain the problem, and this person might come up with a simple solution for you. It all boils down to this.
Unfortunately, many people are not trained to identify automatable jobs.

Re:Not only admins

[drosboro]

It all boils down to this.

That's a great graph. Took me a minute to realize that it was very strange to me to have time on the y axis, since it so rarely appears there - but once I got used to tilting my head at 90 degrees, it worked.. :)

Re:Not only admins

[bbelt16ag]

then i would have no work to do at all..

Re:Not only admins

[Mr2cents]

You're right. But you could always write a script to flip the axes, might come in handy if you encounter similar graphs ;-).

Re:Not only admins

[Mr2cents]

Uhm, did I tell you to inform your boss? You can get all your work done (without errors once you perfected the script), AND read slashdot full-time. Win-win.

Re:Not only admins

[Kjella]

In general, everybody dealing with computers can benefit from a bit of programming knowledge, not only admins. The rule of thumb is: if you're doing a repetitive, braindead job, you're doing it wrong. Computers are built to do exactly that. A small script can automate a lot of work for you, if you have that skill it can help you tremendously.

Let me guess, you're a developer? Yes, yes and it'd be great if developers understand a little about sysadmin so they knew how their software would be run... oh and understand the business a little better... oh and understand support and the issues their having.... oh and understand the sales and marketing people who little who has to try to sell this stuff... oh and understand the economists so we actually do things profitable too... and, yeah okay there's a zillion things that would be somewhat useful. I don't mean everybody should put on blinders and just stare at their own little piece of the big picture, but most people have plenty improvement potential in their own job rather than go off to learn the basics of an entirely different job. You're probably comparing the speed it'd take you to whip up a script rather than solve it the manual way, not the time it'd take them.

I'm barely qualified to change windshield wipers, tires, light bulbs and at a stretch possibly oil on a car. An auto mechanic would probably think I'm a total n00b and I'm sure to him many things looks super easy and he can't understand why people don't bother learning the basics of auto maintenance and repair - though I'm sure he doesn't mind the business. Well life's too short to learn all those things I don't have any particular need or interest in learning and so is work life too. I have no problem with people that choose to be specialists, sure you need some "glue" people to make them talk to each other but the value of real experts who know their area inside and out is immeasurable. If it's not a big deal that needs automation they'll just do it, if it is a big deal they'll get on the phone or email or talk to their boss and find someone who can automate it.

Besides I thought most developers hated most wannabe developers that pop out some kind of abomination in VBA or something similar and when it outgrows their limited skills the whole pile is dropped on somebody on the IT side. Code hacks aren't exactly know to build things that are reliable, maintainable or even sane. I certainly wouldn't want to give them the impression that it's so easy anybody could do it as a side gig to their normal job, that's a disaster waiting to happen. Yes yes, we get it you're doing something very important in automating all the crazy that'd otherwise be a huge time sink. But those other people are probably also doing something fairly essential to the business or else they wouldn't be there. I'm sure they all wish you'd understand some of their job, but you'd feel spread pretty thin then wouldn't you?

Re:Not only admins

[scubamage]

I'm a systems engineer for a fortune 500, and I spend a significant amount of time developing. Now, am I making fancy OOP code using an IDE? Nope, not really. I am however making python/perl/awk/expect/tcl/shell scripts with a heck of a lot of frequency. I don't think I could do my job effectively if I didn't.

Re:Not only admins

[garcia]

Not trained? What? Most people I know realize the tasks they do could be automated. However, they do everything in their power to ensure they aren't because they believe it keeps their relatively mindless and easy job "safe". This mindset is prevalent in the public sector and the unionized public sector especially.

I was able to come in and completely revamp a position I was hired to do to expand it to encompass at least 50x more work with a little Access/VBA and some learned-on-the-job DW knowledge.

They are still running the same exact reports I wrote when I left 5 years ago and haven't added a single one to the mix. Someone has now taken over my position and enters text in the fields the scripts prompt for and passes out the paperwork it automatically prints. It's a sad day for our tax dollars.

Re:Not only admins

[blade8086]

Or - just look at it this way:

The only way to use a *computer* is to program a computer - otherwise, you're just using *software*.

Re:Not only admins

[Mr2cents]

Personally I don't hate "amateur" programmers coming up with a solution that makes my hair stand up right. I applaud it. If it saves them time, they'll use it. If not, I hope they recycle the bits (especially the 1's, they pollute a lot more than 0's). But seriously, you bring up some important points:

1) Is the effort spent worth the time gained? That's something you should *always* ask yourself. Even for seasoned programmers this is not always a clear-cut case. Personally I found this technique quite useful: consider the worst case. If in this case you lose some time, is it a lot? No? Then try to automate (depending on your concrete workload and other factors of course - this is an exercise you'll have to make yourself).
Rationale: You might lose a bit of time, but if you do this consistently, on average you break even or gain time. But more importantly: you'll gain experience in automation tasks and software tools. This will help you make better judgments later on.

2) Every line of code written will have to be maintained. This is a big issue in some cases. Never assume you will live forever, nor work forever for that company. If the tools you create are mission-critical to the company, it should be discussed with higher management. At the last place I worked, an employee created a full system for automating hardware production in Delphi. When he left, there was nobody to maintain it. However, since it was so central to the company (it really increased efficiency), they hired another Delphi programmer to take it over. Not everything is that complicated however. You can use personal scripts that won't be missed but still save you some time. Worst case: your successor will do it all manually again until he gets annoyed and writes his own scripts, or he discovers yours in a backup.

I must say I find your attitude toward learning a bit depressing, though, personally I grab every opportunity to learn something new. Especially if it has direct implications on my job performance, but not even limited to that.

PS: I love your sig :-).

Re:Not only admins

[Mr2cents]

This presumes that people who know how to program are willing to do things for you that they might reasonably expect you to do for yourself.

Indeed. In some companies this might be a problem. If that's the case, it's a mentality problem, not a technical one. Where I worked, increasing productivity has always been appreciated. I believe that's the right attitude.

Re:Not only admins

[Mr. Freeman]

> Never assume you will live forever, nor work forever for that company.

This is very important. I worked in one place where this one guy was responsible for a bunch of shit. I would often think to myself "if this guy is hit by a bus we'd be fucked for weeks". The guy got sick and was out for a few days. Lo and behold, work ground to a halt until he got back. During this fiasco everyone was saying that we needed to make changes and have redundancy so this doesn't happen again but once this guy came back everyone forgot all about that. I guess people don't care as long as things are working. I tried to bring it up a couple times after that but the only response from everyone was "we don't have time for that, we'll do it later". Oh well, it's their funeral.

Re:Not only admins

[tibit]

I think that these days programming is like being literate. Sure you don't really need to know how to read and write, your secretary can do it for you all day long. It just seems like a big waste of effort to me. For many simple and repetitive things related to their jobs, everyone should be able to whip up a quick script, just like everyone takes notes as needs arise, without calling their secretary on the intercom every time they need to write down a phone number.

Re:Not only admins

[tibit]

The "abomination in VBA" may be so, but it has likely replaced tons of manual labor. When it outgrows the original author's skills, then there's no reason not to do a proper reimplementation. Yet all those abominations, usually, make their authors more productive. That's their goal. Scoffing at that is going full retard.

Re:Not only admins

[tibit]

I am such a "responsible for a bunch of shit guy" and these days I pretty much work on solutions to make myself obsolete and available for other work. This includes developing a bunch of software and even hardware to replace various manual processes that ultimately were only as good (or bad) as I was. Where quality is concerned in manufacturing, an automated task always beats my performance, at least in my job. I'd have thought everyone in a similar position would realize this, and the fact that them being irreplaceable isn't good for themselves or the company they work for. Being irreplaceable means you can't do what you'd want or even anything related to what the business may need in the future. It keeps you focused on here and there and it's easy to lose perspective that way. It's hard to innovate while being irreplaceable, I'd think, or at least when you're intermingled in day-to-day operation so that it distracts from "futuristic" projects you may wish to undertake.

Re:Not only admins

[bloodhawk]

The benefit to knowing how to program is not just being able to identify what can be automated, it is about understanding what is happening under the covers which can greatly increase your ability to troubleshoot and identify weaknesses before they become issues.

Re:Not only admins

[strikethree]

The rule of thumb is: if you're doing a repetitive, braindead job, you're doing it wrong.

You are so very correct here. I have ran into numerous situations in my professional life where I was thinking to myself, "Oh. My. God. Are you guys seriously working in that way?!".

Oh, concerning your sig:

"It's too bad that stupidity isn't painful." - Anton LaVey

Stupidity -is- painful. The stupid people just live with the pain... but your sig is apropos concerning scripting. :)

Re:Not only admins

[crash123]

You could write a script that would type my thesis for me? Cool

Re:Not only admins

[Mr2cents]

If your thesis is braindead and repetitive, sure:

for i in `seq 10000`; do echo -n "blah "; done

Re:Not only admins

[Ash Vince]

I'm a systems engineer for a fortune 500, and I spend a significant amount of time developing. Now, am I making fancy OOP code using an IDE? Nope, not really. I am however making python/perl/awk/expect/tcl/shell scripts with a heck of a lot of frequency. I don't think I could do my job effectively if I didn't.

Firstly, let me say I have worked as system admin and programmer, currently I am a technical lead for a small business so do a fair bit of both since our full time system admin left a month or so ago.

The big difference between throwing together few perl or shell scripts and learning to be a full time developer is the complexity of the problems you have to solve using code. Maybe the scripts you write are more complicated than I am giving you credit for, but when I put together scripts I never had to create any longer than 50 or so lines.

Out system admin once told me as a developer that we should not ever give our servers ip address to a partner company so they could lock a webservice down to only being called by us. He did not want this to make it harder for him to do his job and he needed to make some fairly big infrastructure changes to our application servers.

He had a point but the problem is that he didn't appreciate that by us locking down something by IP it made the integration between our to system simpler to implement and far more secure. It might not have been the most flexible solution, but since it saved a great deal of time and hence cost it was the best fit.

By being able to wear both hats it makes you better at both roles in my opinion as you can see ways to simplify a solution via the system admin route that may not have been possible by just looking at it from a coders perspective, and vice versa.

As to whether this justifies the cost of keeping a system admin's coding skills up to date is another matter, I think it can be justified though by instead trying to split the system admin role between more that one staff member even if it is exactly a one person job. By doing this you end up with a number of advantages:

1) Increased resilience to staff turnover. If one of the people job sharing the system admin role leaves then the other person can still be relied upon so covering absolutely every detail during a handover period is not so critical. Obviously this does not mean that no handover is needed though apart from in the simplest of setups.

2) More flexible approach to scheduling. If you have more than once person in a team who can do a given piece of work then it make it easier to schedule in a new task even if you have something similar already.

3) Better oversight. If you happen to have one empire building nut job who is building back doors in left right and centre, he will most likely be caught and reported by the person job sharing with them. If you have two people who work together to screw you over then you have more serious problems than just them. Even assuming you do not have staff trying to do such things having two people who can provide mutual oversight of each others work often helps produce far better solutions.

This will not apply to every company out there since some businesses probably have no need for any software development resource but if you work in a company that does, it makes sense to job share as much as possible between the two roles.

Re:Not only admins

[cas2000]

In general, everybody dealing with computers can benefit from a bit of programming knowledge, not only admins. The rule of thumb is: if you're doing a repetitive, braindead job, you're doing it wrong. Computers are built to do exactly that. A small script can automate a lot of work for you, if you have that skill it can help you tremendously.

Let me guess, you're a developer?

I'm a sysadmin (and all of my coding is systems automation and data mangling stuff - i have no interest in applications development or coding something more than a few hundred lines in size) and think that's a perfectly normal and reasonable attitude to have.

doing stuff manually is boring, time-consuming, and prone to error. for repetitive (or even just one-off but large tasks) I'd rather write a script to do it - even if the script is effectively little more than an executable plan of action, I still come out ahead because writing the script forces me to think about what I'm about to do, what every little step is, what might go wrong at each stage, and so on.

also, writing a script is more interesting work than doing it manually....even if it doesn't save any time, it makes my job more interesting, so i'm less likely to get bored and start job hunting.

my languages of choice are bash, sed, awk, perl, and occasionally python. Sometimes I debug and make tiny fixes to programs written in C or Fortran or some other language. these are all what I would consider to be normal skills, a minimum to be expected of anyone doing sysadmin work.

Yes, yes and it'd be great if developers understand a little about sysadmin so they knew how their software would be run...

yes, that would be great too. my work would be much simpler if devs had a clue about systems.

Duh

[countach74]

Isn't this all rather self evident?

Automating tasks

[gmuslera]

That usually implies a bit of programming at the very least. Maybe a sysadmin no need to be a large projects developer, but the small tools that could make his life easier would make a big difference.

Another skill we will only get underpaid for

[NetNinja]

The ammount of skills Network admins should know and still get paid a 10th of what we are worth is ridiculous.

I should be making close to 300k a year, Do you think any company would pay that?

So now I need to add codder to the resume which will effectively fall on my shoulders to maintain code? Bad idea.

I guess I should be happy and count my lucky stars that I have a job?

The one thing I have noticed in companies that Sales weasels get paid a lot more money than I and the responsibility lies on the network admin to keep it all running.

Re:Another skill we will only get underpaid for

[countach74]

The one thing I have noticed in companies that Sales weasels get paid a lot more money than I and the responsibility lies on the network admin to keep it all running.

Yes. It's a backwards society. Black is white. 2+2=5.

Re:Another skill we will only get underpaid for

[Anonymous Coward]

The ammount of skills Network admins should know and still get paid a 10th of what we are worth is ridiculous.

I should be making close to 300k a year, Do you think any company would pay that?

So now I need to add codder to the resume which will effectively fall on my shoulders to maintain code? Bad idea.

$300,000? Seriously? And you don't even know how to spell amount or coder... ammount? codder?

The one thing I have noticed in companies that Sales weasels get paid a lot more money than I and the responsibility lies on the network admin to keep it all running.

How many sales did you make last quarter? Zero again.

Re:Another skill we will only get underpaid for

[Anonymous Coward]

5? I thought 2+2=3 ... They increased it without paying me more? Bastards!

Re:Another skill we will only get underpaid for

[Anonymous Coward]

Yes. It's a backwards society. Black is white. 2+2=5.

It does, for very large values of 2.

Re:Another skill we will only get underpaid for

[PPH]

The next thing you know, the admins will want to be let out of the basement.

Perl

[slapout]

So, does Perl drive you crazy or do you have to be crazy to program in Perl?

Re:Perl

[guygo]

Yes

Re:Perl

[toygeek]

Yes.

Re:Perl

[bill_mcgonigle]

So, does Perl drive you crazy or do you have to be crazy to program in Perl?

Some people just can't get their head around Perl. That's OK, there's always python. We'll treat you with compassion.

Re:Perl

[scubamage]

Honestly I never thought perl was that bad, but I actually have a pretty good grasp on regular expressions and otherwise I like the shortcuts that perl includes. It lacks the clarity of, say, a nice and well assembled shell script, but if you're making something that tips the balance between trying to make a shell script do something it's not really meant for and just using a normal language, perl is a great choice. Granted you can end up with some cryptic to look at stuff, but that's what documentation is for.

Re:Perl

[grouchomarxist]

Having to document code is a code smell. Documentation is something that can easily fall out of sync with the source. It is best to program in a fashion that doesn't require documentation, if possible.

Re:Perl

[scubamage]

I disagree - if documentation is falling out of sync with the code, then whoever is writing the code isn't doing their job properly.

Re:Perl

[grouchomarxist]

There is a principle you've probably heard of called Don't repeat yourself (DRY). If you're unnecessarily repeating yourself you're creating opportunities to introduce bugs. So it is good to avoid documentation (such as code comments), if it isn't necessary. If it is necessary for whatever reason, then it is a good idea to create an automated system that checks to see if they stay in sync.

Re:Perl

[jampola]

I've moved from PHP to working with Perl and I couldn't be happier. I've even started creating graphical applications (using tk) to support some of my web based apps (created in PHP) and I couldn't be happier! Yet I am the kind of person who can't wrap my head around Python, Haskell or Rails no matter how hard I try!

Re:Perl

[jampola]

Isn't that why functions/subs were invented? :)

Is this what any programmer does?

[PeanutButterBreath]

I'm fairly certain that if all I did was write Perl, I'd go insane.

As a programmer, to me this is like someone equating author and typist. Code is just a medium. Figuring out what to do with it, and how, is the fun part.

Remember the judge?

[michael_rendier]

I remember the judge... http://lazytechguys.com/news/business/judge-for-google-and-oracle-case-is-also-a-programmer/

Unfortunately

[sackofdonuts]

Many IT professionals called systems administrators have no idea how to code. They basically perform the strict description of systems administration activities. But those who can code tend to be the better sys. admins. and usually end up directing those who do not know how to code. Coding is one of many tools a proficient sys. admin. has. Another is problem solving skills. Believe it or not there are systems administrators out there that can actually solve a problem without calling the vendor.

Re:Unfortunately

[scubamage]

It's because coding involves logically breaking down complex problems into simple steps. Now, when you have an entry level sysadmin (someone with 1-5 - sometimes less - years of experience) they basically do simple tasks. Reset a password, clean up an os, do installs, fix issues, etc - these are honestly trained monkey tasks that any geek squad kid can do. I know, I was one of them (before geeksquad, but the point stands). As they get further down their career paths they SHOULD start to undergo two changes. First they should be more proactive/less reactionary, and second, they should learn to plan before executing. Breaking down complex problems into functional components is an essential part of that design part. If you don't teach yourself to break things down into simple steps and specialized software/hardware, you're hamstringing yourself and your career.

An Assortment of Tools is Better Than Just One

[Gim Tom]

I was never a developer as it is now called, but I wrote code in about half a dozen or more languages to solve a problem that needed to be solved. I worked with System Administrators and Engineers that had never written a line of code in any language and to me it seemed that they were working blind and with one hand tied behind their back

Re:Two different worlds

[countach74]

But the sysadmin's job is to support the code and OS, not figure out what is wrong with the code when things break.

Strawman's argument. The article clearly is talking about using programming/scripts to facilitate the system administration jobs, which is a no brainer. It does not say to be a developer and hack in some change to a large app to fix an issue. Or perhaps I missed that part?

It's not all about the code

[boundary]

As an IT manager, I don't want my sysadmins to code. I want my coders to code. I want my testers to test. And I want my sysadmins to...ermmm...admin the sys. Mixing roles & responsibilities can be really dangerous, especially when there's a strict development and deployment lifecycle in place (and if there isn't you're living in uncontrolled chaos), including change management and release management, that must be gone through properly. Uncontrolled changes, with sysadmins using production systems like they're test environments, is a bad idea and will end in tears.

Re:It's not all about the code

[aXis100]

Wow..... just wow. As an IT Manager you have failed to grasp the concept of this article, and that is a worry.

They are not talking about sysadmins writing production code - they are talking about using one of a variety of scripting languages to solve sysadmin problems - eg repetitive tasks like backups, deployment scripts etc, maybe even some html status monitoring screens or a cactus plugin.

Re:It's not all about the code

[PlusFiveTroll]

Err, Jesus man do you work for the telephone company? I swear to god each one of them has a little tiny job they know how to do and all other information is kept from them. Paperwork has to be filed for them to talk to any other person.

Anyway, this article isn't about any of this shit you just stated. It's just stating that sysadmins that know how to code notice problems faster and give better bug reports.. He doesn't even have to 'solve' them, he can pass that information on to the people that do.

Re:It's not all about the code

[scubamage]

So, since you're responsible for the finances of your team, would you prefer to pay your sysadmin for 300 hours of work to walk to 600 workstations and manually make some changes, or pay him for 5 hours to write and test a shell/powershell script allowing him to go back to work?

Re:It's not all about the code

[boundary]

You have failed to grasp that *anything* that could influence correct operation of a production environment, even a little 'harmless' script that manipulates production systems and/or their data, needs to be managed correctly, assessed for impact (by someone other than the guy that wrote it), tested properly and deployed carefully. Risks to production need to be managed, and the biggest risk I know is a sysadmin with a god complex and a hobby in scripting.

Re:It's not all about the code

[boundary]

The sysadmin is one member of a team. The article's talking about 15000 lines of code, which isn't a small amount. That's dangerous territory for sysadmins to get into. Do they understand design for performance, security, error checking, maintainability, corner cases? Have they done the right analysis? Are they solving the right problem? Do they understand how to test? These are not rudimentary skills, even though people think they are. The likelihood is that they haven't done enough to get past the company's (not my) Change Management/Release Management process (which is a fact of life in most organisations that have any level of sophistication) and the coders will be able to do it better, faster and get it through first time. So the total cost of ownership is what I'm looking at, it's not just about solving an immediate problem - it's also about how much effort it will take in future to keep that problem solved.

Re:It's not all about the code

[dbIII]

I think the discussion is about items that do not directly have an impact on production software and about sysadmins being able to understand enough about what the developers are doing to be able to spot obvious mistakes.
I really don't understand how somebody can effectively run a large number of *nix systems without being able to code, and with powershell etc it's probably the same with MS Windows.

Re:It's not all about the code

[boundary]

You don't understand change and release management, which is fine. Having seen the mess people get into without it I'm fine with having such gates in place. They're not hard to get over as long as people understand that the impact of running a shittily-written script in production is wider than them having to debug it. I think if you read the article again (or perhaps for the first time), you will find that actually the article's talking about how having a scripting language under your belt will help you automate a lot of the day to day tasks a sysadmin would otherwise need to be doing manually. This is about business process automation, not raising bug reports.

Re:It's not all about the code

[boundary]

The guy that wrote the article was talking about network monitoring and licence monitoring, so I'd assume it's something important to keep humming, and would therefore be classed as a production environment from some angle. As above, the article's really focusing on how to automate mundane tasks, not how to work with developers. I reckon every sysadmin I've ever known has been able to write scripts. Have all of them been of a level of quality that I'd want near my systems without some additional oversight? Nope.

Re:It's not all about the code

[boundary]

Yes, in the same way I trust a coder to write a Hello World program flawlessly. In anything longer, mistakes creep in.

Re:It's not all about the code

[PPH]

There's a difference between knowing how to code and doing it outside of established processes. The knowledge enhances the admin's ability by giving them a big picture view of the product life cycle. The same holds true the other way around (coders understanding admin tasks).

If the only way you can keep your employees from mucking about outside of established procedures is to hire those who don't know anything beyond their job description, you've lost control of your people.

I've worked at employers who insisted on hiring "technological savants" (a term coined by Scott Adams to describe employees totally unskilled outside of their narrow field). What you end up with is trained monkeys. People with no interest or motivation to improve themselves, their jobs or their products.

Re:It's not all about the code

[PPH]

So, you are willing to accept an admin task performed manually, mistake creepage and all. With 500 workstations, there's no guarantee that the last one will be configured the same as the first. After the first few dozen, fat finger mistakes will undoubtedly creep in. By number 400, your admin won't be seeing straight anymore.

When we say "admins need to understand coding" this includes all of the associated issues of testing and configuration control. Perhaps not to the same level of detail as the code for the product. But in some cases, it can come pretty close.

I'd much rather have my admins script everything. And save the script. So when we come running in and ask, "What the **** did you do??!!", they have the exact steps in hand.

Re:It's not all about the code

[boundary]

1 line of code, run 500 times, does not present the same level of risk as 500 lines of code run once. I'm not saying don't run scripts. I'm saying scripts need to be developed carefully, within a process, and there's better people to do them than sysadmins alone.

Re:It's not all about the code

[boundary]

Fair call from the complexity standpoint. However the assumption that each program is unrelated to the rest is stretching it, I think. Most situations in a relatively simple environment would call for some level of daisy-chaining (I think the author of the article even suggested some level of interaction between his scripts outputs.) So the complexity moves from the length of an individual script to the number of interaction paths between multiple scripts. And of course, as the infrastructure changes, these scripts have to be maintained, etc etc. So yeah, not the WOAH moment the 15000 lines suggest, but it can still be death by a thousand cuts.

Re:It's not all about the code

[boundary]

Absolutely. Knowledge of the lifecycle is one thing, putting it to use in an officially sanctioned capacity is quite another. Your third sentence isn't the case, so it's irrelevant in this context.

Re:It's not all about the code

[boundary]

Ability to cut code isn't a primary reason for hiring a sysadmin. It's absolutely a primary reason for hiring a coder. And well done. The same rules should apply to everyone. Historically sysadmins haven't been held to the same standards as everyone else. So they have been able to get away with murder.

Re:It's not all about the code

[PPH]

scripts need to be developed carefully, within a process, and there's better people to do them than sysadmins alone.

So, who do you suggest write the scripts? And how do you get the domain knowledge and requirements from the system administrators to these script coders?

Re:It's not all about the code

[boundary]

That depends on what the scripts are for. Most of our infrastructure is supported and monitored by relatively robust enterprise management systems now (which is a reaction to feeling the pain of a decade of un-managed scripting [before my time], and doing something about it) so there's not a huge amount of string-and-glue solutions needed at the moment. There's some low-risk stuff that can be done by *certain* sysadmins as the prime, but supported by a part-time BA and tester, with scripting expert resources (usually a coder) on tap as required. They all work within a defined process - that's the agreement we have with them, and they seem happy with it. For the more complex stuff, which is higher risk by nature, I'm keener to prime it with a coder who knows a scripting language inside out, and who has some level of infrastructure experience. They will have a part time BA & tester working with them, and a sysadmin on tap as required. Those are the approaches I've taken, although it will ultimately depend on your organisation's skill mix and personnel availability. This approach seems to work quite nicely, in that we're achieving predictability in our infrastructure operations.

Re:It's not all about the code

[tibit]

Who says the sysadmins are playing with production systems? The sysadmins may code simply to automate their lives, for example. They may code so that they can gather statistics, or configure things, etc. The development and deployment lifecycle sure as heck is needed, but your whole company is probably not subject to it, and even if it were, whatever the sysadmins manually do would be subject to the same process, so whether it's done manually or automated doesn't matter -- auditing etc. is exactly the same.

Re:It's not all about the code

[tibit]

So if a sysadmin does it manually with all the risk of mistakes it entails, that's fine and dandy, but if they dare write code for it, only then it's subject to the holy process? LOL.

Re:It's not all about the code

[tibit]

What you don't understand or just seem intentionally obtuse about is that change and release management applies to the entirety of the process, whether manual, scripted, done by robots, quantum computers, mercury memory lines, million monkeys, or what the heck ever. It doesn't matter if it's done by a script or a $10/hr intern typing the same thing on a thousand consoles. It must be subject to the same process. Once that's clear, I don't see how scripting and manual process differ, besides burning coal vs. carbohydrates.

Re:It's not all about the code

[PPH]

So you are trading an admin that knows coding/scripting for a developer with 'infrastructure experience'. That's fine if its all under the same roof. But what happens when upper management moves your developers/coders to India? Or you never did have developers on staff (installing/maintaining an OTS product, for example). In the first case, you have to formalize the requirements definition for the admin scripting, send it off to the (remote) code group and endure the contract issues with deliverables passing between two businesses. In the second, you'll have to build a software development team to handle the scripting task. And that will be visible to the management that promised an OTS, "no home-brewed apps" environment.

Don't get me wrong, I'm all for establishing and following processes. But with a well disciplined group, that can be achieved with admins that can put on developer, tester, configuration control hats as needed.

Re:It's not all about the code

[boundary]

Remember this line? "Those are the approaches I've taken, although it will ultimately depend on your organisation's skill mix and personnel availability." If you want to continue changing the parameters of the problem, you can find a flaw in any proposed solution. I'll leave you to move the goalposts on your own.

Re:It's not all about the code

[boundary]

Nope. You fail to understand what risk is, and how to characterise it. Stick to your knitting.

Re:It's not all about the code

[Xacid]

Five bucks says his admins are doing it anyway and not telling him.

Realistically you've got at least a dev/test environment to tinker with stuff like this. That's what it's for! Break things, figure out what works, what doesn't. Allow for some innovation!

Re:It's not all about the code

[boundary]

Most sysadmins are playing with prod systems to some degree - even if they're just gathering stats, it can have an impact, because the system they are querying still needs to spend cycles responding to the query. I have seen situations where prod performance has been degraded significantly because of best intentions to gather meaningful data. They are rare occasions, but they are still very damaging.

Re:It's not all about the code

[CAIMLAS]

Yeah, who exactly are you manage? I suspect their resumes will never make it to my desk, so I won't have to worry about it too much, but I am concerned for their well being: they need to be fostered in cross-utility.

The most useless IT employees I've seen are "specialists". They're the "Java specialists" or the "Ruby/web 2.0 experts". They're the "Windows professionals" and what have you - the people who single-skill their way through life, not really bothering to improve themselves beyond the most basic understanding of "specialist" or "expert".

The best IT employees are the ones who can do it all. No, they may not have expertise in all of it, but they're at least knowledgeable in most of it and have one or two areas of preference and excellence (eg. directory services, mail, routing, etc.). Chances are, your 'rockstar' developer or sysadmin is much more versatile than you give him credit for - he or she has just learned to keep their head down for fear of some meddlesome idiot in a different department blaming him for their own mistakes (due to their incompetence and territorial nature). I've seen it time and time again.

Change management and release management (bzzzzt! buzzword alert!) are completely unrelated to the topic at hand. Effective and useful change/release management can still be performed (though I can understand your concern for trying to comply with upper management's concerns of remaining within complete ITIL buzzword compliance, as useless as an actual aim might be). Maybe it is better to just have a 'specialist' in charge of those mundane tasks, allowing the professionals to do their jobs?

Also, if your sysadmins are using production environments as test environments, you either have programmer types (who are incompetent as sysadmins) doing the wrong job and you failed the hiring process, or you've been seriously short changing them on the resources they need to do their jobs (much more likely).

Re:It's not all about the code

[CAIMLAS]

Your concern is understandable, but change/revision/risk management in prod does not necessitate that your people do not cross skill. Cross skilled people can often see a problem where the myopic

For instance, train a skilled and headstrong jack-of-all-trades IT person into any sort of "enterprise" IT position and watch how many toes they step on everyone's toes, fixing problems unprovoked in other departments and making fools out of everyone. It happens pretty frequently (and why good sysadmins are not generally well liked in corporate environments).

Re:It's not all about the code

[boundary]

I'm sometimes guilty of the latter, but not of the former. A manual change effected by a sysadmin typing into a shell is different from one embodied in a piece of code, and they're actually subject to different processes. Change/release management does (should!) indeed apply to sysadmin activities, but there are these things called 'standard changes' which cover the typically mundane things like user Add/Move/Changes, that you don't want to go to a Change Control Board each time to get the go ahead. However they also cover stuff your sysadmin does in his/her/(but probably his) day to day work. Like pretty much everything they do, in fact. The weakness is generally nobody knows how a sysadmin spends their day, which makes these things difficult to control. But anyway, one or two changes made by a sysadmin isn't what we're talking about here. We're talking about automation, which means complexity, which means unforeseen results. That must be held to a higher standard and policed accordingly.

Re:It's not all about the code

[boundary]

I agree, innovation is the way to go, but not at the expense of the business. The team has a lot of flexibility in the way they work. But there are always some places where they're not going.

Re:It's not all about the code

[tibit]

I do engineering risk management as part of my job. All I know is that humans mess up. Routinely. It's guaranteed by design. Processes are in place to mitigate the human propensity for fuckups. Human at the console is, by design, way more likely to fuck up than a human writing a script that will go through at least some process before it even hits the test stands. On a console, you hit enter and go "fuck, now I have to roll back". Writing a script, you take a bathroom break, come back, look at it again while working on the 1-page spec, and you realize "hey, I better make sure this and that won't happen".

Re:It's not all about the code

[Xacid]

Of course, of course.

I'd just recommend not chopping off their balls at every idea. From your initial statement(s) it sounded like that was the case. Once you stifle that creativity the motivation to do a good job flies out the window with it.

Re:It's not all about the code

[tibit]

I don't see how the heck anything can hit production without there being at least a token trail to show what went wrong, if not a full blown change approval process. Everything depends on criticality of the system. If all that's going to happen is that a few people will be unhappy and you'll be sweating for a couple of minutes (likely case in many small businesses), then having full audit trail, change management and rollbacks is all it takes. If 10,000 people will be left waiting at their ATMs, there better be a couple of pairs of eyes to take a look, never mind tests etc.

Re:It's not all about the code

[tibit]

Well, but those add/move/changes should follow an automated workflow that is approved, and any deviations need to be controlled and approved as necessary. IOW anything anyone will be typing at the console is to fill in the fields, not playing at the command prompt.

Re:It's not all about the code

[dropadrop]

I can see where you are coming from, but this is still some of the dumbest shit I've heard.

You must work in a zoo or a place full of recently graduated kids if you think having a sysadmin code is somehow related to mixing test and production systems...

I have worked in telco's where all problems where solved with money and bucket loads of resources. There people had very precisely defined work positions and responsibilities where never mixed. Sure if everybody is brain dead then thinking about what the recipient of the ticket will do with it could cause chaos, otherwise it can actually help everyone do their job better.

In smaller companies problems are occasionally solved with common sense rather then money. It does not mean the changes are uncontrolled, actually I've seen far better testing and quality control in small companies, especially when sysadmins have coded automation for time consuming frequent tasks that are easy to handle automatically. Before updating our product in production servers could take weeks, now with automation it takes less then a day (multiple countries and data centers). Add to this, that mistakes and problems have actually gone down as areas where humans often make mistakes have been automated. This is saving everybody a lot of work and leading to a better product.

You talk about a strict development and deployment lifecycle, why can't it include automation where possible?

Re:It's not all about the code

[dbIII]

would therefore be classed as a production environment from some angle

That is a bit of a stretch, but if that's how you think of it, fair enough. I've always considered the systems that do the core tasks and those that monitor them as separate entities. You can generally muck about and modify things with monitoring systems with little consequence, especially if it's a new system and something else is already doing most of what it's role is already.
Stuff like nagios, openaudit, puppet etc was written by sysadmin developers anyway. That's the sort of thing I'm thinking of - coding by sysadmins to solve tasks in system administration. While I've done other coding while working as a system administrator it was really part of a debugging process while assisting developers that had insufficient skills for the tasks they were attempting - I think it that comes under the heading of controlled changes anyway since several people got to poke at the code before it went near anything important after the first attempts by the developers. In short, I've thrown things back when they showed blatantly obvious signs of a lack of testing, and I needed some coding skills to be able to explain why. In an ideal environment somebody else is the gatekeeper before untried software is pushed out naked to the internet, but sometimes it's just whoever keeps those systems running.

Re:It's not all about the code

[gbjbaanb]

I think the point was - you don't get the sysadmin to write the script to do things, you get a developer to do it for the sysadmin.

Sysadmins don't have the same rigour of software configuration management as developers, as they don't work with code in the same way, nearly all sysadmin code I've ever seen was practically "for personal use" stored somewhere (a shared folder if they're enlightened, home folder if not).

But anyway, the point is that you get a professional developer to do the development, from the requirements of the sysadmin. If you're lucky enough to work in an environment where management allows this kind of work to happen then you'll appreciate it - I imagine most sysadmins just want to do their admin jobs, and not spend all day knocking out scripts. Better to write up a specification and let someone who knows how to program computers do it.

Re:It's not all about the code

[tibit]

Where I work, even very low impact systems have configuration management, auditing and snapshot/rollback functionality... the rigour you claim is something that can be grasped in a week by any sysadmin out there. No magic to it.

Re:It's not all about the code

[ruir]

The comment is spot on! If someone is not capable at least of scripting their problems, then they are not really sysadmin managers. And to call sysadmin managers to people that administer window boxes like PCs, and do it all by hand is idiotic.

Re:It's not all about the code

[gbjbaanb]

yeah, there are always exceptions to the rule. Good for you though, most places don't have such practices from sysadmins. Hell, some places don't have such practices from developers. Some places don't even have sysadmins!

Sounds like a plan!

[Lumpy]

You doubling my salary for that extra work? I'm tired of the constant scope creep people keep shoveling in without an increase in compensation.

Re:Sounds like a plan!

[PeanutButterBreath]

Don't you see? Knowing how to code will make you more efficient. So, if anything, the amount of, er, whatever it is that you people do, can be increased!

Yes, early adopters will be able to demand a higher salary for any additional skills that they bring to the table. That is, until we can update our job requirements to include these skills (5 years minimum).

Re:Sounds like a plan!

[arielCo]

There are "app" languages like Java/Ruby/PHP that will get you in that situation, and there are languages mostly regarded as tools for automation (Perl/$SHELL/Tcl?). So, there are two ways around it:
* stick to scripting (it'll make your everyday life easier) and refrain from learning these "dangerous" languages, or
* learn them (you'll gain additional insight on common programming pitfalls, making you a faster troubleshooter) and DON'T TELL YOUR BOSS.

Re:Sounds like a plan!

[PPH]

Monkeys get time and a half for overtime. Professionals work on salary.

Re:Sounds like a plan!

[boundary]

Stupid professionals work on salary. Smart ones freelance on an hourly rate.

Re:Sounds like a plan!

[rubycodez]

no, typically the pay will stay the same and any improvements won't be appreciated anyway. pissing up a rope.....

Re:Sounds like a plan!

[Xacid]

Eh, I think this is more like an extension of scripting - at least how I'm looking at it.

I've taken some C++ classes in the past and had the opportunity to finally use it a few years back in a practical sense when working a support-type role. Basically created a menu that'd install a set of printers and network shares based on where the person was located. I didn't have the rights to push anything elegant down through policy or similar (a whole separate issue) and batch scripting only did so much of was needed. I'm sure there were better ways but that was what I knew and it saved me hours upon hours of having to do that process by hand.

So maybe while it may not be "true" coding, I was still able to create a tool that the team could use that basically saved a shit ton of time. Totally worth it.

Re:Sounds like a plan!

[Lumpy]

"And yeah, professionals get paid more than monkeys."

Really? After I calculate the hours worked I'm getting paid a LOT less than the guys that make $22.50 and are hourly.

Because they don't work 75 hours a week and are paid for only 40 of them. As a professional you would have actually learned math and understand this.

Sure so long as coders l2sysadmin

[Sycraft-fu]

Then maybe I wouldn't spend an inordinate amount of time fighting with programs that can't understand how to run as a deprivileged user, that can't properly set up their own environment variables and so on.

So I'll promise to learn to program if they'll learn to sysadmin. Since I already know how to program then they'd better get on it.

Re:Sure so long as coders l2sysadmin

[PeanutButterBreath]

Then maybe I wouldn't spend an inordinate amount of time fighting with programs that can't understand how to run as a deprivileged user, that can't properly set up their own environment variables and so on.

We'll get right on that when everyone learns how to accurately analyse and communicate their requirements.

Re:Sure so long as coders l2sysadmin

[CAIMLAS]

Since when wasn't it a programmer's job to effectively analyse the requirements for what they're writing?

Your nonchalant dismissal of the initial claim is kind of like GM claiming they couldn't have possibly known that an 8-track wasn't a suitable commuter vehicle, and that they couldn't have possibly known that a dog wouldn't be driving the vehicle... ie, if you're writing a program for system that is supposed to perform any particular task (regardless of what that task is!), the BARE FUCKING MINIMUM REQUIREMENT IS THAT IT IS WRITTEN FOR A GIVEN SYSTEM.

Kinda fucking trivial, unless you think that it's reasonable that your program should be randomly terminated by the sysadmin, and for you to receive the blame for it.

Re:Sure so long as coders l2sysadmin

[PeanutButterBreath]

Since when wasn't it a programmer's job to effectively analyse the requirements for what they're writing?

It never was. Requirements analysis is not programming, any more than architecture is carpentry. It is a job, sometimes a profession, unto itself. Granted, like many things, it is often not valued and delegated properly.

If the bare minimum is that software is required for that system, then it should be written to meet that requirement. The sorry fact of the matter is that getting even this "obvious" level of requirement from many people is a non-trivial task. Usually because people only stop complaining about how underpaid they are for plying their rare and special skills to complain about how obvious everything they do should be to all the idiots they are surrounded by.

Corollary: All IT People Should Have to Do It All

[Motard]

I've worked in companies ranging from 5 people to 40,000 (and plenty in between). In the smaller shops I've had to do administration, development, desktop, and customer support. In the larger 'enterprise' shops, I'm constantly amused by the myriad breakdowns in communications caused by folks being incapable of putting themselves in the shoes of their coworkers.

Being a developer made me a better system administrator. Being an admin made me a better developer. Same with operations, support, et. al.

Depends on the Job Definition

[bill_mcgonigle]

Sometimes I play 'sysadmin'. It winds up as the "this doesn't work, make it work" role. Today it was reading the RFC's to figure out how DSN's are supposed to be returned, writing the java.mailx code to make the developers' app do that, and explaning how SMTP works. Other times it's working through a SQL query planner, finding why packets are headed in the wrong direction, re-doing an architecture, etc. It's not possible to do the job well without good CS training, a solid background in coding, a solid background in networking, and just plain blood, sweat, and tears.

Then again, you could study for a test exam in 21 days and call yourself a sysadmin too. It's always the definitions.

Ah, age.

[Anonymous Coward]

At one point, we *had* to code. Tools didn't exist until we made them. Or at least tools that did what *we* wanted didn't exist until we made them.

I blame Windows weenies for the loss of this skill. They cannot function without pre-packaged clicky things. Nitwits.

Re:Ah, age.

[mdhoover]

Oh for some mod points...

Re:Ah, age.

[Xacid]

Hey now, I resemble that remark! ;) /weenie.

Re:Ah, age.

[hairyfish]

At one point, we *had* to code. Tools didn't exist until we made them. Or at least tools that did what *we* wanted didn't exist until we made them.

I blame Windows weenies for the loss of this skill. They cannot function without pre-packaged clicky things. Nitwits.

Fucking weenies and their electricity, I bet those nitwits have no idea how to create a flame from their bare hands. Bloody progress, how dare they make life easier for people.

If all I did was write Perl

[Seumas]

I'm fairly certain that if all I did was write Perl, I'd go insane

And that's the point when you officially become a perl hacker.

Re:OO

[OneAhead]

It's probably a lot healthier to stick with proper OO languages.

That statement makes you not crazy enough to be a code monkey.

ANYBODY Would Go Insane...

[Jane Q. Public]

... if all they did was write Perl.

Where do you think Slashdot came from?

Coders Should Know How To Sysadmin

[MisterP]

I don't think anyone could argue that having skills in both areas isn't a good thing. I've been a sysadmin for 20 years and i've had to do basic development over the years (apache modules, ldap-ifying applications, etc). When it comes to troubleshooting complex problems as a sysadmin and the whole team is whiteboarding, you can tell pretty quickly who understands how systems work below the user interface. This is often only learned through writing code.

The opposite is true for coders too. With a few execeptions, the most competent developers I've worked with have had sysadmin duties at some point in their career. Not that long ago, I had to sit down with a Sr. (Java) developer and explain load balancers and TCP session state.

Perl is beautiful

[Frank T. Lofaro Jr.]

I'm fairly certain that if all I did was write Perl, I'd go insane.'"

but Perl is a beautiful language!

Re:Perl is beautiful

[grouchomarxist]

>Perl is a beautiful language!

Yes, for those with twisted minds.

Re:Two different worlds

[PlusFiveTroll]

Yes, it is the Sysadmin's job to figure out the code is broke, knowing how to program can tell you when there is a 'program' problem verses a hardware or network problem. You can then give better test cases and problem reports to the developers which allow the problem to be fixed faster.

Really this isn't any different then any other job out there. Lets say your a truck driver and your truck isn't working right. If you know very little about the internal workings of engines you'd tell the machine shop that 'there is a problem' or 'it's not running right'. If you had a good idea of the operation and components of the engine and drivetrain it is very likely you could give a better bug report 'it's not shifting at 4k rpm, possibly due to loss of vacuum'. Little details like that can save hours of 'fucking with it' to figure out what the issue is.

Really this article is kind of a 'duh', how surprising is it that having more knowledge of how a system works makes you better at working on that system.

It's the mindset that matters

[dbIII]

A lot of the time you don't want developers administrating other people's gear since you get shit like unnecessary reboots of servers during peak working hours leaving 300 people with nothing to do apart from read newspapers (one memorable example). It's not skills that separate a good developers from a good sysadmin but instead a consideration of inter-related systems and caring about consequences of actions.

Re:It's the mindset that matters

[CAIMLAS]

So in other words, conservative people (not necessarily political, but probably financial) tend to be better sysadmins.

(Yes, that's been my observation, too. On the flip side, 'rockstar developers' almost invariably seem to be of the red diaper variety.)

Maybe

[asmkm22]

I don't know about full-blown compiled coding or anything, but sysadmins definitely should have a grasp of a scripting language relevant to their environment (such as vbscript for windows). There are too many times you get requests for stuff that inexplicably have no official support for, such setting up a default Outlook signature for all users that pulls information from their login profile. The sysadmin who can say "give me 30 minutes minutes and I'll have it ready for testing" will look a lot better than the sysadmin who says "we could buy an $800 program that will allow us to do that, but we'll still have to test it out."

Re:Maybe

[Anonymous Coward]

The sysadmin who can say "give me 30 minutes minutes and I'll have it ready for testing" will look a lot better than the sysadmin who says "we could buy an $800 program that will allow us to do that, but we'll still have to test it out."

The hidden cost is you have to support it forever. Your organization is not frozen in time, it will change or grow, and you will have to adapt to changing requirements. Sooner or later you will have to adopt a formal testing process, and stay on top of the documentation because other people will use it. All to say there is overhead associated with this stuff, that's all.

The same is true of commercial software for sure, there is overhead. The point I would like to make is of the in-house, free, and commercial solutions that meet your requirements, they ALL have an maintenance overhead cost in time and effort (and/or money), and you should weight that against the price you pay.

One expensive commercial product with ultra-low overhead might save you enough time to employ many, many, many other free "but rough around the edges", or custom "because nothing else does it" solutions.

Re:Maybe

[petit_robert]

"One expensive commercial product with ultra-low overhead "

Contradiction in terms.

This just in...

[epp_b]

Experience in one field can be an asset in another due to inherent interdependent qualities! Wow!

But as many others here have pointed out, the last thing businesses need is another job to dump on the IT department while continuing to tighten their budget (because, hey, computers are just computers, if you know about them, you know everything about them, right?)

Re:This just in...

[blade8086]

More like:

Knowing how to do your actual job makes you better at doing your actual job!

or alternatively:

Pretending to be something that you are not is not the same as actually being that person!

Re:OO

[scubamage]

Psh, real men use goto statements.

And they should be paid more than programmers...

[Anonymous Coward]

Welding all this developed-in-silos software together so it actually works should have some compensations.

And the oposite.

[stanlyb]

Developers should know how to do the admin job.

I disagree..

[xushi]

However I do believe they should know how to script..

Just my HO. Coding can be a bit too excessive for a sysadmin - otherwise he'd shift more towards programming. Scripting can be more than enough with bash/perl/php/python.

Welcome

[HogGeek]

Captain Obvious...

Agreed

[roc97007]

> I'm fairly certain that if all I did was write Perl, I'd go insane.

Especially Perl.

100%

[Niobe]

If you can't code, you're probably a Windows admin.

I've been preaching this to my help desk

[Fosterocalypse]

Often my help desk guys ask what they can do to get off the phone and do their job better and I say learn from something I over looked for a long time...learn VB and Perl. You can do so many awesome things with minimal code that can help take you systems to the next level. They should learn simple bat & vb scripts and then move to Perl and on from there. I think the biggest problem is that they over look the tools you can create with them. Like logon scripts and scripts to make mapping a printer as simple as double clicking a file.

Re:Romney to win tonight, win the election

[Vladius]

The cowards are out in force today.

Those who can and those who can't.

[JakFrost]

In my years as a Windows admin I found it interesting to find out which other admins could or could not write scripts and then classify them by the level of their abilities:

- The GUI clicker Guy
- The Command Line ipconfig.exe Guy
- The Google for a Script Guy
- The Cut-and-Paste Each Line Separately Guy
- The Excel Drag-and-Fill Guy
- The Search and Replace In a Script Guy
- The Batch Script with No "@echo off" Guy
+ The For loop Guy
+ The Reg.exe Guy
+ The PsExec Guy
+ The 2>&1 Redirect Guy
+ The Pushd/Popd Guy
+ The Setlocal Guy
+ The Rundll32.exe Guy
+ The Findstr.exe RegEx Guy
* The GnuWin32 Sed/Grep/Tee Guy
* The Cygwin Guy
* The Perl Guy
* The VBScript Retarted Syntax Language Guy
* The JScript Cool Web Language Guy
* The Script Signing Certificate Guy
@ The PowerShell Guy
@ The PowerShell & Quest PowerGUI Guy
@ The PowerShell & PowerGUI & 3rd Party Cmdlets Guy
@ The PowerShell & [.NET Framework] Accessing Guy
@ The PowerShell & .NET FrameWork, Activator, Marshall, Reflection, COM+, Jobs, Runspaces Guy
$ The Visual Studio C, C++, .NET, API, SDK, MSDN, Compiler, Remote Debugger, Memory Dump Analyzer Guy
$ Kernel Developer

Re:Still missing the point Re:It's not all about

[boundary]

You're quite correct - scripting *done properly* is the key. The hobby scripting mentality is something that we have historically had some challenges with, and it has ended in tears. And blood. Lots of blood. Dev and test environments have been made available in much the way you describe, but it has been a case of you can lead a horse to water but you can't make it drink. In my experience some people, especially those who choose to work in the operations space rather than project side, seem to have problems working within a process that involves other people. To mitigate risk they need to be part of a bigger process that involves independent checking and some level of traceability, but they don't necessarily want to be. How else can they cultivate their image as a worker of miracles? A proper mindset is just as important at proper procedures and environments if you want decent scripts.

Re:Two different worlds

[pla]

Should a developer need to know the ins and outs of the Data Center? Should they know Unix/Linux/Windows/App Servers and everything else that is running?

If you want that code to run in a massively-parallel hot-swappable environment, then you bet your ass you want your devs to understand everything running in that environment.

If, however, you want a standalone app to run on a few desktops, then no, your dev doesn't need to understand data center architecture; but in that environment, neither does your admin / ops guy.


Realistically, I give this FP a 9/10 for managing to troll a group of people who largely overlap. Most programmers can do 95% of sysadmin work; and most sysadmins can code reasonably well. People go into one or the other not for their capacity to do the other job, but either as a preference or even blind luck.

Personally, I would rather deal with pure code than dust, but I've damned well pulled cable while crawling through 70 years of dust inside a ceiling at 120F. On the flip side, I know guys who just love the shininess and the contented hum of a new rack, and consider scripting one of the "dirty but necessary" aspects of their job.


But can we all just settle our differences, come together in peace, and agree that the rest of the world sucks and needs to learn to both how to 1) use a vacuum cleaner, 2) plug in their own damned monitors (and perhaps the occasional printer), and 3) write simple Excel macros?

In other News...

[i58]

Mechanics should have mechanical skills.

Re:total malarkey

[Anonymous Coward]

If you are a sys admin and can't find any boring/repetitive tasks for which you can write scripts, you're not a real sys admin.

Just have the Unix team do it

[shuz]

I've found that what can happen in the large corporate world is that you have Developer teams, Production run teams, various IT infrastructure teams, and the Systems Engineers. Networking gets blamed for every outage because well, they are the common thread that all the bits run over. Storage gets stressed out because it is the last thing anyone thinks about until they need it, and the Systems engineers well the Developers want to play all roles unless they don't want to. Production run sometimes lacks the deeper skills of either programming or IT infrastructure.

Enter Unix Engineers. We are expected to have general knowledge of all IT infrastructure, which we do, we program and script extensively, because our automation depends on it, and we have extensive production application run experience do to managing all the different back end services that everyone depends on. mail, dns, ftp, sftp, web server engines, monitoring, etc. Yes a good sys admin must know how to at least read code and ought to be able to code/script in at least one language even if that is Dos Batch. The problem in the end though is as others point out. The more you know the more production run teams may lean on you to solve their problems. "Just ask the Unix team", Not because it is their responsibility, but because they can solve the problem quickly as they have the widest berth of knowledge.

Coding as a Sys Admin is crucial. Just know when to say I can't(or won't), for your sanity.

Re:Just have the Unix team do it

[CAIMLAS]

Pretty much. "They know UNIX. They can do this."

Re:Just have the Unix team do it

[murphyd311]

"Just ask the Unix team", Not because it is their responsibility, but because they can solve the problem quickly as they have the widest berth of knowledge.

Coding as a Sys Admin is crucial. Just know when to say I can't(or won't), for your sanity.

This, a thousand times! I'm a UNIX Engineer for a large corporation, and I find that the UNIX team gets pulled into every problem, when the reality is, it is seldom an OS issue. It's because we have to have that depth of knowledge, just to set up a single box, never mind a fully automated HA environment. I find that we also have the job of calling out bullshitters who say "it's not our problem (I'm looking at you Network Team) or "no I didn't change anything!" (mister chmod -R 777).

On topic: Scripting is key to solving a lot of "run this on 300 AIX, 50 RHEL, 20 HP-UX, and 5 Solaris boxes" and staying sane doing it. Our problem is we have a ton of legacy scripts that do a lot of day-to-day stuff, which was great for 300 total boxes. On an enterprise level with thousands of machines, not so much. Now we're trying to replace a mess of scripts with enterprise level solutions, but figuring out how we do it now, and how we want to do it with some real software is more of a challenge than we thought. We're in a bit of a mess because someone automated everything with scripts, and it is not scaling.

Re:Just have the Unix team do it

[shuz]

lookup cfengine, it takes a while to get setup right but it scales nicely once it is. If you are a hybrid shop look at what scom, Microsoft system center can do for you. Ya it is a microsoft product but it was designed to be an industry disruptive do all things tool. Might not be so bad if you were building a solution from scratch.

Re:Corollary: All IT People Should Have to Do It A

[DigiShaman]

I've been at fortune 500s in which I was tasked to provide some system administration/helpdesk for a branch office of about 50 employees. It wasn't every day, just a few times a month I would be onsite. The hilarity ensues when I have to call the main office to speak with the IT dept. In one recent example, I had some trouble with Exchange connectivity inside the office. He tested the cluster of servers and ran mail flow diagnostics. Looked fine. He said it was the network guys job. So I called the network admin, he said traffic and routing looked good on his end, it must be an Exchange problem. Back and forth we went. The problem was a combinations of undocumented server and DNS changes. Nothing got resolved until I got at least five people in conference call. It was only then that people started to fess up as to what was going on and when.

In short, it was a breakdown in communication. I only got things working because I took it upon myself to wear the managers hat and get shit done. Way above my pay grade, and I'm sure I get paid half as much as those fucking monkeys behind the phone!

Ya, I'm bitter.

Please do not

[Anonymous Coward]

Please no, I enjoy both programming and systems work and am equally good at both. I enjoy the lack of competition presented by those that can only do one or the other.

Re:Corollary: All IT People Should Have to Do It A

[robot5x]

you're bitter because you solved a problem that was 'above your pay grade'??

Wow, maybe this is what's wrong with the IT industry. Yeah, big corps are complicated and bureaucratic etc etc etc, but if more people actually WANTED to make an effort to do stuff, and worked outside their own defined sphere of influence just to y'know, actually get shit done, then maybe things wouldn't be so bad?

All the discussion here leads me to the conclusion that the key to happiness is as simple as continuing to learn and develop your skills and to occasionally stick your neck out and do whatever it takes to solve the problem.

Ridiculous!

[dutchwhizzman]

Your proposal is ridiculous! Next, you'll be saying politicians should know how to run a country!

o.O

[lennier1]

In this day and age it's kinda amazing to still find admins who can't even script enough to take some of those annoying petty tasks off their hands.

Code is now a Buzzword

[Cinnaman]

Using code as a buzzword seems to have caught in within the last year or so, I now see it on a regular basis. And I always think "what programming language are they referring to?", apparently html is also included.

Re:Code is now a Buzzword

[Cinnaman]

I definitely think it's a case of people think it's cool to use code as a verb.
Nerds trying to break away the externally imposed stereotype perhaps and make programming trendy.

I remembered that fads and being cool tend to be confined to teenage and maybe early 20s years so I'm guessing that demographic has started this and is on a mission to create a new definition for "code".

Re:Corollary: All IT People Should Have to Do It A

[SomeStupidNickName12]

Buddy, this how you get promoted - doing things above your pay grade and making sure management knows about it. You then become the go to guy for management to solve problems. Then during salary/bonus/promotion time if they don't reward you drop them and leave.

What?!?

[QuasiRob]

It had never occurred to me that there might be sys admins out there who *can't* program to some degree!

Re:I thought sysadmin = failed developer

[QuasiRob]

Nah, I programmed for a while, decided I didn't want to do it full time and had more fun messing around with sys admin type stuff.

Re:Corollary: All IT People Should Have to Do It A

[DigiShaman]

Been working IT for almost 15 years now. The respect is the same then as it is now. You wade through shit to solve a problem, only later to get bitched at for not finding it sooner when in fact you were originally trying to make a case of how pro-active you've been. Oh, and because I've been wading in shit, I smell bad.

There is no respect in IT and the pay sucks. I'm looking to find another career patch that isn't already tainted with disdain. Fuck this, life is too sort. I'm tired of falling on the sword and not getting any recognition for it...for 15 years.

Sure, and all drivers should be mechanics...

[Tomsk70]

....would you accept that statement from your mechanic when their fix doesn't work?

Respectfully disagree

[Larry_Dillon]

Knowing a little programming is good (for scripting and "glue" programs), but I've seen too many CS majors that try to reinvent the wheel (because they like to program) rather than getting to know the built-in tools that are fairly standard and vetted.

e.g., I had a CS major ask if there was a Linux version of WinDiff because he was thinking of writing a Linux version.

sync

[Onymous Coward]

Is there a code v. comment synchronization system yet? I'm imagining something where a comment is associated with nearby code (delineated by whitespace or scope) and includes a checksum of that code.

foreach my $atom (@atoms)
{
    my $radius;
    # calculate thermal radii as inverse of temperature, scaled to allowed range (CC 58e2 c902)
    $radius = $atom->{'temp'} / ($max_temp * ($radius_max - $radius_min) + $radius_min);
    print "$atom->{'x'},$atom->{'y'},$atom->{'z'} $radius\n";
}

And your editor can highlight it for you if it falls out of sync.

Re:sync

[slapout]

That might be interesting if the compiler wouldn't let the code compile until you did something.

Different strokes for different folks

[s.petry]

If all you plan to do is camp where you are, and you don't see a need to code.. well good for you. I'd still recommend knowing at least the basics since it can, and more often than not does, improve your ability to perform your job. I use scripts all the time, but then again I'm not an average system admin. Let me give you a few examples.

I wrote a script for resetting LDAP passwords for Unix, since most Unix admins don't understand LDAP. It had to meet NISPOM compliance, randomly generate secure passwords, inventory LDAP to ensure the user exists, poll People Soft to ensure the employee was valid, etc.. It was a cool little project that took me about a week. However when it was done our level 1 support could reset passwords for users, saving time in the long run for everyone else (including myself). They still use the same code to reset passwords, and I wrote the utility 7 years ago.

That little project, since I wrote it modular, was expanded to full user and group management. Need to add a user, type in a badge number and answer a couple basic questions. Again, level 1 could add users and groups to Unix LDAP without issue. 6+ years later, they still use those too.

Analysis applications often have countless requirements for Engineers to run jobs. I wrote wrappers for everything from MSC software to PAM CRASH. wrappers were simple to modify for new versions and again those are being used some 10 years after I wrote them.

Need to validate users from disparate systems, write a script. Need to check for compiler functionality, write some basic C code. If you can't get a simple program to compile, then neither will your users. In my opinion, a sign of the best administrators are that they test and think things through.

Coding can make your job much easier, as well as those around you. It's all about what you can innovate, then how you innovate a solution. I use scripting much more than C, but sometimes more complex problems come with C. Testing things like memory with an active OS are simple in C, why reboot and wait for a vendor disk?

Like I started with, it may be something you don't have to do now. However, to make yourself more valuable I'd highly recommend it.

all sys admins can program

[syleishere]

I have never met one that couldn't program, I have actually found the opposite, the sys admins I have worked with could program better in more languages, and the coders could only code better at maybe 1 or 2. Sys admins need to know C, unix tools, windows tools, perl, php, shell, and a multitude of operating systems and different packaging systems that go along with that, they are generally more intelligent than the most intelligent programmer, and generally on spare time kernel hackers, so not sure what types you hang out with, but please don't insult the real admins :)