System Admins Should Know How To Code

snydeq writes "You don't need to be a programmer, but you'll solve harder problems faster if you can write your own code, writes Paul Venezia. 'The fact is, while we may know several programming languages to varying degrees, most IT ninjas aren't developers, per se. I've put in weeks and months of work on various large coding projects, but that's certainly not how I spend most of my time. Frankly, I don't think I could just write code day in and day out, but when I need to develop a tool to deal with a random problem, I dive right in. ... It's not a vocation, and it's not a clear focus of the job, but it's a substantial weapon when tackling many problems. I'm fairly certain that if all I did was write Perl, I'd go insane.'"

Comments

Very true, for many reasons.

[uCallHimDrJ0NES]

People who've never coded tend to have many "magic boxes" in their thinking about systems. I find it hard to fully trust an administrator who can't at least parse through other people's code.

Re:Very true, for many reasons.

[bolthole]

Depends what they're administering.
There are plenty of systems that are more closed, and the sysadmin should be spending their time living in the pre-provided frameworks, rather than coding their own.

Many times, it's a matter of learning what is already available for the system, rather than coding your own, lesser quality replacement.

Re:Very true, for many reasons.

[pinfall]

I don't code a lick after 15 years of sysadmin. I do solve incredibly interesting problems with hardware and that ability alone provides a serious interface to all those 'needs software' perspectives. All of my best friends are coders, we just solve problems differently.

Re:Very true, for many reasons.

[hairyfeet]

The problem is the way corps shit all over IT they'll just go "Hey, one more job we can dump on them without raising their pay!" and that will be that.

Lets face it folks, we are gonna end up with critical shortages of IT and infrastructure workers because between the offshoring, the H1-Bs, and the PHBs treating IT as this money pit that doesn't give them any profits? IT has been shat upon for the good part of the last decade.

I know myself and most of the old guard guys I knew ended up getting out of corporate IT for just this reason, piling more and more work upon us while expecting everything to be done with less help and a shrinking budget...now you want to add coding to the requirements? You gonna add a pay raise and pay for the classes? Yeah, thought not.

Re:Very true, for many reasons.

[countach74]

I think you missed the point. The coding helps save you time in the long run. Not the other way around.

Re:Very true, for many reasons.

[Mr2cents]

You don't or you can't? Because even when I'm not coding, just in day-to-day computer use, I often write scripts to batch-process or automate various things. I have found this highly beneficial. I don't want to break you down, I just want to know your reasoning. Personally, I can highly recommend learning scripting, be it bash/awk/php (yes, I wrote php scripts)/powershell/whatever, I'm sure you'll benefit from it.

Re:Very true, for many reasons.

[fsck1nhippies]

2Cents, you are absolutely right. Even in windows systems, a basic understanding of what can be done with code can stop 5 people from running around to a couple hundred machines each.

Should the sysadmin be a programmer? Not in the conventional sense, but they should be able to programmatically attack the problems placed before them before they just brute force their way through them.

Re:Very true, for many reasons.

[karnal]

I have coded in the past using Perl to give us some reporting ability from some non network-connected (think no email/notification ability) phone switches. They had relay outputs that would trigger on an alarm; so we used a box to generate a trap that would fire up my perl script to send out emails to whomever was in the config. I don't know a lot - I mostly use bits and pieces of code found around the web and customized to purpose. I initially thought I wanted to code for a living, but found in my first job that politics trump the technical, and boy were the politics thick. I do more security/administration nowadays, but still draw on the basic coding ability to create automated notifications for anything non-standardized. Really what I'm trying to say here is sometimes the basic experience of coding as well as a need to fulfill a solution that doesn't have a canned solution readily available can push you to do some interesting things.

Re:Very true, for many reasons.

[scubamage]

Exactly - it comes down to a matter of cost very often. If you can spend a half hour throwing together a script to make a configuration change across, say, 2000 devices (or in our case, several million), it is far cheaper than trying to find a vendor solution, or having folks go out and do the work themselves. The vendor will often have maintenance fees and high initial costs for a tool that "sort of" does what you want. You can have people go out and pound pavement, but if you have 2000 devices and send out 50 people, and it takes each person 20 minutes to do the work on the device plus 30 minutes trave time, times 17$ an hour... well, that adds up fast too. Not to mention the opportunity cost and backlog of tickets that that would generate.

Re:Very true, for many reasons.

[fsck1nhippies]

I see a lot of admins in very large companies throw labor at a problem as their first course of action. It is typically a face palm moment for me as I often see the problems as fixable in minutes. I believe that all sysadmins should be able to program, but think that making a programmer a sysadmin is generally a bad idea.

Re:Very true, for many reasons.

[hawguy]

To be honest, most of my managers have been very nervous about my suggestions to even use tools which I'm knowledgable in because they don't understand the concepts. Ad the bigger the company the more likely they are to want a vendor solution rather than use internal resources.

As a manager, sometimes I see the tool to automate and script changes to be a risk of taking down the network faster than anyone can do anything about it.

Like the time a senior network admin accidentally took down our network by setting the IP address of all of our network switches to the same set of IP address. He tested the script and it worked perfectly on one network closet - then he proceeded to apply the remaining set of IP addresses the the remaining hundred network switches in our organization.

He knew as soon as the first alarm went off what had happened, but by then it was too late to stop it. Fortunately, he had the good sense to not write the config to memory, so recovery just walking to each network closet to power cycle the switches -- much faster than if we had to walk around with a console cable to back out the changes on each switch

That said, he still uses the same tool to push out changes, but he tests his scripts on 2 IDF's before pushing out changes and has the junior network admin double check his work.

But I can see why a manager would be cautious about a new and untested automation tool.

Re:Very true, for many reasons.

[CAIMLAS]

Experience has told me something about automated batching of config changes which your 'senior' admin should've known as well. This is crucial:

DEPLOY IN (SEQUENTIAL) PHASES. I don't care if you tested it in one situation or 5: if you've got 50 different situations, you need to 'test' it in each of those unless you know those different situations are in fact the same situation. This means a longer deployment, but you need to test ad verify everything, especially something infrastructural. It will take longer, but it's inerrantly necessary.

Not only does it reduce the outage potential, but it allows you to more quickly back out of it. Sure, it's not as lazy as deploying everywhere at once after a couple quick tests, but it's more foolproof than having half a dozen people look at your changes. Yes, having a second set of eyes is important, but its mostly pointless if they don't fully understand what they're looking - that's why they're the junior, after all.

Re:Very true, for many reasons.

[LongearedBat]

make a configuration change across, say, 2000 devices (or in our case, several million)

Running a bot net, are ya?

Coders should know how to admin

[Anonymous Coward]

...and support for that matter.

Sure

[xaoslaad]

And if you're going to say I'm a programmer then pay me like one. I don't think most sysadmins get paid as much as programmers, and I don't think most companies want to pay sysadmins as much as developers.

Also, developers trying to write tools for sysadmins usually suck at it, unless they've been a sysadmin at some time in the past. I have used a few products lately which are trying to solve all our sysadmin problems, and the one that doesn't suck comes from a dev who is a former sysadmin. And when I talk to him and make suggestions he sees exactly where I'm coming from.

Developers just want to solve use cases that fit neat little scenarios without any corner cases, and it shows when their tool is so inflexible as to be useless.

Not only admins

[Mr2cents]

In general, everybody dealing with computers can benefit from a bit of programming knowledge, not only admins. The rule of thumb is: if you're doing a repetitive, braindead job, you're doing it wrong. Computers are built to do exactly that. A small script can automate a lot of work for you, if you have that skill it can help you tremendously.

Re:Not only admins

[Mr2cents]

PS: YOu don't really need to know how to program, if you can just identify tasks that are braindead and repetitive, that's already a plus. Then you can go and talk to someone who does know how to program, explain the problem, and this person might come up with a simple solution for you. It all boils down to this.
Unfortunately, many people are not trained to identify automatable jobs.

Perl

[slapout]

So, does Perl drive you crazy or do you have to be crazy to program in Perl?

Is this what any programmer does?

[PeanutButterBreath]

I'm fairly certain that if all I did was write Perl, I'd go insane.

As a programmer, to me this is like someone equating author and typist. Code is just a medium. Figuring out what to do with it, and how, is the fun part.

Sounds like a plan!

[Lumpy]

You doubling my salary for that extra work? I'm tired of the constant scope creep people keep shoveling in without an increase in compensation.

Corollary: All IT People Should Have to Do It All

[Motard]

I've worked in companies ranging from 5 people to 40,000 (and plenty in between). In the smaller shops I've had to do administration, development, desktop, and customer support. In the larger 'enterprise' shops, I'm constantly amused by the myriad breakdowns in communications caused by folks being incapable of putting themselves in the shoes of their coworkers.

Being a developer made me a better system administrator. Being an admin made me a better developer. Same with operations, support, et. al.

Depends on the Job Definition

[bill_mcgonigle]

Sometimes I play 'sysadmin'. It winds up as the "this doesn't work, make it work" role. Today it was reading the RFC's to figure out how DSN's are supposed to be returned, writing the java.mailx code to make the developers' app do that, and explaning how SMTP works. Other times it's working through a SQL query planner, finding why packets are headed in the wrong direction, re-doing an architecture, etc. It's not possible to do the job well without good CS training, a solid background in coding, a solid background in networking, and just plain blood, sweat, and tears.

Then again, you could study for a test exam in 21 days and call yourself a sysadmin too. It's always the definitions.

Ah, age.

[Anonymous Coward]

At one point, we *had* to code. Tools didn't exist until we made them. Or at least tools that did what *we* wanted didn't exist until we made them.

I blame Windows weenies for the loss of this skill. They cannot function without pre-packaged clicky things. Nitwits.

It's the mindset that matters

[dbIII]

A lot of the time you don't want developers administrating other people's gear since you get shit like unnecessary reboots of servers during peak working hours leaving 300 people with nothing to do apart from read newspapers (one memorable example). It's not skills that separate a good developers from a good sysadmin but instead a consideration of inter-related systems and caring about consequences of actions.

Re:It's not all about the code

[aXis100]

Wow..... just wow. As an IT Manager you have failed to grasp the concept of this article, and that is a worry.

They are not talking about sysadmins writing production code - they are talking about using one of a variety of scripting languages to solve sysadmin problems - eg repetitive tasks like backups, deployment scripts etc, maybe even some html status monitoring screens or a cactus plugin.

Re:It's not all about the code

[PPH]

So, you are willing to accept an admin task performed manually, mistake creepage and all. With 500 workstations, there's no guarantee that the last one will be configured the same as the first. After the first few dozen, fat finger mistakes will undoubtedly creep in. By number 400, your admin won't be seeing straight anymore.

When we say "admins need to understand coding" this includes all of the associated issues of testing and configuration control. Perhaps not to the same level of detail as the code for the product. But in some cases, it can come pretty close.

I'd much rather have my admins script everything. And save the script. So when we come running in and ask, "What the **** did you do??!!", they have the exact steps in hand.

Just have the Unix team do it

[shuz]

I've found that what can happen in the large corporate world is that you have Developer teams, Production run teams, various IT infrastructure teams, and the Systems Engineers. Networking gets blamed for every outage because well, they are the common thread that all the bits run over. Storage gets stressed out because it is the last thing anyone thinks about until they need it, and the Systems engineers well the Developers want to play all roles unless they don't want to. Production run sometimes lacks the deeper skills of either programming or IT infrastructure.

Enter Unix Engineers. We are expected to have general knowledge of all IT infrastructure, which we do, we program and script extensively, because our automation depends on it, and we have extensive production application run experience do to managing all the different back end services that everyone depends on. mail, dns, ftp, sftp, web server engines, monitoring, etc. Yes a good sys admin must know how to at least read code and ought to be able to code/script in at least one language even if that is Dos Batch. The problem in the end though is as others point out. The more you know the more production run teams may lean on you to solve their problems. "Just ask the Unix team", Not because it is their responsibility, but because they can solve the problem quickly as they have the widest berth of knowledge.

Coding as a Sys Admin is crucial. Just know when to say I can't(or won't), for your sanity.