Managing Young Sys Admins At Oregon State Open Source Lab

mstansberry writes "Lance Albertson, architect and systems administrator at the Oregon State University Open Source Lab, uses a sys admin staff of 18-21-year-old undergrads to manage servers for some high-profile, open-source projects (Linux Master Kernel, Linux Foundation, Apache Software Foundation, and Drupal to name a few). In this Q&A, Albertson talks about the challenges of using young sys admins and the lab's plans to move from Cfengine to Puppet for systems management."

Comments

Especially if they are training developers

[xzvf]

Most universities don't teach good system management. The CS departments are training developers and programmers. Since good SA's like stability and good developers like chaos the two normally don't mix. Does OSU have a SA degree program?

Re:Especially if they are training developers

[fuzzyfuzzyfungus]

The nasty trick is that, while not touching systems is very, very bad practice, being the first guy to touch something that hasn't been touched in a while is not a pleasant thing.

In an ideal world, all systems get regular attention and everything hums along smoothly. In a less ideal world, people are distracted from what is working by what isn't working, and their knowledge gradually atrophies, until they no longer dare touch what is working for fear of making it join what isn't working. This is a thoroughly pathological situation; but, if you are stuck in it, Just Not Touching and hoping for the best is quite possibly more logical than biting the bullet and taking one for the team.

Re:Especially if they are training developers

[russotto]

An excellent developer has a work ethic that would mesh nicely with an excellent sysadmin.

It's not the work ethic. It's temperment, the whole attitude towards things. As a developer, I'm very uncomfortable working on production systems, because there's a lot of walking-on-eggshells. If something goes wrong in the dev or QA environments, I can do a lot of "OK, try this... try this... try this", and if one of those things brings the environment down in flames, such is life. When I find the right combination it can be restored to the original broken state, then I can re-apply the thing I think worked, and if that works, I can then hand that off to ops to put on the production system with some confidence that it will work.

If a production environment has some problem which has to be corrected "live", though, that's a very BAD way to try to fix it, for obvious reasons. Instead, a lot more passive analysis has to be done before trying anything, and more important than "will this fix the problem" is "will this bring the system down (worse than it is) or worse, destroy customer data". And since when a developer is called in to help with production there's probably a major problem, the developer usually has to deal with the ops folks and management breathing down his neck as well.

Re:Especially if they are training developers

[BitZtream]

No, they weren't excellent developers.

They may have been great code monkeys, but those are literally a dime a dozen if you don't mind importing from India.

A GOOD developer is also a GOOD sysadmin.

I've been doing both for ~15 years, in that time, I've met 2 good developers. The rest have are monkeys who write code, many of which who write well. Maybe in the next 15 to 30 years I'll meet a few more. With everything going networked now days, you have to be a good sysadmin to understand how to write software that admins can use.

You don't write an apache webserver type application without being an admin, you just don't know what you are trying to hit. This isn't unique to developers writing for admins, its true for all development. If you don't know how to do and have experience doing what the software is supposed to be used for then its practically impossible for you to write software that doesn't suck. This is why there isn't a point of sale system on the planet that isn't asstastic. You'll be hard pressed to find a developer now days who actually had a job when they bothered to pay attention enough to know what good POS software needs to do.

Interestingly enough, I've never met a CS student or recent grad who was even a good code monkey. After several bad experiences our company has developed a 'no CS grad' policy for developers. We'll take you after you have 5 years or more experience, but with less than that all you are is an arrogant asshole who thinks he deserves to get paid ridiculous amounts of money and you still spend a few years breaking them down into something useful. (Read: removing the arrogance of youth and learning that there is far more software AND administration than just writing code or installing Linux)

Re:Especially if they are training developers

[mysidia]

The ideal combination would be for them to administer live systems and test systems at the same time, e.g. have access to both.

And utilize change control.

Use test systems to experiment and learn. Make proposals to accomplish tasks/maintenance that need to be done to live systems (such as upgrades).

On lab systems, they test/experiment with their proposed administration procedures/configuration changes. Before using them in the maintenance of live systems, they document what they plan to do step by step, and two other admins, a "partner", and a senior admin go over the procedure with them, question them about any apparent gaps, lookup any missing information needed, and take a copy of the procedure.

Then at the planned time, the young admin will run the procedure exactly as documented.

If they need help, or something (bad) should happen to them in the middle (e.g. they tripped over something, broke a leg, or broke their keyboard in the middle of maintenance), such that they can't complete the maintenance task, or get stuck, the other two admins both agree to be available to help and pick up if necessary.

Lesson 1

[tverbeek]

The main thing that people that age need to learn (both professionally and personally) is that Their Actions Have Consequences.

Amazing.

[CannonballHead]

Young people with their heads on straight. Definitely newsworthy.

I know the whole "you young'un, you can't manage a server to save your life!" feeling and all that, but really... is managing a server, even an important one, really that hard - when you have someone to go to when you have questions? A lot of lab administration seems to be finding problems before they become a real problem, which is time consuming.

You may as well have a story about dental work done by *gasp* dental students and, lo and behold, they are actually doing a good job! Shocking. To think that young people could actually learn something. :)

OTOH, it's interesting to read about the difficulties he brings up. They're pretty ... boring, IMO.

It generally takes around six months for a student to feel comfortable with our environment.

Like most jobs?

Another challenge is the short turnaround with students, as we usually only have them for two to three years before they graduate. This creates a constant issue to ensure our documentation and training is honed.

Two to three years, that's not too short, is it? And it's interesting that it's an "issue" to him to keep their documentation good/honed. I hope the graduates are learning that documentation is a BIG ISSUE in real jobs, for exactly that purpose: if something happens to you, the business can't just stop for 3 months while someone else tries to figure out what you did :)

Nope

[autocracy]

The members of the CS department at my college actually petitioned to have me take over as their lab admin. The incumbent staff admin was notorious for breaking things and making it a chore to use the systems. Despite the complaints against him and requests specifically to hire me on, the department chair kept the incumbent.

I found it all very amusing, especially since I'm not a CS student. I'm just well-known enough to the group. I'm also greatly amused by how often I get asked for help when I'm around there, specifically one case where a student was in a 390-something class. I replied, "We really don't know each other at all, and I'm not a CS student. What made you think I am a good person to ask?" He said he'd just seen me help with enough other people's problems... and so I gave him a hand too.

Long-windedness aside, my university only uses students to provide, "Cean the viruses off your personal computer," services.

Re:Nope

[Sir_Lewk]

And you wonder why people don't like your type...

Sysadmins have good growth opportunity

[omgwtfroflbbqwasd]

While entry-level programmers may make a slightly higher salary than a similar systems administrator, over time there's a lot more upward opportunity for the sysadmin. Systems Engineering and Systems Architecture - being the guy that ties the network, the server, and the apps together, is a very in-demand skill and is something programmers will never have the opportunity to become. Programmers only make the big bucks when they have other specialized knowledge that's specific to the apps they are developing, i.e. finance, GIS, physics, etc..

I'm personally glad I made the decision 12 years ago to move into systems after earning my Comp. Sci. degree. I went from web app development for an ISP to Linux/Solaris/HPUX sysadmin, to Systems Architecure, to Info Security.

Re:single point of failure?

[gchaix]

I work for the OSU OSL.

Actually, we're more than a mirror. While mirroring is a major part of the services we provide, we also provide hosting for many projects' core infrastructure - Apache, Linux Foundation, Drupal, kernel.org, etc. Google is a major supporter of the OSL because we provide a place for projects whose needs have outgrown the more "off-the-shelf" structured hosting provided by Google Code or Sourceforge and need a more customizable environment.

As to the single point of failure concern - I disagree for several reasons:

• We are not funded by the university. The OSL's activities are funded almost entirely by donations (both personal and corporate) and agreements with the projects we host. While we are all university employees, our wages are not paid using university dollars. Also, as part of the administrative computing organization at the university (as opposed to part of an academic department), the OSL falls under the university's CIO instead of a dean or department. The financial independence and organizational structure provides us with a significant amount of autonomy and insulation from the vagaries of university politics.
• OSU President Ed Ray has stated time and time again that the role of a land grant university in the 21st century is to provide leadership and assistance in information technology - much the same way the land grants provided support to agriculture and industry in past centuries. The OSL helps OSU fulfill that goal.
• On the FOSS community side, the OSL provides a vendor-neutral environment. We're not tied to any one distribution or manufacturer - we work with Dell, HP, and IBM all equally. The same goes for SuSE, Ubuntu, Gentoo, Red Hat, etc. IIRC, our neutrality one of the reasons master.kernel.org and the Linux Foundation reside at the OSL. We (and the university) consider that neutrality a very valuable asset.

It would take something more than a "pissed off dean" to summarily shut the OSL down.

-Greg

Re:Heh, they aren't admins

[gchaix]

I beg to differ. I've been a sysadmin for 15 years. The professionalism and quality of the work done by the students here at the OSL is quite often indistinguishable from many of the people I've worked with over the years. Many of the people working on our hosted projects can't tell whether they're working with our professional staff or student workers.

We teach them to be sysadmins. They may not be sysadmins when they come to us, but they sure as hell are professional sysadmins when they leave.

It's not trolling if it's true ;-)

[Zero__Kelvin]

"The ignorance of your post is one good indication of why they didn't replace him with you."

Well, I haven't seen someone display such blatant ignorance while calling someone with a clue ignorant for quite some time, so I guess I'll set the record straight ...

"An admins job isn't just 'make things easy on users'. There is a lot more that goes into it, which generally results in ignorant users getting mad at a good admin and wanting someone else."

That is an absurd thing to say, and the irony is that you claim to be a great sysadmin, but can't figure out that a good sysadmin doesn't have ignorant users (at least not for long.)

"Making users happy is rarely part of the job description anywhere. Making it so users can get what they need accomplished is."

And how do you plan to accomplish that while leaving them ignorant? You'd be surprised how much happier users are when you actually know how to do your job and educate the users so that they understand why something has to be done the way it does.

"Its fine for you to dick around with your own machine and have it offline, but the majority of a sysadmins work should be done without the users EVER HAVING ANY IDEA that its happening."

Are you fscking serious? Why the hell do you think they came up with /etc/motd ? (Message Of The Day for those who don't know and are following along.) If you are doing your job right then users know when backups happen. They know what new software you are installing, and when; you have visibility.

"Its cute though, that you think that while you're still in school, you're more capable to know what to do than all the other people, which have been running a school for years."

Maybe he has people similar to you setting the bar ;-)

Non-disclaimer: I was a VAX/VMS system manager at the age of 22, having been professionally trained by DEC at their Burlington Training facility, and I have been involved in various aspects of technology from sysadmin, hardware and software development, SQA my entire adult life (I'm now "over the hill"). I have had to deal with idiots like the parent my whole life, and his/her/it's attitude is outdone only by phenomenal cluelessness.