Managing Young Sys Admins At Oregon State Open Source Lab

mstansberry writes "Lance Albertson, architect and systems administrator at the Oregon State University Open Source Lab, uses a sys admin staff of 18-21-year-old undergrads to manage servers for some high-profile, open-source projects (Linux Master Kernel, Linux Foundation, Apache Software Foundation, and Drupal to name a few). In this Q&A, Albertson talks about the challenges of using young sys admins and the lab's plans to move from Cfengine to Puppet for systems management."

Comments

Especially if they are training developers

[xzvf]

Most universities don't teach good system management. The CS departments are training developers and programmers. Since good SA's like stability and good developers like chaos the two normally don't mix. Does OSU have a SA degree program?

Re:Especially if they are training developers

[jimbobborg]

Really? I've had the OPPOSITE experience. I've had to fix more crap done by developers who thought they could do sysadmin work than I have dealing with other SAs.

Re:Especially if they are training developers

[nine-times]

Planned upgrades are one thing. "I wonder what happens if I do this..." is another.

Re:Especially if they are training developers

[fuzzyfuzzyfungus]

The nasty trick is that, while not touching systems is very, very bad practice, being the first guy to touch something that hasn't been touched in a while is not a pleasant thing.

In an ideal world, all systems get regular attention and everything hums along smoothly. In a less ideal world, people are distracted from what is working by what isn't working, and their knowledge gradually atrophies, until they no longer dare touch what is working for fear of making it join what isn't working. This is a thoroughly pathological situation; but, if you are stuck in it, Just Not Touching and hoping for the best is quite possibly more logical than biting the bullet and taking one for the team.

Re:Especially if they are training developers

[russotto]

An excellent developer has a work ethic that would mesh nicely with an excellent sysadmin.

It's not the work ethic. It's temperment, the whole attitude towards things. As a developer, I'm very uncomfortable working on production systems, because there's a lot of walking-on-eggshells. If something goes wrong in the dev or QA environments, I can do a lot of "OK, try this... try this... try this", and if one of those things brings the environment down in flames, such is life. When I find the right combination it can be restored to the original broken state, then I can re-apply the thing I think worked, and if that works, I can then hand that off to ops to put on the production system with some confidence that it will work.

If a production environment has some problem which has to be corrected "live", though, that's a very BAD way to try to fix it, for obvious reasons. Instead, a lot more passive analysis has to be done before trying anything, and more important than "will this fix the problem" is "will this bring the system down (worse than it is) or worse, destroy customer data". And since when a developer is called in to help with production there's probably a major problem, the developer usually has to deal with the ops folks and management breathing down his neck as well.

Re:Especially if they are training developers

[BitZtream]

No, they weren't excellent developers.

They may have been great code monkeys, but those are literally a dime a dozen if you don't mind importing from India.

A GOOD developer is also a GOOD sysadmin.

I've been doing both for ~15 years, in that time, I've met 2 good developers. The rest have are monkeys who write code, many of which who write well. Maybe in the next 15 to 30 years I'll meet a few more. With everything going networked now days, you have to be a good sysadmin to understand how to write software that admins can use.

You don't write an apache webserver type application without being an admin, you just don't know what you are trying to hit. This isn't unique to developers writing for admins, its true for all development. If you don't know how to do and have experience doing what the software is supposed to be used for then its practically impossible for you to write software that doesn't suck. This is why there isn't a point of sale system on the planet that isn't asstastic. You'll be hard pressed to find a developer now days who actually had a job when they bothered to pay attention enough to know what good POS software needs to do.

Interestingly enough, I've never met a CS student or recent grad who was even a good code monkey. After several bad experiences our company has developed a 'no CS grad' policy for developers. We'll take you after you have 5 years or more experience, but with less than that all you are is an arrogant asshole who thinks he deserves to get paid ridiculous amounts of money and you still spend a few years breaking them down into something useful. (Read: removing the arrogance of youth and learning that there is far more software AND administration than just writing code or installing Linux)

Re:Especially if they are training developers

[mysidia]

The ideal combination would be for them to administer live systems and test systems at the same time, e.g. have access to both.

And utilize change control.

Use test systems to experiment and learn. Make proposals to accomplish tasks/maintenance that need to be done to live systems (such as upgrades).

On lab systems, they test/experiment with their proposed administration procedures/configuration changes. Before using them in the maintenance of live systems, they document what they plan to do step by step, and two other admins, a "partner", and a senior admin go over the procedure with them, question them about any apparent gaps, lookup any missing information needed, and take a copy of the procedure.

Then at the planned time, the young admin will run the procedure exactly as documented.

If they need help, or something (bad) should happen to them in the middle (e.g. they tripped over something, broke a leg, or broke their keyboard in the middle of maintenance), such that they can't complete the maintenance task, or get stuck, the other two admins both agree to be available to help and pick up if necessary.

Lesson 1

[tverbeek]

The main thing that people that age need to learn (both professionally and personally) is that Their Actions Have Consequences.

Re:Lesson 1

[bzipitidoo]

I think patience and learning when to say "no" are big ones too.

It's so tempting and easy to take shortcuts in system administration. "We don't need to waste time checking our backups" or, worse, "we don't need to backup" before doing major work is just the sort of time saving notion that can really haunt you if something goes wrong. Ugly when you need those backups and you discover the backup system you put into place in a similarly hasty fashion has some tiny little problem, maybe an incorrect flag on a command, and so the backups are no good. Can't spend all your time on paranoid checking either, of course. It's an art juggling these risks, deciding what is critical and what is not. There are never enough resources. If you have to make room in order to back up something, and it's going to take an hour or more to find things that can be deleted, clean out trash, compress directories that haven't been used recently, move files around, and so on, it's tempting to skip it, particularly if an impatient PHB is breathing down your neck, and other users are just waiting to pounce on that space the minute you free it up. Then there are the programmers who can't write anything that doesn't waste gobs of disk space and RAM. Someone notices when their code makes excessive use of the CPU, but a few megabytes of hard drive space flys under the radar. Some really think it isn't worth even a few minutes of their time to fix things like that, not when they're under the gun themselves to bang out more features as fast as possible.

Amazing.

[CannonballHead]

Young people with their heads on straight. Definitely newsworthy.

I know the whole "you young'un, you can't manage a server to save your life!" feeling and all that, but really... is managing a server, even an important one, really that hard - when you have someone to go to when you have questions? A lot of lab administration seems to be finding problems before they become a real problem, which is time consuming.

You may as well have a story about dental work done by *gasp* dental students and, lo and behold, they are actually doing a good job! Shocking. To think that young people could actually learn something. :)

OTOH, it's interesting to read about the difficulties he brings up. They're pretty ... boring, IMO.

It generally takes around six months for a student to feel comfortable with our environment.

Like most jobs?

Another challenge is the short turnaround with students, as we usually only have them for two to three years before they graduate. This creates a constant issue to ensure our documentation and training is honed.

Two to three years, that's not too short, is it? And it's interesting that it's an "issue" to him to keep their documentation good/honed. I hope the graduates are learning that documentation is a BIG ISSUE in real jobs, for exactly that purpose: if something happens to you, the business can't just stop for 3 months while someone else tries to figure out what you did :)

Nope

[autocracy]

The members of the CS department at my college actually petitioned to have me take over as their lab admin. The incumbent staff admin was notorious for breaking things and making it a chore to use the systems. Despite the complaints against him and requests specifically to hire me on, the department chair kept the incumbent.

I found it all very amusing, especially since I'm not a CS student. I'm just well-known enough to the group. I'm also greatly amused by how often I get asked for help when I'm around there, specifically one case where a student was in a 390-something class. I replied, "We really don't know each other at all, and I'm not a CS student. What made you think I am a good person to ask?" He said he'd just seen me help with enough other people's problems... and so I gave him a hand too.

Long-windedness aside, my university only uses students to provide, "Cean the viruses off your personal computer," services.

Re:Nope

[Sir_Lewk]

And you wonder why people don't like your type...

18-21

[Monkeedude1212]

I am in the upper bounds of that range. I do Sysadmin stuff in our corporation, though not as much as the Chief IT Manager. I do the cabling, I set up the racks, I make sure the UPS are tested regularily. All the grunt work a Sysadmin would do. I help with decisions on new network policies, and dealing with security and updates. Network Topology is something I wish I had a say in, but don't. I will on occaison, be called in to reboot a server, or replace a bad drive.

I had to learn the Help-Ticket system on the job, but really that was like a 5 minute breeze because most of it is common sense. (Ticket comes in, prioritize, assign, do)

I'm glad to see that younger people are getting into these positions, since I think they help push forward newer technologies and methodologies. It'll sound like I'm tooting my own horn here (and Maybe I am just a little :P) but we've got a dozen boxes in our server room plugged into the rack so that people from other branches across Canada can Remote in to access certain software. It's a nightmare to look at, and it takes up alot of space. The IT Manager isn't fully familiar with Virtualization, though thats something I was taught in school less than 2 years ago. I'm sure you can see where this is going.

All in all, the only thing holding back us young people from these positions is just experience. Almost any school you graduate from with a CS degree will teach you the fundamentals of system administration. However you can't exactly apply for that position with little to no experience (don't get me wrong, you CAN apply, but the guy who has 5+ years experience managing Windows Server 2003 is going to look a bit shinier).

It's good to have a Looong project like this to show you DO have experience. I went and switched from a CS Degree to simply an Object Oriented Programming because it was shorter and I enjoyed programming more, but now that I'm out here working I wish I had that education. (I know right, how did I land a Sysadmin/Technician job as an OOP grad? Funny story, ask me later). Anyways, If I could show my boss "Here's the webserver that I set up and maintained" I think he'd be more lenient with letting me handle things I know how to handle. It's frustrating when he mentions a problem and you know a solution but he won't admit its a good idea because you're fresh. That's more a problem with my boss though, and probably isn't a good representation of every manager out there.

Re:single point of failure?

[phoenix0783]

They're a mirror.

Re:I think I've been on the internet too long...

[Titoxd]

They are server administrators, which means they probably look like the average slashdotter. Do you really want to see that?

Sysadmins have good growth opportunity

[omgwtfroflbbqwasd]

While entry-level programmers may make a slightly higher salary than a similar systems administrator, over time there's a lot more upward opportunity for the sysadmin. Systems Engineering and Systems Architecture - being the guy that ties the network, the server, and the apps together, is a very in-demand skill and is something programmers will never have the opportunity to become. Programmers only make the big bucks when they have other specialized knowledge that's specific to the apps they are developing, i.e. finance, GIS, physics, etc..

I'm personally glad I made the decision 12 years ago to move into systems after earning my Comp. Sci. degree. I went from web app development for an ISP to Linux/Solaris/HPUX sysadmin, to Systems Architecure, to Info Security.

Re:single point of failure?

[MostAwesomeDude]

We're a rather bright spot on the university's record; we are the largest open-source datacenter in the hemisphere, and that causes a lot of donations to come in. Take it from Ed: http://osuosl.org/sites/osuosl.org/files/ed_ray.png Nobody will shut us down.

Re:single point of failure?

[gchaix]

I work for the OSU OSL.

Actually, we're more than a mirror. While mirroring is a major part of the services we provide, we also provide hosting for many projects' core infrastructure - Apache, Linux Foundation, Drupal, kernel.org, etc. Google is a major supporter of the OSL because we provide a place for projects whose needs have outgrown the more "off-the-shelf" structured hosting provided by Google Code or Sourceforge and need a more customizable environment.

As to the single point of failure concern - I disagree for several reasons:

• We are not funded by the university. The OSL's activities are funded almost entirely by donations (both personal and corporate) and agreements with the projects we host. While we are all university employees, our wages are not paid using university dollars. Also, as part of the administrative computing organization at the university (as opposed to part of an academic department), the OSL falls under the university's CIO instead of a dean or department. The financial independence and organizational structure provides us with a significant amount of autonomy and insulation from the vagaries of university politics.
• OSU President Ed Ray has stated time and time again that the role of a land grant university in the 21st century is to provide leadership and assistance in information technology - much the same way the land grants provided support to agriculture and industry in past centuries. The OSL helps OSU fulfill that goal.
• On the FOSS community side, the OSL provides a vendor-neutral environment. We're not tied to any one distribution or manufacturer - we work with Dell, HP, and IBM all equally. The same goes for SuSE, Ubuntu, Gentoo, Red Hat, etc. IIRC, our neutrality one of the reasons master.kernel.org and the Linux Foundation reside at the OSL. We (and the university) consider that neutrality a very valuable asset.

It would take something more than a "pissed off dean" to summarily shut the OSL down.

-Greg

Re:Heh, they aren't admins

[gchaix]

I beg to differ. I've been a sysadmin for 15 years. The professionalism and quality of the work done by the students here at the OSL is quite often indistinguishable from many of the people I've worked with over the years. Many of the people working on our hosted projects can't tell whether they're working with our professional staff or student workers.

We teach them to be sysadmins. They may not be sysadmins when they come to us, but they sure as hell are professional sysadmins when they leave.

It's not trolling if it's true ;-)

[Zero__Kelvin]

"The ignorance of your post is one good indication of why they didn't replace him with you."

Well, I haven't seen someone display such blatant ignorance while calling someone with a clue ignorant for quite some time, so I guess I'll set the record straight ...

"An admins job isn't just 'make things easy on users'. There is a lot more that goes into it, which generally results in ignorant users getting mad at a good admin and wanting someone else."

That is an absurd thing to say, and the irony is that you claim to be a great sysadmin, but can't figure out that a good sysadmin doesn't have ignorant users (at least not for long.)

"Making users happy is rarely part of the job description anywhere. Making it so users can get what they need accomplished is."

And how do you plan to accomplish that while leaving them ignorant? You'd be surprised how much happier users are when you actually know how to do your job and educate the users so that they understand why something has to be done the way it does.

"Its fine for you to dick around with your own machine and have it offline, but the majority of a sysadmins work should be done without the users EVER HAVING ANY IDEA that its happening."

Are you fscking serious? Why the hell do you think they came up with /etc/motd ? (Message Of The Day for those who don't know and are following along.) If you are doing your job right then users know when backups happen. They know what new software you are installing, and when; you have visibility.

"Its cute though, that you think that while you're still in school, you're more capable to know what to do than all the other people, which have been running a school for years."

Maybe he has people similar to you setting the bar ;-)

Non-disclaimer: I was a VAX/VMS system manager at the age of 22, having been professionally trained by DEC at their Burlington Training facility, and I have been involved in various aspects of technology from sysadmin, hardware and software development, SQA my entire adult life (I'm now "over the hill"). I have had to deal with idiots like the parent my whole life, and his/her/it's attitude is outdone only by phenomenal cluelessness.

Re:How do you find a young sys admin??

[gchaix]

99.99 of sysadmin'ing comes from experience

Right ... which is why we here at the OSL give them the opportunity to gain that experience in a real-world production environment while providing the mentorship they need. It dovetails nicely with the theoretical knowledge they're getting in their CS classroom work.

Yes it is.

[Toze]

An admins job isn't just 'make things easy on users'.

Yes it is. It is an admin's job to make things as easy as possible on the users over as long a period as possible. That is why backups are made; so the users don't have to redo all their work if there's a failure. That's why there's firewalls; so the users' machines don't get infected and their network isn't crippled. Without an admin, small organizations can chug along until something breaks (and they have to contract an admin to patch it), but life isn't easy. A full-time sysadmin for a company or a department has only one purpose; to make things easy on the users.