Managing Young Sys Admins At Oregon State Open Source Lab

mstansberry writes "Lance Albertson, architect and systems administrator at the Oregon State University Open Source Lab, uses a sys admin staff of 18-21-year-old undergrads to manage servers for some high-profile, open-source projects (Linux Master Kernel, Linux Foundation, Apache Software Foundation, and Drupal to name a few). In this Q&A, Albertson talks about the challenges of using young sys admins and the lab's plans to move from Cfengine to Puppet for systems management."

Comments

Especially if they are training developers

[xzvf]

Most universities don't teach good system management. The CS departments are training developers and programmers. Since good SA's like stability and good developers like chaos the two normally don't mix. Does OSU have a SA degree program?

Re:Especially if they are training developers

[nine-times]

Planned upgrades are one thing. "I wonder what happens if I do this..." is another.

Re:Especially if they are training developers

[mysidia]

I think a better analogy might be professional diver, airplane pilot, or astronaut

It's not a field where you can simply be educated on the subject, and become a practitioner that way, you need hands-on experience. A large amount of intense training is required to do it competently.

And is also a risky business... if you're the DBA, and the enterprise database server breaks (hardware failure)... who will the fingers be pointing at for blame?

What are people going to say to you, when you tell them... yeah.. our RAID array got hosed, due to 2 simultaneous disk failures. we gotta rebuild from backups, it's going to take 72 hours.

And no business can be done until it finishes. Yeah... you don't want to be sysadmin at a time like that. Even if you tried to sell management before on a better disaster recovery plan. You're fortunate if they don't start wondering if you broke it in retaliation for them refusing to adopt your plan... and the scary thing is some (unprofessional/immature) sysadmins might do that sort of thing.

It's a myth that people believe in "don't shoot the messenger".

Large-scale sysadmin work is beyond the abilities of the average person. Both knowledge and skill are required. In many cases... programming skill; to an extent, all good sysadmins are programmers (scripters usually), but that doesn't mean all programmers are good sysadmins.

Skill is developed over years of experience, it can't be taught beyond a certain extent.

It's also a niche field. At the high end, there are not very many system administrators in the world, perhaps a few hundred thousand at most.

At the low end, everyone and their brother, thinks they're a system administrator, with zero training, zero schooling, etc.

You know how to plug a cable in? You have a CCNA? Great, you're hired.. network admin for a 5000 workstation network..

So even if you do study for years, get lots of experience, become a really skilled system admin... you still have the lowest common denominator as competition, when it comes to employment.

Anyways... I lost track of my point.. it's nothing like "Janitor"

Almost all janitors basically do the same thing, it's not complicated at all, some skill might be required, but not much.

Nothing that can't be picked up in a few days. The skills and knowledge required to be a janitor are common, and don't need to be taught -- just about every member of the public has them.

The same is not true of a good sysadmin.

I'll admit there are some exceptions, but there are a lot more janitors in the world than real system admins.

Nope

[autocracy]

The members of the CS department at my college actually petitioned to have me take over as their lab admin. The incumbent staff admin was notorious for breaking things and making it a chore to use the systems. Despite the complaints against him and requests specifically to hire me on, the department chair kept the incumbent.

I found it all very amusing, especially since I'm not a CS student. I'm just well-known enough to the group. I'm also greatly amused by how often I get asked for help when I'm around there, specifically one case where a student was in a 390-something class. I replied, "We really don't know each other at all, and I'm not a CS student. What made you think I am a good person to ask?" He said he'd just seen me help with enough other people's problems... and so I gave him a hand too.

Long-windedness aside, my university only uses students to provide, "Cean the viruses off your personal computer," services.

Re:Nope

[BitZtream]

The ignorance of your post is one good indication of why they didn't replace him with you.

If you have a bunch of CS students petitioning to make you the admin, thats another good indication that you shouldn't be doing it. Part of this I know because I'll bet a months pay that the job description for the position doesn't include 'CS students must think your a swell guy and a good admin', which you seem to think IS part of it.

An admins job isn't just 'make things easy on users'. There is a lot more that goes into it, which generally results in ignorant users getting mad at a good admin and wanting someone else. Making users happy is rarely part of the job description anywhere. Making it so users can get what they need accomplished is. Sometimes part of the requirements, especially at an educational facility is to specifically PREVENT users from doing things the easy way. You'll understand some of this more when you get older and realize that most of the education you get in college isn't what you hear in lectures or read in books.

Your post smacks of a young, know it all with no experience and a lot less skill than you realize. Its great that you think the management at your school is stupid, I mean, they've only been doing it longer than you, you must obviously be better at it than them and know more than they do, I mean, thats why your going to their school rather than managing their school. You always want to learn from people who know less than you do.

Just because you know how to use a computer, doesn't make you an admin. It doesn't make you aware of all the stuff that goes on behind the scenes in a large organization such as a university. You THINK you can do better when you really don't know what this person does across the board.

CS students are most certainly always at odds with their admins. Its a bunch of arrogant socially inept kids with no real world experience who think they know everything there is to know about technology and that no one else has any idea how it works. To top it off, most CS students that come out suck ass at CS. I've hired from UNC, NCSU and Duke university for CS, obviously these aren't strong points here, however, our company now has a policy of not hiring anyone out of college with less than 5 years work experience. I'll take you off the street with a high school degree in a minute if you impress me, CS students on the other hand take far too long to knock the ignorance out of and get them to realize they don't actually know that much. This certainly isn't unique to CS, but it is more common there. The result is a bunch of CS people who think they should be able to do whatever they want, whenever they want without regards for anyone or anything else. Virus authors are less damaging to a network than a group of CS grads with root.

Do you know every task those machines were intended to be capable of performing? Do you know the laws regarding security requirements for your state? Do you know what rules they have for vetting software to ensure its compatible? Have you ever actually been involved in the process of upgrading software across a campus? This isn't like when you run apt-get on the Ubuntu box in your basement. Its fine for you to dick around with your own machine and have it offline, but the majority of a sysadmins work should be done without the users EVER HAVING ANY IDEA that its happening.

Its cute though, that you think that while you're still in school, you're more capable to know what to do than all the other people, which have been running a school for years. What I wouldn't give to go back to the time in my life where I knew more than everyone else. Ignorance was bliss. Those were good times. I guess when you have the cockiness knocked out of you in a few years after you rich the real world and fuck something up due to your arrogance that you may be better at it, my first half a million dollar mistake because I left a couple 0s off the end of a polling time did a good job putting me in my place, yours will probably do the same. It it doesn't, you won't be in the field long anyway so either way the damage will likely be contained. Theres a grad students get hired as interns.

Re:Nope

[CAIMLAS]

As someone who has approximately 5 years of experience, including 6 years of part-time and contractual small business network/system admin work and several years in small-medium hospital sysadmin work, let me just say that his attitude seems to be very, very prevalent.

Granted, I may be getting lied to, but I've been told that I don't have any "big shop experience" even though I was one of three admins handling 250 Linux servers and several thousand workstations. This just happened to be a number larger than they had: this organization was simply in a different sector.

My experience is that there are two lower level tiers of sysadmins: those which require 2 years of experience, for which anyone with over 3 years of experience is over-qualified for; and 5 years of experience, for which anyone with less than 6 or 7 years of experience is under qualified for. So if you want to do sysadmin work, you better reconsider your options, or hope to god you're able to hop jobs and/or avoid being laid off at the same place for a span of 4 years. Once you're unemployed, you're done.

18-21

[Monkeedude1212]

I am in the upper bounds of that range. I do Sysadmin stuff in our corporation, though not as much as the Chief IT Manager. I do the cabling, I set up the racks, I make sure the UPS are tested regularily. All the grunt work a Sysadmin would do. I help with decisions on new network policies, and dealing with security and updates. Network Topology is something I wish I had a say in, but don't. I will on occaison, be called in to reboot a server, or replace a bad drive.

I had to learn the Help-Ticket system on the job, but really that was like a 5 minute breeze because most of it is common sense. (Ticket comes in, prioritize, assign, do)

I'm glad to see that younger people are getting into these positions, since I think they help push forward newer technologies and methodologies. It'll sound like I'm tooting my own horn here (and Maybe I am just a little :P) but we've got a dozen boxes in our server room plugged into the rack so that people from other branches across Canada can Remote in to access certain software. It's a nightmare to look at, and it takes up alot of space. The IT Manager isn't fully familiar with Virtualization, though thats something I was taught in school less than 2 years ago. I'm sure you can see where this is going.

All in all, the only thing holding back us young people from these positions is just experience. Almost any school you graduate from with a CS degree will teach you the fundamentals of system administration. However you can't exactly apply for that position with little to no experience (don't get me wrong, you CAN apply, but the guy who has 5+ years experience managing Windows Server 2003 is going to look a bit shinier).

It's good to have a Looong project like this to show you DO have experience. I went and switched from a CS Degree to simply an Object Oriented Programming because it was shorter and I enjoyed programming more, but now that I'm out here working I wish I had that education. (I know right, how did I land a Sysadmin/Technician job as an OOP grad? Funny story, ask me later). Anyways, If I could show my boss "Here's the webserver that I set up and maintained" I think he'd be more lenient with letting me handle things I know how to handle. It's frustrating when he mentions a problem and you know a solution but he won't admit its a good idea because you're fresh. That's more a problem with my boss though, and probably isn't a good representation of every manager out there.

Re:Lesson 1

[bzipitidoo]

I think patience and learning when to say "no" are big ones too.

It's so tempting and easy to take shortcuts in system administration. "We don't need to waste time checking our backups" or, worse, "we don't need to backup" before doing major work is just the sort of time saving notion that can really haunt you if something goes wrong. Ugly when you need those backups and you discover the backup system you put into place in a similarly hasty fashion has some tiny little problem, maybe an incorrect flag on a command, and so the backups are no good. Can't spend all your time on paranoid checking either, of course. It's an art juggling these risks, deciding what is critical and what is not. There are never enough resources. If you have to make room in order to back up something, and it's going to take an hour or more to find things that can be deleted, clean out trash, compress directories that haven't been used recently, move files around, and so on, it's tempting to skip it, particularly if an impatient PHB is breathing down your neck, and other users are just waiting to pounce on that space the minute you free it up. Then there are the programmers who can't write anything that doesn't waste gobs of disk space and RAM. Someone notices when their code makes excessive use of the CPU, but a few megabytes of hard drive space flys under the radar. Some really think it isn't worth even a few minutes of their time to fix things like that, not when they're under the gun themselves to bang out more features as fast as possible.

Re:Lesson 1

[Bandman]

I agree, as well. 90% of my time spent when teaching my junior admin is teaching him how to think like a sysadmin instead of a hobbyist.

Re:single point of failure?

[MostAwesomeDude]

We're a rather bright spot on the university's record; we are the largest open-source datacenter in the hemisphere, and that causes a lot of donations to come in. Take it from Ed: http://osuosl.org/sites/osuosl.org/files/ed_ray.png Nobody will shut us down.

Re:single point of failure?

[gchaix]

I work for the OSU OSL.

Actually, we're more than a mirror. While mirroring is a major part of the services we provide, we also provide hosting for many projects' core infrastructure - Apache, Linux Foundation, Drupal, kernel.org, etc. Google is a major supporter of the OSL because we provide a place for projects whose needs have outgrown the more "off-the-shelf" structured hosting provided by Google Code or Sourceforge and need a more customizable environment.

As to the single point of failure concern - I disagree for several reasons:

• We are not funded by the university. The OSL's activities are funded almost entirely by donations (both personal and corporate) and agreements with the projects we host. While we are all university employees, our wages are not paid using university dollars. Also, as part of the administrative computing organization at the university (as opposed to part of an academic department), the OSL falls under the university's CIO instead of a dean or department. The financial independence and organizational structure provides us with a significant amount of autonomy and insulation from the vagaries of university politics.
• OSU President Ed Ray has stated time and time again that the role of a land grant university in the 21st century is to provide leadership and assistance in information technology - much the same way the land grants provided support to agriculture and industry in past centuries. The OSL helps OSU fulfill that goal.
• On the FOSS community side, the OSL provides a vendor-neutral environment. We're not tied to any one distribution or manufacturer - we work with Dell, HP, and IBM all equally. The same goes for SuSE, Ubuntu, Gentoo, Red Hat, etc. IIRC, our neutrality one of the reasons master.kernel.org and the Linux Foundation reside at the OSL. We (and the university) consider that neutrality a very valuable asset.

It would take something more than a "pissed off dean" to summarily shut the OSL down.

-Greg

Re:Lesson 1

[russ1337]

I've seen single seat fighter jocks in that age range.... age has little to do with it. Training and attitude have lots to do with it.

Re:How do you find a young sys admin??

[gchaix]

99.99 of sysadmin'ing comes from experience

Right ... which is why we here at the OSL give them the opportunity to gain that experience in a real-world production environment while providing the mentorship they need. It dovetails nicely with the theoretical knowledge they're getting in their CS classroom work.