A Sysadmin for Sysadmins?

crazyharry asks: "I have recently been hired to be a system administrator to a bunch of system administrators. Aside from my personal experience, which is probably biased, I would like to know from the disproportionately large number of IT people here: if you, as a system administrator, were forced to have a system administrator, what would you expect of that role? How would you want your business machines (not the ones you admin, but your daily use machines) managed, if they were not up to this point? This is a mixed environment (Windows, Mac, and Linux/Unix), so feel free to assume I've already heard the 'leave me the FSCK alone' comments. What other issues are probably going to crop up, if you have been in a similar situation?"

Comments

Quit

[KPU]

Sysadmins are going to make your job hard (wouldn't you?). Nobody likes knowing how to fix a problem but having to go through somebody else. Why are you needed? This smells like a manager came up with the idea without understanding how sysadmins operate.

Re:Quit

[cooley]

I think a sysadmin can be of good use to sysadmins, in the right environment, though I must admit I've never done that. Right now I'm sysadmin for a bunch of software developers, but I've never admin-ed admins.

Anyway, I look at it like this. Part of what a sysadmin does is decide what hardware we need, and order stuff. If I had another sysadmin ordering crap for me, that would be great! I'll worry about what the clients need, you order me a copy of the new version of XXX, or roll up to my desk with a gigabit network card and a grin on your face. That'd be great.

When was the last time any of us Sysadmins came to work and somebody said "I ordered us all a sweet new mouse" or "hey the router config was horked, that's why we were having problems" or "hey, the RAID we back up to was getting kinda full, so I got us a shitload more drives". All those things, I ordered or fixed myself and *then* had to go be a sysadmin for the clients' machines.

In a lot of companies, you're ordering/configuring/maintaining licenses and hardware for somebody else's business, or somebody else's department, or whatever. In my book, if the funds are there to have soembody around to take over those duties for my own department, that's cool. Just because I *can* do something doesn't mean I have time to.

Sure, I'll fix my box when it has an issue. It's not like these sysadmins will need tech support; In the end IMHO your best use as an admin's admin isn't that much different though, aside from that. Keep stuff running smoothly so that I don't have to think about it. With users it's the same thing, except in this case the users are thinking about somebody else's network instead of shiny foil or whatever it is non-sysadmins think about.

Good, Bad, Ugly

[Dukeofshadows]

As I see it:

The *good* news is it might make existing sysadmins more symapthetic to the needs of their users if they have to experience the same sort of interactions with them that some experience with their co-workers (some of my own experiences have been negative in the past and as such are biased).

The *bad* news is that more bureaucracy means more places for people to hide, more paperwork for everyone to got through, and another layer of clearance required for people to do their jobs.

The *ugly* news is that a single feudal overlord of a sysadmin with political qualifications instead of technical ones might turn a workplace from a productive well-oiled machine to a mess rather quickly.

2 types

[drDugan]

in my experience, there are 2 completely differnt types of sys admins:

- ones who think computers are their clients
- ones who think people are their clients

both have their plusses and minuses. it seems that some people fit one camp - other people fit into the other camp, and they don't easily change. personally I prefer the sys admins who focus on the people first, and get the computers to meet their needs. I'd make sure in your case you know the expectations of the people who you work for and work with and see if you fit their expectations.

I would guess from your post, and the fact you wanted input from a large group of people on /. you are more the "people first type"

Good luck!

Re:2 types

[GuruBuckaroo]

Make that three - you forgot the type that feel the computers are their property, and the users would be better trusted with legal pads and abacuses.

You can include me in the third type.

Not much difference...

[yuriismaster]

....than how your sysadmins would run their network.

You keep it open enough for them to do their job, and not much else, provide the proper storage and network services that they require, and that's about it. What I see as the main difference is that your users aren't dumb enough to open .exe file attachments, which is good for you.

Expect a lot of griping from your sysadmins, mainly involving filtering out Quake server traffic, if it comes to that. You have a job to do, so just do it.

So, do you LIKE herding cats?

[wcrowe]

You're in for a tough job. This is bound to be even worse than managing a group of programmers.

A Suggestion

[Mr_Tulip]

Delegate responsibility to the sysadmins, and set up guidelines as to who is responsible for what. Generally, be more of a manager than an active administrator.

Also, don't be afraid to impose restrictions on the other administrators. Communicate clearly why these restrictions are required, and where possible, allow the administrators to make their case as to why they need the restrictions listed. Listen to their arguments, and alter your guidelines if needed.

If you have time and money, play with the budget you have at your disposal to make life easier for yourself and your charges.

Re:A Suggestion

[ePhil_One]

Delegate responsibility to the sysadmins, and set up guidelines as to who is responsible for what

You are suggesting he doesn't do his job? Personally, I'd lock them out of their machines and buy them all a second "Test" system that they could abuse (or better, a VMware/Xen virtual system that can be quickly restored when they screw it up), reformat at will, screw royally, etc. They can SSH / Terminal Server / X-window to THAT achine when they get the urge to "play", that way their system stays nice and stable. I think I'd also set that machine up on a isolated VPN just to be safe. Build them a sandbox. Just don't beleive because they can admin whatever systems they are responsible for they can admin their desktops. Thats HIS responsibility.

Re:A Suggestion

[Mr_Tulip]

Depends on your sysadmins, I expect.

In a multiplatform setup, I reckon it's better to acknowledge that you are not an expert in every field, and let those who now best manage their domain.

Hats off to you if you are the uber-expert, of course.

Simple.

[Telastyn]

Don't enforce; Provide.

Re:Simple.

[MarkChovain]

Excellent advice.

I have seen two types of sysadmins in the past: the good ones, who see their role as a provider of IT services; and the bad ones, who see their role as the enforcers of rules and regulations. Consider both organisms such that you spend a lot better. Most of the things don't make you addicted to water and air. Computers don't have a game incorporated into your surroundings?

When you get the kind of sysadmin who see their users as subordinates (rather than both groups working together toward a common goal), it can destroy the effectiveness of a company.

Now, if you act as merely a "policy police officer" on the other admins, then they are always going to treat their users the same way. Be there to help and support them, not hinder them to give yourself a power rush.

So many other folks don't seem to conflate everything around eros and thereby miss the whole point of mental distress. This is not the team is. Management makes money regardless of how it has lost any meaning which we breathe, move, and have it delivered free.

Me, bitter? ;).

suck

[kassemi]

if you, as a system administrator, were forced to have a system administrator, what would you expect of that role

You said it... If I were forced to have a system administrator... You should maintain your distance, be amiable. You should offer all the help you can to those who are either not doing things properly (in which case you can be somewhat forceful, but be sure you know they're doing it wrong), and to those who actually ask for it. If you start bossing them around, you're not going to last long (psychologically) in your position.

Hell, depending on how many of them there are, just buy them all a beer Friday after work... That ought to make thing easier

I'm a sysadmin who has a sysadmin

[Theatetus]

This seems to be the opposite of what other people say, but as a sysadmin who has a sysadmin I can say I like mine because I never have to apply my sysadmin-ing to our internal computers. I don't expect to be given Special Powers just because I've got root somewhere else, but I expect the same quality of service I deliver to our external clients (well, OK, I expect better than that).

I'm not root on our local Linux boxes; I'm not a domain admin on our local Windows domain (though I think I'm a schema admin for some reason) -- I don't want to be. I want the local resources I need to connect out and do my work, and I don't want to have to think about them.

YMMV.

Uh, don't take the job?

[davie]

You're being asked to fill a position that amounts to a collective slap in the face to a whole department. How do you think this is going to turn out for you? I don't have much information to work with here, but my suspicion is that you know that this situation is a bit awkward, to say the least, and you're not sure about it. Follow your gut and take a different job.

Managing people or machines?

[cswiger2005]

Is your job to manage a team of sysadmins, or just manage a bunch of desktop machines which happen to be used by sysadmins? The latter is generally a lot easier, although some admins can get as picky about someone managing their personal box as any huffy user.

If you're just managing the machines, make sure you've got a software baseline (start with, what software do we have and how many legit licenses have we paid for), and make sure that all of the machines which are supposed to have that stuff do, and it is patched and up-to-date. Keep a mental checklist of any machines which have severe problems; that may not be the sysadmin user's fault, but it still ought to be a warning sign.

Sysadmins always seem to get buried in the never-ending stream of building new machines, or running Windows update, or virus-scanning, etc, but try to keep the team aware of longer-term goals beyond the humdrum of such routine daily tasks. Try to give everyone at least two long-term goals or tasks to complete: one that's fun or interesting or cool, and one that sucks or is boring but needs to be done. Make sure everyone knows that everyone has got to deal with some suckatude, publicly praise/reward the first guy who finishes a sucky task. On the other hand, if someone gives you problems or blows off the difficult task, make another public point of awarding another sucky task to the guy.

Only answer I can think of

[br00tus]

Since usually I'm the sysadmin, I can only think of machines I used where I didn't have root access, yet used the machines.

Two thing thats piss me off the most usually is limitations on my access, or annoying security measures, both of which I look at similarly because they are different sides of the same coin often. I host a website on a host where I don't have root access. They are supposed to be good, and a place geeks like to host, and for the most part they are. But having no root access can be annoying. For example, the machine load average was very high for some weeks. It would shoot up to something like 10 or 20 times the number of processors for an hour and then go back to normal. My e-mails to the techs didn't do anything for some weeks. My ps only let me see my own processes, I couldn't see what processes were hogging the machine. The first time they checked, the spike hadn't happened, so they had no idea what was wrong. So they were slow to do anything about it, I had the ability to better diagnose what was wrong. Eventually I ran a script that did an uptime every minute and wrote it to a file. But after two days they killed that - that's another thing, they killed a script that I was running. Although if it was an attempt to find this rogue process, I didn't care as much. Anyhow, eventually they fixed the problem.

Another thing that happened with these hosters, which again is related to me not being able to see system processes with ps - one day my password protection for directories (htaccess) died. I had to recreate everything with their automatic system in terms of the htaccess and htpasswd files. I couldn't see what user was running our Apache web server processes, I just had no idea why it broke.

Once I worked at a company where you needed SecureID to log into their machine for customers, among other security provisions. I thought it was rather silly - I only read mail from the machine, and not much else, why do I need a SecureID card to do that? Wasn't ssh enough? Did I have to carry around a SecureID card just to access this one machine and my e-mail which I read with pine? Again, a mixture of limited access and what I felt was unnecessary security is what pissed me off. Our company had a lot of smart programmers and sysadmins, I'm sure anyone motivated enough to hack in could get in and get root despite the SecureIDs. It sort of reminds me of the World Trade Center. The security to get in was ridiculous after the first bombing. But they hadn't walked into, but drove into the building the first time, so why was taking my picture and other silly measures necessary? It did little for them as they eventually got flown into, which destroyed the buildings. As I said, once something becomes a target for somebody motivated enough, there is little you can do.

Senior Administrator

[Alien54]

The point should not be to have your finger in everyone's pie. The analogy should be closer to a hospital with a senior surgeon, vs a manager of managers.

That said, if everything is working well, you become the buffer between the sysadmins and the rest of the world.

You get to be the one that goes to HR and complain about Clueless User #69 in cubicle 18 with his inappropraite visit to the wrestling website that installed spyware for a solid hour over lunch. You would also get to run the pilot projects before they role out company wide. You test the new toys, using the other sysadmins in fair rotation as project managers for the test.

You also get the really big headaches, like when Clueless User #69 is the incredibly cute and hot granddaughter of the boss, or some such thing (who never does anything wrong. No. Really.)

Re:2 types - can we make that 3 types?

[karearea]

Hmm, yes I agree ... sort of.

I think that there is a 3rd type.

- ones who think the business is their client.

Where ultimately although people are important, they aren't more important than the business - no precious people please.

Yes, people have to be able to do the job. But if the machines aren't working then the peope can't do the job. In a way you almost need to assume the worst of people - not nice, but by doing that you protect the business by ensure those that don't kow any better don't get 'their' machines into a state where they can no longer do their job. Machines weighed down by extra 'applications' such as viruses and trojans don't help people work.

A lot of windows (for those that use it) applications are now better able to deal with multiple people using the same machine and are more intelligent about permissions and what people need access to to get an application working - still far from perfect, but better than 5 years ago!

By ensuring the machines are stable and in a controlled, locked down environment (Control Freak Alert!) you can ensure that a business can continue. Everyone (including me) must remember that the business network is more important than any one person.

Just be the cool guy...

[AEther141]

with a cupboard full of spare parts and useful bits, a headful of clue and an open office door. Talk to all the people you're adminning for, ask them what they want doing and make sure you don't overstep your agreed-upon boundaries. Make it clear to everyone that you're just there to help everyone get their job done faster and whether they want root or just a reliable box to SSH from, that's what you'll provide. Deal with pissy bureocrats on their behalf, harangue the network guys when things go wrong, just try and create as pleasant and hassle-free environment as possible.

An effective compromise that works for us...

[Anonymous Coward]

I work in a department of computer scientists, where the average user is more than capable of providing their own sysadmin support. We do have a computing support department.

Effectively, the users partition themselves into two camps.

The first camp is more supported. They have the "official" image loaded on their hard drives, they log into the domain as users, and so on. When their stuff breaks, support comes out and fixes it.

The second camp is less supported. They have a "if I don't bother you, can you not bother me?" policy. They can run whatever they want, but if something breaks, they have to fix it themselves. Want to load Yellow Dog BSD on that old AIX box? Sure thing, just don't call us. Support will (at most) reimage you and put you in the first camp. Access to certain resources is more limited (non-support-sanctioned UNIX machines cannot mount NFS, for example).

I'm in Group 2. For us, support has been liberally good with requests outside the user's local domain of responsibility. For example, if a printer I use breaks, or I need an IP address assigned, or I'm having a problem with mail on the server side (and I'm really, really sure it's not a client side problem) they are more than happy to help. Activities where a user in Group 2 negatively affects other users (getting infected with a worm or port scanning the local net for fun) are rare and handled on a case-by-case basis. The worst penalty is that they'll unplug your wall jack from the switch and that generally keeps the rest of the users humming along quite nicely until you get your situation resolved.

In my experience, this is an effective compromise. Users in the first camp get more of support's facilities in exchange for some freedom. Users in the second camp get more freedom in exchange for less support. Anything in the "neutral zone" gets handled equally for both parties. In general, this keeps most people happy, including support, advanced users, and more mainstream users.

Re:Gah!

[Forge]

I work for a company of contract geeks. One rule to remember is that "if everyone is responsible, Nobody is responsible." I.e. Who adds and deletes accounts for admins as they join and leave the team? You must have 1 individual with that duty otherwise it may not get done or worse get redone.

Also there is the matter of accountability. If every nerd in the department has full read/write access to the Email server who do you fire (or shoot) when the mail you need for evidence disappears?

So yes. Every IT teem must have a sysadmin. Even if it's all sysadmins. For really small teems with limited internal requirements you can add other duties to that goys list until he gets up to averaging 40 hours per weak. I.e. He could maintain your library of technical documentation and software or act as secretary to the team. Imagine this exchange:

Mark -: "Joe. I feel like I'm Ready for CCIE, Book me for an exam next week"

Joe -: "cool."

Joe (Next day, via email) -: "You're on the 10:00 AM flight from Norman Manley Airport on Tuesday, You will be staying at the Marriott inn. Your exam is on Wednesday morning and you head back here on the 12 noon flight."

BTW: Andrew and myself will be handling your workload for those days. Any outstanding issues we need to know about in advance?