Seven Habits of Highly Effective Unix Admins

jfruh writes: "Being a Unix or Linux admin tends to be an odd kind of job: you often spend much of your workday on your own, with lots of time when you don't have a specific pressing task, punctuated by moments of panic where you need to do something very important right away. Sandra Henry-Stocker, a veteran sysadmin, offers suggestions on how to structure your professional life if you're in this job. Her advice includes setting priorities, knowing your tools, and providing explanations to the co-workers whom you help." What habits have you found effective for system administration?

Comments

Number 6 Problem

[magamiako1]

The issue with #6 is that users almost invariably never accept an answer here. And a lot of the time it may be something you can't adequately explain, which is something they don't like even more. Especially if you know the problem wasn't the result of something you did.

Tmux

[matthiasvegh]

I discovered tmux (terminal multiplexer) a while back, and is a very potent replacement for screen, it supports splitting windows, having multiple sessions, sharing windows between sessions, customizable status bars etc. Try it out!

Re:Tmux

[oodaloop]

Sudo try it out!

Re:Tmux

[evilviper]

For my use cases, I could not find a compelling reason to use tmux

Obviously if you've been limiting yourself to the features of "screen" for many years, you're not going to think you need the added features of "tmux"...

A big one is sharing:
"window can be linked to an arbitrary number of sessions". If you or somebody else has a screen session open, you don't have to detach it from their terminal to see what's on it. You can just attach it to your terminal as well. Works great when you've got a session attached to your desktop, then want to access it on your laptop/tablet/phone/etc. The tmux session will even change geometry to match the smallest terminal window.

Being more lightweight and responsive is good. Saner keys for some functions, like ctrl-a pg-up to access scrollback. And just the fact that it's still getting active development is an important feature.

Re:Tmux

[Chalex]

screen -x shares the screen just fine for me.

Re:Tmux

[dissy]

Here it is in nicm@'s words:
"In particular, being able to share a single window between multiple terminals, with other windows in the same session but entirely separate. Adding this to screen was implausible"

Perhaps I am still misunderstanding the features of tmux (most likely in fact), but to say that is implausible to add to screen is misleading to say the least, since I have been doing exactly that in screen for nearly a decade.

On one terminal, either start a new screen session or -r to a detached session.
If starting a new one, try: screen -S LetsShare

On a second terminal, run: screen -list
You should see a list of screen sessions and their status (attached, detached, multi, etc)
If you used -S on start that will be the name, otherwise it's some tty.host.number string.

Now on that second terminal run: screen -x

Try to adjust both terminal sessions so you can see them at the same time. Type in either, watch in either. They are shared seemingly matching your tmux description.

You can change permissions per terminal so others can't type but will see everything you do (aka tutorial mode) using ^a *

Also for split/multiple windows showing on the same terminal, use ^a S (control-a capital-S)
To switch between split windows use ^a tab
Close a section of split window with ^a Q

The status bar problem is true and pretty annoying. I fixed it myself with a line in ~/.screenrc but of course I have to pretty much install that user config file on every new system I use which can get annoying.
If you want an always-on status bar showing window numbers and titles (^a A to change the title), add this to .screenrc (and hope slashdot doesn't munge it!)

hardstatus alwayslastline "%{= wk}%-Lw%{= BW}%n%f* %t%{-}%+Lw %-=%{= BW}%H%{-}%{-}"

Note the two "BW" bits? That's background blue and foreground white, and applies to the window with focus. Change B to R for red for example (production vs not-production in my case)

Here is my whole .screenrc file for copy/paste purposes: http://pastebin.com/kMkuFXi9
No splash screen, always on status bar, 10k line scrollback history for copy/paste (^a [ and ^a ] ), and auto-open three windows with preset titles and commands running in them.

I don't mean to knock tmux in any way at all, having not used it (and I do plan to check it out now) - but hopefully these screen tips help out others here.

Using multiple shells

[Neruocomp]

When working on a problem, I usually have two or more shells open. I don't mean multitasking, but with more then one open, I can issue commands from one and use the others to monitor logs/etc.

i was so wrong

[zlives]

i thought they were
sloth, gluttony, pride,...

Re:i was so wrong

[kimvette]

That's "How to be a BOFH"

Habits ...

[Xaemyl]

What habits have I found effective for system administration? BOFH spring to mind ...

Knowing your tools

[Rosco P. Coltrane]

I know them all. They all work in Marketing.

Re:Knowing your tools

[rcamans]

Apparently you have not interacted with management much, or you would not have restricted your answer to marketing...

#7 Be Appriopriately Lazy

[tiberus]

The first time a task comes up deal with it manually, it may or may not be related to a problem.

The second time this task occurs deal with it manually.

The third time this task occurs, it's time to start scripting.

It may take you a day or more to write the script, test debug, etc. or even longer for complex tasks but, this behavior tends to be a winner. The script is already some degree of documentation, it records the steps, etc. If it's robust enough it can be used to by your support techs to resolve issues, expanding the number of people who can resolve an issue, freeing the admin for other tasks. Scripts tend not to make typos (yes, I know your command line skills are legendary) and can save a lot of time and effort in the long run.

Re:One habit is ...

[Fallen Kell]

The reason there are more fat people in IT isn't because we want to be. It is because the GOOD IT people get fat because they know that the best IT people never need to leave their seats. If you have to leave your seat to do something as an admin, you are doing something wrong and not using the technology that is available to you to be able to fix everything but physical hardware failure or installation from your seat.

Re:Bait

[cusco]

From TFS, I really don't get why that applies only to Unix admins. That describes the years I've spent as a Windows admin as well.

Rebooting is not a fix

[hawguy]

As someone who's managed a team of sysadmins that moved to the Linux world from Windows, I have this tip: "Reboot does not fix anything, it just hides things".

For some reason, Windows admins have been trained to reboot immediately when things don't work well rather than to figure out why something is failing. I'm sure this was a valid "fix" in older versions of Windows, but Windows has been stable for quite some time, and things shouldn't mysteriously stop working for no reason. Take a bit of time to figure out *why* the CPU is suddenly spiking on the database server, since if you reboot it, you will have lost most of the evidence for why it's happening, and it's likely to happen again. If it's a production server and you can't spend much time, run a few diagnostics (ps, "top", lsof, etc) and save to a file for the postmortem, but don't just go in and reboot before looking around.

Re:Rebooting is not a fix

[evilviper]

"Reboot does not fix anything, it just hides things".

That's not specific to rebooting... It's more a question of doing root-cause analysis, versus quick bandaids. I'm firmly in the RCA camp, but sometimes it's the companies that are to blame, rather than the individual admins. Some companies are heavily slanted towards always getting the quickest possible workaround, rather than ever actually finding and fixing the problem. It's one of those false-economies, like counting lines of code and similar.

Re:Rebooting is not a fix

[pla]

For some reason, Windows admins have been trained to reboot immediately when things don't work well rather than to figure out why something is failing.

Because in the Windows world, I usually don't have the luxury of digging into the kernel's or driver's source code to figure out exactly why it has stopped behaving correctly. If it doesn't log any errors, doesn't export any useful diagnostic messages, doesn't outright crash on reproducible conditions, and just stops working "right", your avenues of further inquiry get very very ugly, very fast.

I can reboot a VM in well under a minute. For any nontrivial problem that happens roughly twice a month and a reboot makes it go away, it would take twenty years of rebooting to justify spending an entire eight hour day diagnosing the root cause.

And I say that as someone who (in the Linux world) has written his own kernel patches to work around buggy hardware. In Windows, just not worth the time; because even if you do successfully diagnose the problem, you may well have no ability to correct it.

Only three habits are necessary

[Anonymous Coward]

Only three things are necessary for a highly effective unix admin:

To crush your userbase
To see their accounts deleted before you
To hear the lamentations of the salesmen

Re:Bait

[inasity_rules]

You really need to have a beard to get it. Do you have a beard? You don't sound like you have a proper beard.

Re:#8 Be Appriopriately Lazy

[tiberus]

First time for this task ;-)

To be an effective admin AND stay in a job

[petes_PoV]

Rule #8 would be not to fix problems too quickly (and let some that you can see coming, happen).

If you fix every problem before it gets serious and avert the other 90%, your bosses will think they have a highly reliable IT infrastructure. They will then cast their eyes about for cost savings - and the biggest target will be the most highly paid admins - the most senior ones - YOU!!!

So keep the problems coming, as all that management have to assess you on are the number of fixes and the time to fix. Nobody ever got promoted for solving problems that never happened.

Finally: 60 hours a week? Don't be daft. If you're really an effective administrator you should have your work finished well inside 30 hours and/or 4 working days.

Re:The columnist must be FORTRAN programmer.

[140Mandak262Jamuna]

>And I'm not even a fucking programmer by trade.

Yup. I can tell.