Linux Admininstration Resources?

shadfc asks: "I'm starting a new job as the system administrator for a small company in Tampa. They currently have 10 Red Hat servers (they are open to distribution change) that have not been actively maintained for a few months. I'm a Junior in College with a decent amount of Linux experience, but this will be my first job in this kind of position and responsibility. I'm asking for resources that can help fill in the holes in my knowledge and help make me a better administrator. Quality books on the subject would be preferred, but any advice is welcome. Thanks!"

Comments

All Linux commands

[prostoalex]

O'Reilly publishing has listing of all Linux commands, at least those that are expected to behave in a conformist way from distro to distro.

The Practical Guide series.

[RuneB]

Try the "Practical Guide" series by Mark Sobell; the homepage is here.

You better patch them first

[SpaFF]

They currently have 10 Red Hat servers (they are open to distribution change) that have not been actively maintained for a few months.

Can you give us the IP addresses of these machines?

Seriously though, make sure those babies are patched and secure before you worry about learning anything.

Re:You better patch them first

[dpilot]

Perhaps you miss *no comment*'s point. With most distributions, you can defer a lot of learning until later. Many times these days, later never comes.

With Gentoo, you really don't have the opportunity to defer a lot of learning. You need it just to get up and running.

Of course a diligent sysadmin *will* do the learning, but *no comment* advocates Gentoo as a not-so-gentle prod.

Re:You better patch them first

[j-turkey]

I'd recommend installing gentoo.

Without getting into a flame war (I won't reply to flames) -- I'd suggest thinking long and hard about installing Gentoo on 10 machines running an existing environment. By all means, install it on your home computer(s). It's quite flexible and is perhaps one of the coolest Linux distros that I've ever worked with.

Presumably, these Linux boxes are actually doing something useful, and they're important to the company that you're working for. I also presume that you'll be leaving the company in 18 months when you finish up your undergrad degree. Now, if you move to a semi-obscure distro like Gentoo, you will leave them with Gentoo. How many admins really know the ins and outs of Gentoo as well as they know some of the more popular distributions like RH, SUSE, Debian, etc? They may have a difficult time finding a competent SA to run those Gentoo boxes...they'll curse your name.

For many (and I'd say that these are the more forward-thinking competent SA's) a large part of systems administration is building an environment that's scalable, easily reproducable, and have everything documented so that he next guy can pick right up where you left off. Building very complex systems from the ground up is very cool...but IMO, is probably best for larger companies with money to toss into a proper staff. I always saw one of Gentoo's strongest points as being an easy way out for large companies who would want to roll their own. Unfortunately, you are one guy -- with ten systems (and I don't know how many users). Best bet for someone like you? Keep it simple.

I would suggest looking into a mainstream Linux distro that will be very easy for you to troubleshoot and maintain (I know that Gentoo is easy to maintain from an update perspective -- don't go there). There are a lot of good distributions out there, and I won't recommend any particular one.

Next step (and here's where I actually get into answering your question): Learn (bourne) shell scripting. Even if you already know shell scripting, your best bet is to learn how to do it in practical situations -- figure out which tedious tasks you tend to perform regularly and start there. The next step is to apply this knowledge to bigger jobs, that you might only perform once. While you're at shell scripting, regular expressions are of very high importance. Learn them. Finally, a higher-level language (like Perl) is very useful; especially when dealing with strings. In my earlier days of SA work, not knowing Perl was a big hinderance -- no sense writing 50 lines of shellcode when 15 lines of Perl could handle it (and faster taboot).

Anyway, that should get you started. Good luck at this gig -- and don't blow it. Good SA jobs can be tricky to come by.

Re:You better patch them first

[Zapman]

Gentoo is a wonderful distribution... It's the only thing recent that I could get to install on my sparc64 box.

That said, I would never run it in a production environment. It's tendancy to encourage bleeding edge packages WILL come back to bite you at some point.

RedHat is an excelent choice for production systems, if for no other reason than easily available and proven support contracts. I know that it's 'leet' to be able to look up things in google, but if you get hit by a bus, it will let the company survive while they find your replacement.

Having a support contract is also wonderful for getting to REAL support. If you're dealing with something really esoteric, you will often be much better off with a support contract. Let them fight to find the answer out of some kernel developer in New Zealand. You have the rest of your job to do too.

For your desktop box, I would urge you to do at least a gentoo stage 1 build, if not a Linux From Scratch install. These will take you forever to finish, but your knowledge of the linux as an OS will skyrocket. And while you're learning, you won't be affecting the company's bottom line, which ultimately provides you with the paycheck.

As for books, the armadillo book from ORA is wonderful, as is the 'purple book' (the successor to the highly acclaimed 'red book'. King of unix system admin books). The purple book will run you about $60-70, but reading through it will help you learn a lot.

Let's see: General notes:

1) Run postfix rather than sendmail. More secure, and easier to deal with. Less hair loss is to be encouraged.

2) Ban telnet, and use ssh.

3) Learn firewalling. Become hyper anal.

3a) Learn DMZ's. Limit exposure. There are some people who have 1 firewall interface per application (my company is moving that way). It's great for fine grained access control.

4) You don't and can't know everything. Admit this often. It's part of the key to learning.

Re:You better patch them first

[nocomment]

I'd suggest thinking long and hard about installing Gentoo on 10 machines running an existing environment.

no no no no never install it on a production environment as a test. Find another machine and install it a few times, on that same machine. Don't format your production machines until you're confident in your skills.

Re:You better patch them first

[nocomment]

Gentoo is a wonderful distribution... It's the only thing recent that I could get to install on my sparc64 box. Debian installed just fine, and in about 2/3rds the time it took to install gentoo :-)

For your desktop box, I would urge you to do at least a gentoo stage 1 build, if not a Linux From Scratch install. These will take you forever to finish, but your knowledge of the linux as an OS will skyrocket.

Absolutely!

You don't and can't know everything. Admit this often. It's part of the key to learning.

Old tech support adage (I used to do bob jobs before becoming SA a few years ago) "It's not that you know all the answers but you know where to find them". It's impossible to know it all, let alone remember it all. Keep track of where you find things so you can find them again.

My #1 resource for 5 years

[linzeal]

Ever since I began using Linux on a Day to Day basis I have had this book ( I have 3rd edition though). Some people say you can learn all you need through man pages and Faqs but this book like others in the nutshell series by O'Reilly exposes you to information in a way that you can digest bits and nuggets at your leisure instead of plodding through terse texts or poor examples in larger texts.

Not a techincal reference

[deque_alpha]

but "The Practice of System and Network Administration" is very, very handy. Full of best practices and day-to-day scenarios and how best to handle them. See it here at Amazon. I have found the advice contained in there to be indispensible as I am maturing as a sysadmin.

UNIX System Administration Handbook

[Aniquel]

First link on Amazon. Indispensible.

LINUX: Rute User's Tutorial and Exposition

[Alethes]

LINUX: Rute User's Tutorial and Exposition

From the Introduction:
This book covers GNU/LINUX system administration, for popular distributions like RedHat and Debian, as a tutorial for new users and a reference for advanced administrators. It aims to give concise, thorough explanations and practical examples of each aspect of a UNIX system. Anyone who wants a comprehensive text on (what is commercially called) ``LINUX'' need look no further--there is little that is not covered here.

Linux Administration Handbook

[linuxwrangler]

Check out the "Linux Administration Handbook" by Evi Nemith, Garth Snyder, Trent R. Hein et. al. It's published by Prentice Hall and is a pretty good overview of the tasks you'll be expected to do.

Also, check out the books in Sybex's Craig Hunt Linux Library series - he doesn't actually write all of them but most are pretty good. (Don't know how O'Reilly let him escape after writing the excellent "TCP/IP Network Administration".)

Nemeth

[the eric conspiracy]

Linux Administration Handbook by Nemeth et al. Her Unix System Administration Handbook is a classic. This one is targetted at Linux. Very nice. Great artwork too.

How about...

[.@.]

Limoncelli and Hogan.
Evi Nemeth's book.
Aeleen Frisch's book.
Mark Burgess' book.

http://www.sage.org/

Note that all are active in SAGE.

Skip the RPMs

[Graelin]

Before I get modded to oblivion, hear me out.

Whenever you install software, or perform an update, don't just jump into the RPMs. Build it from scratch on a dev box or something. Get really really familiar with the package. RPMs gloss over a lot of detail that a good sys admin should know or at least have written down somewhere. Aside from the minuta of the package you're bound to learn a thing or two about how to set up a system. Some packages require a lot of security prep-work before they will work. Others will not. After you've seen enough of both worlds you'll understand why they should and how to implement it. Last but not least, all the README files you'll go through will likely teach you some neat tricks that can be applied everywhere.

Second, embrace your distro. If you're going to stick with RedHat see if you can get up2date working properly. Or with debian, apt-get hourly from a local "approved" package mirror. These things make your life a lot easier if done right.

Books are fine and good but they're usually out of date. Understanding the system will enable you to handle the changes between the print date of the book and the release date of the software.

Try to get topic-specific books if you can. It's impossible to cram all aspects of the admin life into a great tomb - even a dozen of them. You'll certainly be lacking detail. Check out Safari (no link, sorry.) They have an enourmous library and their parent company makes some of the best techincal books ever.

Lastly, KISS. Use a real load balancer, get an SSL accelerator, get a hardware firewall. Yes yes, Linux can do all these things - but you'll spend much more time maintaining it than you would the Cisco box. (If that won't start a flamewar on here, nothing will.)

And, lest I forget, good luck!

Get a "playground" intranet box for experiments.

[DocSnyder]

Especially in enterprise environments, a wrong command or insufficient planning of some critical tasks can have severe side-effects. When I started administration, I installed GNU/Linux onto an old desktop PC which wasn't any longer good enough as a workstation but sufficient as a "playground" box. System upgrades, new kernel releases, complex shell scripts and even MTA or WWW server settings can be tested without disturbing other people's work. Internet access is only necessary as far as a HTTP proxy is concerned, to get updates.

Know your resources

[mnmn]

When I was learning Linux, I visited the Guides and HOWTOs every 5 minutes. www.linuxdoc.org and click on the sysadmin guide, networking guide etc.

To learn Linux itself, do a very basic install of a simple distro like slackware, or just a basic install of redhat on a test box, goto each directory like /etc, /sbin, /usr/sbin and read the man pages of every file you dont understand... for example you run into tune2fs, want to know what the heck is it, so you read the man page.

After a while you'll get the feel of Linux. You really dont have to know each command or how to use it.. man pages are available everywhere.

Try to compile your own kernel. That in itself teaches you alot about Linux and its capabilities. Beside that its the tools you have to know, such as apache, php, mysql, samba, nfs, ftpd, nmap, snort, sendmail/qmail/exim/postfix etc. Know the HOWTOs, guides, and man pages and youll never really need to buy books.

Any major problem you run into has already been fixed in the newsgroups. Goto groups.google.ca, and find your problem. Remember not to run Beta versions of services on your server for now... I'd even stay away from the 2.6 kernels until youve really tested the hardware on your side and are sure of it.

BOFH

[arcanumas]

All you need is BOFH
Read up on the true professionals

Re:I'll just ask what everyone's thinking...

[linzeal]

Companies should be able to hire basic competancy and willing to learn enthusiasium over stodgy experience and self assuredness. I hate working with cocky sysadmins, and imho you run into more and more of them that are older nowadays. Young blood that feigns wisdom usually looks like a fool, and old folks that flaunt wisdom are no better. People that know who they are but do not need to keep reminding everyone else are the best people to do business or work with and I would rather deal with them over percieved security advantages for the ease of interaction when shit does go wrong.

Why up2date?

[magefile]

If you're going to stick with RedHat see if you can get up2date working properly.

I'm not a sysadmin, I just use my home box (FC1, soon FC2), but in my experience, up2date is a slow, buggy, unreliable piece of crap. Go with yum. Not only is it faster and more stable, but you get more data from it, it allows you to install and uninstall stuff semi-automagically, and you can script it if you want.

Note: NEVER script upgrades on a production machine. Useful stuff to script would be "yum check-upgrade", and maybe add a file with a list of packages that you're anxiously awaiting an upgrade for (say, if you know that there's a security hole that will be fixed soon).

screen

[magefile]

Learn to use screen. It'll let you keep a "virtual terminal" open from day to day without leaving the physical terminal at all unlocked, you'll be able to transfer the virtual terminal home (or from one computer to another) very easily, and it allows for easy logging, which you'll definitely want (hmm, what was it I did yesterday that made the box crash?).

It's easy to use, and it comes installed by default in most (all?) distros I've ever seen.

Ideal solution!

[exp(pi*sqrt(163))]

I'm asking for resources that can help fill in the holes in my knowledge

Every time a hole comes up you can post your question to askslashdot. Hordes of geeks, eager to demonstrate their manhood, will answer your question for you. Best of all: it's free, yet you're the one who gets paid for your job.

Google Groups

[k_stamour]

Google Groups

Re:Painful, but true.

[rusty0101]

Likewise, if you don't know when to start asking for help, you're probably the wrong guy as well.

Asking on slashdot is setting up a usable resource for finding many of the other resources you will probably never encounter otherwise. You could search Amazon, BN, and several Linux specific book resources, and never get an idea for how various books actually work out for the people who buy them. Ask on slashdot, and you will find out that author x in the second edition of book y, really couldn't find his ass if you told him to reach down and behind himself, then bring his hands forwards.

So far I see he has gotten referals to books that look like they are going to be great resources for pointing his users at when they have questions about the user side of the platform, a couple of administrator level guides to linux from O'Reilly, and several suggestions that he build a test box of his own to try out the things that he believes needs to be done on the servers he is becoming the administrator for.

I have also seen at least one recomendation to 'patch everything' which may, or may not be a good idea.

I have seen companies "patch everything" because they were using an outdated version of PHP, only to discover that the new version of PHP requires several completely different libraries, effectively taking the entire business offline for the week or two that it took to get all the dependencies resolved. If they had first tried the 'patch everything' approach on a dummy machine that was a mirror of the operating machine, they would have known what else needed to be done, and might not have had any business down time.

So, asking on slashdot suggests he is probably the wrong person? For a small shop, I don't think so. I think it shows that he is concerned, recognizes that there are several people here who do more than berate others, and allows him to pick out the good advice from the bad. Personally I would think the guy has a pretty good head on his neck.

But that's just my opinion, and I am not in the hiring department of the company that brought him in.

-Rusty

To start

[Beaker1]

http://www.google.com is your best friend

Nothing get's debuged on a production system. If it doesn't work it gets pulled off and fixed in the development environment.

Take root away from everybody and never give it out. Everyone has to learn this the hard way. Maybe you won't have to.

Standardize your OS installations and push back on mass customization. The users complain, but in the end they're more appreciative of a consistent working environment, then anything else.

Following these guidelines can help you sleep at night. When the pager goes off it's because a piece of hardware failed, not because some jackasses custom compiled perl installation that they didn't tell you about is chewing CPU and allowing hackers to use your systems as a pr0n site.

Essential System Administration

[WSSA]

Another book I highly recommend is Essential System Administration by Aeleen Frisch (O'Reilly).

Btw, this and the other books listed in replies are on Canonical Tomes in the System Administration section which is confirmation that they are highly thought of.