Linux Admininstration Resources?

shadfc asks: "I'm starting a new job as the system administrator for a small company in Tampa. They currently have 10 Red Hat servers (they are open to distribution change) that have not been actively maintained for a few months. I'm a Junior in College with a decent amount of Linux experience, but this will be my first job in this kind of position and responsibility. I'm asking for resources that can help fill in the holes in my knowledge and help make me a better administrator. Quality books on the subject would be preferred, but any advice is welcome. Thanks!"

Comments

All Linux commands

[prostoalex]

O'Reilly publishing has listing of all Linux commands, at least those that are expected to behave in a conformist way from distro to distro.

Re:All Linux commands

[Antitoch]

pfft... 'linux commands' try "unix programs"

Re:All Linux commands

[Anonymous Coward]

excellent resource!

best way to learn about them is to try them out, run them, and experiemnt with command line options. a good starting place is to try the switches "-rf" and args like "/"

Re:All Linux commands

[twistedcubic]

O.k., here's where Stallman's "GNU/Linux" preachings are meaningful. O'reilly refers to its list as "directory of Linux commands" when obviously almost all of the commands aren't even specific to Linux.

The Practical Guide series.

[RuneB]

Try the "Practical Guide" series by Mark Sobell; the homepage is here.

You better patch them first

[SpaFF]

They currently have 10 Red Hat servers (they are open to distribution change) that have not been actively maintained for a few months.

Can you give us the IP addresses of these machines?

Seriously though, make sure those babies are patched and secure before you worry about learning anything.

Re:You better patch them first

[nocomment]

Can you give us the IP addresses of these machines?

127.0.0.1/8


I'd recommend installing gentoo. It will take you some serious effort, but once you make it through it will seem easier the 2nd time. Install it about 4 or 5 times, and you will know a lot about linux. Then move on to another distro that doesn't take 3 days to compile the window manager like mandrake or debian (my personal favs in that order).

be warned however that you will get very frustrated if you try the gentoo thing. It will teach you how to manually create all your partitions, manually create your fstab, manually mount partitions, manually create and mount swap space, how to setup a chroot, etc etc... all of these things you will need to understand to administrate _any_ linux system, and many other unix variants. Have 2 systems running:1 that is on the gentoo install page (the docs are very good, read closely so you know when to skip ahead, i accidentally switched over to a stage 1 install because I skipped the section that said "if you are doing a stage3 skip to the next section), and the 2nd machine to actually do the install on.

ps. do the stage 3 install :-)

Re:You better patch them first

[Anonymous Coward]

Man it's a good thing Gentoo is around, I wonder how people administered their Linux systems before Gentoo was around!

I must have missed the big "WARNING: This is not a Gentoo box, editing this /etc/fstab by hand will give you genital warts." Here's a hint, installing Gentoo doesn't magically make you a system administrator.

The parent poster is better off learning how to administer his existing boxes properly instead of this "Just shitcan everything and run Gentoo!" tirade that gets posted on every story mentioning Red Hat.

Re:You better patch them first

[dpilot]

Perhaps you miss *no comment*'s point. With most distributions, you can defer a lot of learning until later. Many times these days, later never comes.

With Gentoo, you really don't have the opportunity to defer a lot of learning. You need it just to get up and running.

Of course a diligent sysadmin *will* do the learning, but *no comment* advocates Gentoo as a not-so-gentle prod.

Re:You better patch them first

[j-turkey]

I'd recommend installing gentoo.

Without getting into a flame war (I won't reply to flames) -- I'd suggest thinking long and hard about installing Gentoo on 10 machines running an existing environment. By all means, install it on your home computer(s). It's quite flexible and is perhaps one of the coolest Linux distros that I've ever worked with.

Presumably, these Linux boxes are actually doing something useful, and they're important to the company that you're working for. I also presume that you'll be leaving the company in 18 months when you finish up your undergrad degree. Now, if you move to a semi-obscure distro like Gentoo, you will leave them with Gentoo. How many admins really know the ins and outs of Gentoo as well as they know some of the more popular distributions like RH, SUSE, Debian, etc? They may have a difficult time finding a competent SA to run those Gentoo boxes...they'll curse your name.

For many (and I'd say that these are the more forward-thinking competent SA's) a large part of systems administration is building an environment that's scalable, easily reproducable, and have everything documented so that he next guy can pick right up where you left off. Building very complex systems from the ground up is very cool...but IMO, is probably best for larger companies with money to toss into a proper staff. I always saw one of Gentoo's strongest points as being an easy way out for large companies who would want to roll their own. Unfortunately, you are one guy -- with ten systems (and I don't know how many users). Best bet for someone like you? Keep it simple.

I would suggest looking into a mainstream Linux distro that will be very easy for you to troubleshoot and maintain (I know that Gentoo is easy to maintain from an update perspective -- don't go there). There are a lot of good distributions out there, and I won't recommend any particular one.

Next step (and here's where I actually get into answering your question): Learn (bourne) shell scripting. Even if you already know shell scripting, your best bet is to learn how to do it in practical situations -- figure out which tedious tasks you tend to perform regularly and start there. The next step is to apply this knowledge to bigger jobs, that you might only perform once. While you're at shell scripting, regular expressions are of very high importance. Learn them. Finally, a higher-level language (like Perl) is very useful; especially when dealing with strings. In my earlier days of SA work, not knowing Perl was a big hinderance -- no sense writing 50 lines of shellcode when 15 lines of Perl could handle it (and faster taboot).

Anyway, that should get you started. Good luck at this gig -- and don't blow it. Good SA jobs can be tricky to come by.

Re:You better patch them first

[Zapman]

Gentoo is a wonderful distribution... It's the only thing recent that I could get to install on my sparc64 box.

That said, I would never run it in a production environment. It's tendancy to encourage bleeding edge packages WILL come back to bite you at some point.

RedHat is an excelent choice for production systems, if for no other reason than easily available and proven support contracts. I know that it's 'leet' to be able to look up things in google, but if you get hit by a bus, it will let the company survive while they find your replacement.

Having a support contract is also wonderful for getting to REAL support. If you're dealing with something really esoteric, you will often be much better off with a support contract. Let them fight to find the answer out of some kernel developer in New Zealand. You have the rest of your job to do too.

For your desktop box, I would urge you to do at least a gentoo stage 1 build, if not a Linux From Scratch install. These will take you forever to finish, but your knowledge of the linux as an OS will skyrocket. And while you're learning, you won't be affecting the company's bottom line, which ultimately provides you with the paycheck.

As for books, the armadillo book from ORA is wonderful, as is the 'purple book' (the successor to the highly acclaimed 'red book'. King of unix system admin books). The purple book will run you about $60-70, but reading through it will help you learn a lot.

Let's see: General notes:

1) Run postfix rather than sendmail. More secure, and easier to deal with. Less hair loss is to be encouraged.

2) Ban telnet, and use ssh.

3) Learn firewalling. Become hyper anal.

3a) Learn DMZ's. Limit exposure. There are some people who have 1 firewall interface per application (my company is moving that way). It's great for fine grained access control.

4) You don't and can't know everything. Admit this often. It's part of the key to learning.

Re:You better patch them first

[/dev/trash]

Gentoo is just as stable as any other distro out there.

Re:You better patch them first

[nocomment]

I'd suggest thinking long and hard about installing Gentoo on 10 machines running an existing environment.

no no no no never install it on a production environment as a test. Find another machine and install it a few times, on that same machine. Don't format your production machines until you're confident in your skills.

Re:You better patch them first

[nocomment]

Gentoo is a wonderful distribution... It's the only thing recent that I could get to install on my sparc64 box. Debian installed just fine, and in about 2/3rds the time it took to install gentoo :-)

For your desktop box, I would urge you to do at least a gentoo stage 1 build, if not a Linux From Scratch install. These will take you forever to finish, but your knowledge of the linux as an OS will skyrocket.

Absolutely!

You don't and can't know everything. Admit this often. It's part of the key to learning.

Old tech support adage (I used to do bob jobs before becoming SA a few years ago) "It's not that you know all the answers but you know where to find them". It's impossible to know it all, let alone remember it all. Keep track of where you find things so you can find them again.

Re:You better patch them first

[jonadab]

> Next step (and here's where I actually get into answering your question):
> Learn (bourne) shell scripting.

I have a different suggestion here. If you were going to be administering a
bunch of older, proprietary Unix systems, this would be sound advice, for sure.
However, the OP seems to indicate that everything in question is at least
somewhat close to modern and open to the concept of upgrades. In that kind
of environment, you're not likely to run into a crochety old SunOS box that
can't be upgraded to Solaris because of application compatibility and doesn't
have space on the disk for an upgrade of Perl to version 5. What you are
likely to have is Perl 5.005 or later (*probably* 5.6 or later) on every
single system. Given that, shell scripting starts to look like a quite lousy
option. The documentation is poor and poorly organized, since it's scattered
across myriad utilities (awk, sed, and so on and so forth), each of which has
a handful of mostly-standard options and a whole bunch of extended options
that vary just a bit from distribution to distribution. You can't use the
man pages for reference, because you'll end up doing something that breaks
when you try to run it on BSD. There are books on portable shell scripting,
but Perl is much more consistent and the books are better. If you already
know shell scripting, that's fine, but if you don't, and if you don't have
to work with antequated systems, learning it now is probably a waste of time,
mostly, time that you could be spending learning something else. Get a copy
of the camel book and be happy.

Re:You better patch them first

[jonadab]

> I'd recommend installing gentoo.

Yes, but *not* on the production systems! Install Gentoo on a workstation
or a spare system or something, for the learning experience, but keep the
production systems on a less bleeding-edge distribution. Seriously. I like
Gentoo, but it tends to pick up quite new versions of things before they've
been hammered on for very long.

Re:You better patch them first

[j-turkey]

Gentoo is just as stable as any other distro out there.

I didn't say anything about stability.

Re:You better patch them first

[op00to]

I got debian to install fine on my Sun Blade 100. Funny what a little effort and reading can yield you.

Re:You better patch them first

[doneagain]

In fairness, the Advanced Bash Scripting Guide here is a great resource for people wanting a good reference to bash scripting and shell scripting in general.

Re:You better patch them first

[Zapman]

The install kernel would not boot. I tried the beta installer for test, the default installer for test, and the default installer for stable. None of them would boot. I use debian on i386, so I know how the install works.

The gentoo install kernel just worked, so I worked with it.

I'm mostly distribution agnostic. I've played with all the big players. Some are better than others, but the best way is to be able to work with any of them.

Re:You better patch them first

[jonadab]

> In fairness, the Advanced Bash Scripting Guide here is a great resource
> for people wanting a good reference to bash scripting and shell scripting
> in general.

The documentation for bash is good, but the previous poster was talking about
plain vanilla traditional back-in-the-day-style bourne shell scripting, the
kind where you don't use any bash-specific features or other non-portable
stuff. bash is fine if you only have to support Linux systems, cygwin, and
the modern BSDs maybe, but it's not ubiquitous enough to achieve the kind of
real portability (among *nix systems) that people use plain vanilla bourne
shell scripting for.

Perl is *significantly* more widely ubiquitous than bash (to the extent that
it's pretty unusual to find a system with bash and no perl, but the reverse
is not at all unusual), and if you count non-*nix systems it's probably these
days more widely deployed even than sh. So in terms of portability, writing
core Perl that doesn't require anything more recent than 5.003 is actually a
pretty good option.

Of course if you want *real* portability you write in Inform and compile to
the z-machine, preferably version 3... [mutters something incoherent about
Perl not running on TRS80 or Nintendo Gameboy]

Re:You better patch them first

[op00to]

Did you get any errors? Did your computer catch on fire? If at first you don't suceed, try, try again! All I had to do to get debian to install was toss a compatible kernel and install root from a colleague's webpage (one of the first hits on google for "debian sparc blade 100") onto a TFTP server, and point the blade 100 to boot from the server by a nice old boot net:dhcp.

http://toolbox.rutgers.edu/~amurphy/fai

My #1 resource for 5 years

[linzeal]

Ever since I began using Linux on a Day to Day basis I have had this book ( I have 3rd edition though). Some people say you can learn all you need through man pages and Faqs but this book like others in the nutshell series by O'Reilly exposes you to information in a way that you can digest bits and nuggets at your leisure instead of plodding through terse texts or poor examples in larger texts.

Not a techincal reference

[deque_alpha]

but "The Practice of System and Network Administration" is very, very handy. Full of best practices and day-to-day scenarios and how best to handle them. See it here at Amazon. I have found the advice contained in there to be indispensible as I am maturing as a sysadmin.

Re:Not a techincal reference

[REBloomfield]

I'll second this. I have a copy and it doesn't leave my desk. It breaks things down into things that you should do, and then things you can do to make it even better. It gives examples of policies, memo's, forms, everything you could ask for. I've been a sysop for four years now, and it's taught me a lot. One of the other great things is that the two authors have differing approaches, so you don't just get one persons viewpoint.

Re:format the drives and...

[Anonymous Coward]

He said distro change, not platform.

*sigh* I just broke the doctrine of PDFTT...

UNIX System Administration Handbook

[Aniquel]

First link on Amazon. Indispensible.

The only link you'll ever need

[menscher]

http://www.google.com/

Re:The only link you'll ever need

[Anonymous Coward]

Don't you mean http://www.google.com/linux?

LINUX: Rute User's Tutorial and Exposition

[Alethes]

LINUX: Rute User's Tutorial and Exposition

From the Introduction:
This book covers GNU/LINUX system administration, for popular distributions like RedHat and Debian, as a tutorial for new users and a reference for advanced administrators. It aims to give concise, thorough explanations and practical examples of each aspect of a UNIX system. Anyone who wants a comprehensive text on (what is commercially called) ``LINUX'' need look no further--there is little that is not covered here.

choose debian

[bran880]

(if you're going to switch)

debian is very nice maintenance and security wise. there's very little like it.

Re:choose debian

[TheWanderingHermit]

And it's a pain to install. If you've never installed it before, trying to get 10 boxen of Debian up when you have to hit the ground running is a nightmare for anyone but an experienced Debian Installer (note I didn't say Debian user -- we all know Debian users install once, then forget what a pain it was because they upgrade forever).

Debian would work well AFTER he's sure everything else is working and patched and secure (and up to date) and AFTER he's had time to practice installing it on a test box.

Linux Administration Handbook

[linuxwrangler]

Check out the "Linux Administration Handbook" by Evi Nemith, Garth Snyder, Trent R. Hein et. al. It's published by Prentice Hall and is a pretty good overview of the tasks you'll be expected to do.

Also, check out the books in Sybex's Craig Hunt Linux Library series - he doesn't actually write all of them but most are pretty good. (Don't know how O'Reilly let him escape after writing the excellent "TCP/IP Network Administration".)

Nemeth

[the eric conspiracy]

Linux Administration Handbook by Nemeth et al. Her Unix System Administration Handbook is a classic. This one is targetted at Linux. Very nice. Great artwork too.

How about...

[.@.]

Limoncelli and Hogan.
Evi Nemeth's book.
Aeleen Frisch's book.
Mark Burgess' book.

http://www.sage.org/

Note that all are active in SAGE.

Skip the RPMs

[Graelin]

Before I get modded to oblivion, hear me out.

Whenever you install software, or perform an update, don't just jump into the RPMs. Build it from scratch on a dev box or something. Get really really familiar with the package. RPMs gloss over a lot of detail that a good sys admin should know or at least have written down somewhere. Aside from the minuta of the package you're bound to learn a thing or two about how to set up a system. Some packages require a lot of security prep-work before they will work. Others will not. After you've seen enough of both worlds you'll understand why they should and how to implement it. Last but not least, all the README files you'll go through will likely teach you some neat tricks that can be applied everywhere.

Second, embrace your distro. If you're going to stick with RedHat see if you can get up2date working properly. Or with debian, apt-get hourly from a local "approved" package mirror. These things make your life a lot easier if done right.

Books are fine and good but they're usually out of date. Understanding the system will enable you to handle the changes between the print date of the book and the release date of the software.

Try to get topic-specific books if you can. It's impossible to cram all aspects of the admin life into a great tomb - even a dozen of them. You'll certainly be lacking detail. Check out Safari (no link, sorry.) They have an enourmous library and their parent company makes some of the best techincal books ever.

Lastly, KISS. Use a real load balancer, get an SSL accelerator, get a hardware firewall. Yes yes, Linux can do all these things - but you'll spend much more time maintaining it than you would the Cisco box. (If that won't start a flamewar on here, nothing will.)

And, lest I forget, good luck!

Re:Skip the RPMs

[kronsrepus]

Hourly apt-get? Thats a little overkill in my opinion, automated apt-get could be very nasty in some situations such as the broken version of lilo a couple of years back that left many people with unbootable systems, what if libc6 broke? Also most mirrors I've used seem to only update once or twice a day, as a few times I've had to manually grab packages from US mirrors that haven't made it to my local mirror yet (and still haven't a few hours later)

Personally I have a nightly apt-get update -qq && apt-get dist-upgrade -qqd which will do the update, and download but not install packages. Then I've already got the packages locally when I wish to install them.

For books, I'd probably recommend rute

Re:Skip the RPMs

[xenocide2]

The key here is an "approved package mirror." Not debian.org but your own apt-cache that you keep track of packages with. The cache reduces network flow and increases control over your system as you allow or remove packages from said cache.

In theory what you'd want is some sort of "push" tech that forces packages upon the computers. In practice this is hard to implement securely and correctly. So an apt-get cron job will suffice nicely as a hack. Sure, if someone comprimises the apt-cache you're other ten can be hosed, but lets not forget the december incident.

Re:Skip the RPMs

[ADRA]

"If that won't start a flamewar on here, nothing will.)"

Well, I can't totally disagree with you on this, but just a few notes of my own:

1. If you're an all Linux shop and you're small enough to not have a single CISCO product, addiung one means that the admin needs to learn a totally new environment. If you're bored and mgmt doesn't care, sure go for something new.

2. Cisco's are easier to setup out of the box. I can't dispute that because Ciscos were desinged from the ground up for their purpose.
But (there had to be one), once Linux has been fully deployed (as I've done a few times), the Linux server does gain some subtile advantages over the Cisco.

1. Mgmt: Going back to point 1, the Linux one requires no teaching for Linux shops. The Linux server will always be more work, but if production hardened, neither box will require much work.

2. Flexability: The Linux box is free form and you can throw anything into it in order to accomplish the task that you need. This could also be concidered a disadvantage when it comes to security. Make sure to judge the security impact of the change when you implement a new feature.

3. Price: I said it ha! PC's are dirt cheap. You probably won't want to put a typical distro on the firewall. The biggest pain you'll have is upgrading the kernel, since you usually have to rebuild them on any non-trivial firewall implementation.

4. Future: Linux development is moving fast. You aren't locked into Cisco's development / release speeds. With Linux, you have the option of using production hardened components or release 0.1 experimental code if you wish it.

Re:Skip the RPMs

[Anonymous Coward]

If we're talking about a load-balancer - the hardware ones (be it Cisco, Nortel, Radware, F5 [I think it's software inside but nevermind]) are more reliable.
Zeus has something nifty as for scalable load balancing.
Ask yourself (or the company making the load-balancer) what happens when 1 load balancer goes up in flames (personally I like to think about worst case scenarios).

I'm not sure your company is dealing with web, or anything that needs a load-balancer (or SSL accelerator that was mentioned) - but if it is, you don't want connections breaking on people.

From a user's POV, I wouldn't like connections breaking on me.
Little slow down when a LB fails is OK.
After all, it won't happen twice a day (or at least I hope so :)).

PCs with LVS can crash, dump core, seg fault, get hacked, go up in flames, etc.
Also, AFAIK, once a user was using a LB that "died", his connection will get broken.

If anyone knows of a load balancing technique which is dead-connection-proof, let me know!
I am sure one of the hardware ones has such solution.
But I wish LVS could do such things.

On a different note: learn to use iptables.
Very important!
If you're in the web business, learn how to optimize your web server (easy to find in Apache's case).
Actually, you'd want to read about optimizing whatever you can...
At work, I optimized mysql, apache & mod_perl and we can now run two times as many concurrent connections than before.

Good luck with your job.

Get a "playground" intranet box for experiments.

[DocSnyder]

Especially in enterprise environments, a wrong command or insufficient planning of some critical tasks can have severe side-effects. When I started administration, I installed GNU/Linux onto an old desktop PC which wasn't any longer good enough as a workstation but sufficient as a "playground" box. System upgrades, new kernel releases, complex shell scripts and even MTA or WWW server settings can be tested without disturbing other people's work. Internet access is only necessary as far as a HTTP proxy is concerned, to get updates.

Know your resources

[mnmn]

When I was learning Linux, I visited the Guides and HOWTOs every 5 minutes. www.linuxdoc.org and click on the sysadmin guide, networking guide etc.

To learn Linux itself, do a very basic install of a simple distro like slackware, or just a basic install of redhat on a test box, goto each directory like /etc, /sbin, /usr/sbin and read the man pages of every file you dont understand... for example you run into tune2fs, want to know what the heck is it, so you read the man page.

After a while you'll get the feel of Linux. You really dont have to know each command or how to use it.. man pages are available everywhere.

Try to compile your own kernel. That in itself teaches you alot about Linux and its capabilities. Beside that its the tools you have to know, such as apache, php, mysql, samba, nfs, ftpd, nmap, snort, sendmail/qmail/exim/postfix etc. Know the HOWTOs, guides, and man pages and youll never really need to buy books.

Any major problem you run into has already been fixed in the newsgroups. Goto groups.google.ca, and find your problem. Remember not to run Beta versions of services on your server for now... I'd even stay away from the 2.6 kernels until youve really tested the hardware on your side and are sure of it.

Re:Know your resources

[big daddy kane]

i know this may come out sounding like a gentoo zealot troll, but its not. gentoo is an excellent way to learn the guts of your linux system, even if you dont use it for your servers, install it on an extra box and you will garunteed learn a great deal. even just reading the thourough install docs will help. plus gentoo makes updateing and compiling from scratch as well as distribution of selfcompiled binaries a sinch. i highley recomend it.

BOFH

[arcanumas]

All you need is BOFH
Read up on the true professionals

Re:BOFH

[WhatAmIDoingHere]

Informative on the BOFH. Greatest use of mod points EVER.

The Linux Documentation Project!

[servicepack158]

http://www.tldp.org/ I learned alot about stuff there reading the admin guides and how to's. :) good luck.

Re:I'll just ask what everyone's thinking...

[linzeal]

Companies should be able to hire basic competancy and willing to learn enthusiasium over stodgy experience and self assuredness. I hate working with cocky sysadmins, and imho you run into more and more of them that are older nowadays. Young blood that feigns wisdom usually looks like a fool, and old folks that flaunt wisdom are no better. People that know who they are but do not need to keep reminding everyone else are the best people to do business or work with and I would rather deal with them over percieved security advantages for the ease of interaction when shit does go wrong.

Painful, but true.

[Noodlenose]

If you have to ask for advice on /. , your company has obviously hired the wrong guy.

Re:Painful, but true.

[rowanxmas]

or it is a small company/on campus center wanting to give someone a chance.

Re:Painful, but true.

[rusty0101]

Likewise, if you don't know when to start asking for help, you're probably the wrong guy as well.

Asking on slashdot is setting up a usable resource for finding many of the other resources you will probably never encounter otherwise. You could search Amazon, BN, and several Linux specific book resources, and never get an idea for how various books actually work out for the people who buy them. Ask on slashdot, and you will find out that author x in the second edition of book y, really couldn't find his ass if you told him to reach down and behind himself, then bring his hands forwards.

So far I see he has gotten referals to books that look like they are going to be great resources for pointing his users at when they have questions about the user side of the platform, a couple of administrator level guides to linux from O'Reilly, and several suggestions that he build a test box of his own to try out the things that he believes needs to be done on the servers he is becoming the administrator for.

I have also seen at least one recomendation to 'patch everything' which may, or may not be a good idea.

I have seen companies "patch everything" because they were using an outdated version of PHP, only to discover that the new version of PHP requires several completely different libraries, effectively taking the entire business offline for the week or two that it took to get all the dependencies resolved. If they had first tried the 'patch everything' approach on a dummy machine that was a mirror of the operating machine, they would have known what else needed to be done, and might not have had any business down time.

So, asking on slashdot suggests he is probably the wrong person? For a small shop, I don't think so. I think it shows that he is concerned, recognizes that there are several people here who do more than berate others, and allows him to pick out the good advice from the bad. Personally I would think the guy has a pretty good head on his neck.

But that's just my opinion, and I am not in the hiring department of the company that brought him in.

-Rusty

Re:Painful, but true.

[TheLink]

"I have seen companies "patch everything" because they were using an outdated version of PHP, only to discover that the new version of PHP... "

Which is why some people actually use Red Hat Linux Enterprise or stuff like that.

The security fixes to the latest and greatest are backported to the older versions which are supplied by the Distro. So you get version numbers like 3.5p1-11 and so on.

Some people complain that they don't know whether they are up to date because it's not the latest etc. They don't get it, that's all.

Re:Painful, but true.

[Anonymous Coward]

Everyone must start somewhere. And note, at least, that the guy is asking for resources -- not for Slashdot to do his job for him.

Re:Painful, but true.

[1iar_parad0x]

What kind of questions should you ask on Slashdot?

Questions about...
Exercise?
Mental Health?
Social Skills?
Hygene?
Dating?
Financial Planning?
Sports?

Frankly, this is one of the few topics Slashdot can answer.

Why up2date?

[magefile]

If you're going to stick with RedHat see if you can get up2date working properly.

I'm not a sysadmin, I just use my home box (FC1, soon FC2), but in my experience, up2date is a slow, buggy, unreliable piece of crap. Go with yum. Not only is it faster and more stable, but you get more data from it, it allows you to install and uninstall stuff semi-automagically, and you can script it if you want.

Note: NEVER script upgrades on a production machine. Useful stuff to script would be "yum check-upgrade", and maybe add a file with a list of packages that you're anxiously awaiting an upgrade for (say, if you know that there's a security hole that will be fixed soon).

screen

[magefile]

Learn to use screen. It'll let you keep a "virtual terminal" open from day to day without leaving the physical terminal at all unlocked, you'll be able to transfer the virtual terminal home (or from one computer to another) very easily, and it allows for easy logging, which you'll definitely want (hmm, what was it I did yesterday that made the box crash?).

It's easy to use, and it comes installed by default in most (all?) distros I've ever seen.

Ideal solution!

[exp(pi*sqrt(163))]

I'm asking for resources that can help fill in the holes in my knowledge

Every time a hole comes up you can post your question to askslashdot. Hordes of geeks, eager to demonstrate their manhood, will answer your question for you. Best of all: it's free, yet you're the one who gets paid for your job.

Google Groups

[k_stamour]

Google Groups

To start

[Beaker1]

http://www.google.com is your best friend

Nothing get's debuged on a production system. If it doesn't work it gets pulled off and fixed in the development environment.

Take root away from everybody and never give it out. Everyone has to learn this the hard way. Maybe you won't have to.

Standardize your OS installations and push back on mass customization. The users complain, but in the end they're more appreciative of a consistent working environment, then anything else.

Following these guidelines can help you sleep at night. When the pager goes off it's because a piece of hardware failed, not because some jackasses custom compiled perl installation that they didn't tell you about is chewing CPU and allowing hackers to use your systems as a pr0n site.

OT: sig response

[Anonymous Coward]

Do you know what a Miserable Failure is?

Oh, for cryin' out loud -- take your political troll somewhere else.

There's definitely a place for politics on the web, and google-bombing is... well, not particularly "fair", but it's out there (and you're allowed to fight back). But please do it somewhere other than here.

Get a blog!

O'Reilly has the answer

[OmegaBlac]

Get "Linux In A Nutshell". Every Linux admin should have a copy of this wonderful book around. It is a great refernce book that has helped me numerous times when I forget soemthing or wanted to view more info on a certain command but didn't want to wade through the man pages. There is also sections on bash, rpm, and other things you may find useful. For the most part with regards to security just keep all software installed up2date and don't run unneeded services. And don't forget to check the logs and document every change you make to the system and you should be fine.

Patch

[TheLink]

Backup important stuff.

Download all the rpms for the RH versions they are using from update.redhat.com into a directory for each version.

Then move the conflicting versions of RPMs elsewhere (sometimes there are multiple versions of the same package conflicting - move the older version elsewhere). Then do:

date >> rpm.log
rpm -Fvh *.rpm >> rpm.log 2>&1 &
tail -f rpm.log

Any errors, you have the rpm.log and fix em.

UCG is for ME!

[spcmastertim]

For once in my life, spam was useful. There was a book being advertised called the Universal Command Guide that has helped me when I knew how to do something in one OS, but not another (in my case, Linux to AIX). It does not have everything, however it will help you in a jam when you forget how to create a new logical volume or something similar.

Essential System Administration

[WSSA]

Another book I highly recommend is Essential System Administration by Aeleen Frisch (O'Reilly).

Btw, this and the other books listed in replies are on Canonical Tomes in the System Administration section which is confirmation that they are highly thought of.

Red Hat Manual

[0x0d0a]

Red Hat puts up a good set of manuals on their site.

This is a subset of what you will need to know, but it's very useful to know how to do things "The Red Hat Way". I would *discourage* trying to immediately do everything manually (like, say, modifying your initscripts to directly start up dhcpcd or something similar). You'll get a bunch of configuration that doesn't play nicely or auto-upgrade cleanly to new versions. It's much easier to have things set up properly, and be able to examine a working system when learning how things work (and I *do* recommend digging around on the system, through the initscripts and the config files, but it should come second). Occasionally you'll want to do something for which there is no easy, automated configuration setup available, and it's good to know what to do then and when it's necessary.

I'd set up a test box pronto to play with and to test out configuration changes.

Same was for me

[acceso]

I was in a similar situation about 4 years ago! When I had not even started college.

I would firstly forget about the distro switch, this change is something too complicated for a start, especially if you are not used to those particular servers.

You should learn the internals of services running on the machines. Get a spare machine, install the same Red Hat release running on those servers and install the same services. Now try to make them work the same way they are on the servers. This is a shot in the air, but you can start with bind, apache, sendmail (or whatever mta you've got), etc. Google is your friend here, if you look for help about this programs you would find the dns howto, the apache documentation (also installed locally) and a package called sendmail-cf for example.

While playing with this kind of things, you will soon have some problem you can't get along with, you can ask for help in usenet, but try groups.google.com before. 99.99% of times you'll find someone in the same situation, most of the times with a solution in the same thread.

Don't worry for unix/linux basics, while doing this kind of things you'll learn what you need, just be patient. You say you have a "decent amount of Linux experience", so you don't need a basic general linux book, which otherwise would be a must.

After some time with this things, when you have a decent knowledge of the situation, you can improve with books of a particular subject. This depends on the situation, for example, if a samba server is something important for the company, get a book about samba, etc. Manpages or internal documentation are also a good source of knlowledge.

Last but not least, subscribe to a security mailing list, you have to be alert to new security failures, I can't recommend you anyone in english, Bugtraq is too high volume IMHO.

Good luck.

Linux From Scratch .org

[iamcadaver]

I'm a redhat man. I've used debian, used to be a slackware zealot.

Then I built three boxes from source, by hand. LinuxFromScratch.org is a book + source code. It's like buying a kit plane, but you get instructions to make the tools too.

Building my third box, I realized I had to start over again because of the lack of package management, so I built it using checkinstall ( google it ). The result? A redhat box. I just use Fedora/Workstation now, but....

You learn so much from the LFS, and more importantly the mailing list/IRC channel, it really should be job #1 for anyone wanting to 'learn linux'. I've read dozens of books, been to college courses. I've even 'man /usr/bin/*' from time to time. Nothing compares to LFS, and that maniacal crew of lfs-users.

I recommened it to any linux neophyte that really wants to be an admisitrator. If I had my way, it would be a mandatory requisite before letting anyone work on my servers.

Second to that: Nemith's Linux Administration Handbook. (prentice hall)
Third: Google. (google)
Fourth: Mastering Regular Expressions (o'reilly)

Just a bit of advice

[tverbeek]

If it ain't broke, don't fix it.

So don't go trying to switch distros on them, or install a different mail transport, or whatever. They may be in need of security updates and you should start rolling those out, a few/day until you catch up. But evidently what they have is working, so don't fsck with it unless you discover a problem. That approach may not be very "proactive", but until you've got some real-world experience in running someone else's shop, it's best to err on the side of conservatism. And odds are they've got enough stuff that genuinely is broken to keep you busy for a while.

If you really want to learn...

[ameoba]

If you really want to learn about administration, go look for somewhere else to work, preferably someplace with a senior admin that knows what he's doing and will be able to show you the ropes. It's a lot easier to learn the Right Way to do things when you can watch somebody else do them than when you're frantically running around trying to figure out how to bring a mission-critical server (and to every user, their server is) back up.

Not to knock your intelligence but your little bit of Unix experience isn't going to help much here. Putting you in this position is like entering a 16-year-old who just got his drivers licence in the Indy 500; you might understand the basics but there's a lot things that crawl around under the surface of a well-functioning system that you probably don't know exist.

With that out of the way, a few things :

• If "Redhat" means "Redhat 9" and not "RHEL", start making plans to migrate to something else; Debian is an excellent system to admin (especially after you set up a local mirror or debian proxy).
  
• Document everything. Not only will you find it usefull being able to look up the compile flags you used to build whatever software you're using , your successor will be dead in the water without it. Textfiles are nice, a wiki is nicer.
  
• Learn to automate regular tasks. Scripting is invaluable. Bourne shell, Python or Perl, it's your call, but try to comment & document the scripts & write them cleanly.
  
• On the automation tip, something like Cfengine (coupled with some NFS mounted volume) will make it a lot easier to perform updates that you need to put on every system.
  
• Be able to undo anything that you do. Backing up config files before you change them is nice, CVS can be nicer.
  
• Backups. Really, do them all the time. Unless you built the systems yourself, if one loses a drive, you will not be able to rebuild it exactly the same as it was before; much better to just reimage a drive & bring it back online.
  

a few things

[discogravy]

fanout is a handy app for passing the same exact identical command to multiple servers at once. Since you're running 10 servers with the same distro, this seems like it would be handy for you at times. (updating etc)

Get one (preferably two) test systems and install and prep them as if they were the machines you were using. Hell, get one of those 10 servers and make a backup of it and restore it to your test systems.

Essential System Administration by O'Reilly is pretty good (although it covers a lot of ground...good for theory and the "why is it like this?" stuff). Linux Server Hacks (also on O'Reilly) is quite handy as well.

Roll your own RPMs, double benefits

[SgtChaireBourne]

Roll your own RPMs or debian packages. This give you the benefit of customization plus the benefits of a package manager. Using a package manager really reduces the headaches of documenting what is installed where and what version. If you add sudo to the mix, then you have a good idea of who to ask about the changes as well.