Server Optimization For Newbies?

supaneko writes "I recently took a new job as a network and server administration for a small IT company. I am absolutely shocked at how much is taking place within this company that I have little to no experience with. To help bolster my experience, I purchased a used server to use for hands-on training and practice. My ultimate goal is to have a complete, secure LAMP server available to the public running CentOS. I have been browsing the Net for various guides and tips on setup, optimization, security, and maintenance, but nothing I've found really gives me a hands-on approach to the topics I want to learn about. When you all started out, what route did you take to pick up the server setup and maintenance skills you have now? Is there anything in particular that you would recommend to someone who has excellent skills with consumer PCs and servers but is a total newbie to corporate and enterprise networking and servers?"

Comments

Virtualization

[bigtallmofo]

Learn about virtualization. Take your pick of free offerings: ESXi and Virtual Server from VMWare, Xen, Virtual Server from Microsoft, etc.

Using virtual servers that are hosted on your new physical server will allow you to set up any kind of operating system you want and any applications on that operating system again and again and again with no fear of messing anything important up. Also, you can run (depending on memory) multiple operating systems side by side.

From there, you can start diving into learning all the operating system, application server, database server, etc minutia you like!

Oh, and don't forget learning about P2V. That will allow you to do all kinds of "what if" scenarios without affecting real servers.

Re:Virtualization

[Glonoinha]

Spend some time playing with vmware - I think you will be pleasantly surprised with just how close it is to running on the bare metal.

The only thing I don't use virtualization for is playing games that rely on frames per second - other than that, I honestly doubt you could tell the difference (and funny thing is - some things run FASTER - backup and recovery of the entire machine is as simple as copying some files from one hard drive (your backup set of vm files) to another. I can have a complete restore in about 5 minutes, and I can dupe a machine in about 6 minutes.)

Re:Virtualization

[atraintocry]

Emulation is more useful for playing older games, since many of the old consoles and computers ran on Z80s or 68000s. A very crude way of putting it would be to say that, while emulation completely recreates the old hardware in code, virtualization (in this context) simply runs the code in a sandboxed portion of memory.

Re:Virtualization

[BitZtream]

This is a great idea.

I'd like to add that your two basic options for learning to optimize are: hiring someone that already knows it and you can learn from and time.

The first is obvious as to how it works, though it may be difficult for you to find someone to learn from since you lack the experience to know what you need to learn at this point. Keep in mind though, the most experienced admins have not seen EVERYTHING that can be a problem, so they too are going to be in the same position you are on occassion. You can still learn from them in that situation by watching how they go about finding a solution.

The second is pretty much a brute force method, and the way most of the IT industry learns it. You'll simply get better over time as you gain experience. Occasionally you'll have a problem that will require you to figure out the solution sooner than you'd like, especially if your business does well and the servers become loaded sooner rather than later.

I'm all for setting up your server to be as optimium as possible from the start, but that also has its problems. Most of the time when you start you don't actually know what you need to optimize. Sometimes you do, like a SPAM type company needs mail servers that can handle large volumes of traffic and deal with large queues for sites that don't respond on the first try. 10 years ago, pretty much every site was going to accept your mail on the first try, now, due to greylisting for instance, many sites outright reject everyone on the first attempt. You could at best have built for that when you started just out of luck (or perhaps you have great natural insight ;). But like the most of us, you wouldn't have predicted that you'd need to change your configuration later to deal with the new sending delays.

I'm currently in the process of rewriting our companies core service engine, fortunately I have a good idea of where the load and performance issues are based on the current system and I've planned in ways to deal with those situations. But in the process, I've also subtly changed the service and the users are no longer going to use it the EXACT same way they did previously. We've add new features, removed old ones that were hardly used or can be done differently, ect. As such I can only make an educated guess at how to setup the load balancing, web farms and database servers. I won't get the perfect setup on the first try, and even if I did, it would for all intents and purposes be just luck.

Read a lot about the software you are using. Get on the users or developers mailing lists. You absolutely want to be on the users lists as they will see many questions from people just like you, and while you may not have the same problem now, you may have it in the future, and just remembering that you saw the problem before can in itself be a massive help when you are faced with it and know that someone else has seen it, so you can search for it. The developers mailing lists are generally not for users of the software, but I've learned that its sometimes the best way to find solutions to my problems as many times any actual problem with the software will make it to the developers lists and be discussed there, in which case I can tell if its been resolved or if I have to work around it until someone thinks its a big enough problem to resolve it (or I pay someone to resolve it because its that important to my needs).

If you take the parents idea of virtualization into the picture you can accelerate all of the learning to an extent by setting up various test scenarios and figuring out how to work around the problems in those scenarios. You can setup a mirror images of your production systems and when you start to notice problems or potential problems on the production systems you can duplicate it in the test enviroment and figure out how to fix it there, trying several different options to find the one that yeilds the best results without screwing up your production servers.

Its more important that

Re:Virtualization

[masdog]

At the same time, virtualization will enable him to learn multiple skills at the same time. Not only will he learn the virtualization platform, but he can run multiple OSes serving multiple apps. He could have Server 2003/Server 2008 Active Directory and Sharepoint running on one machine, Exchange on another, Centos with LedgerSMB on a third, a FreeBSD machine running App X on a fourth, etc with a safety net to roll back to a snapshot if he makes a mistake.

As for quitting, that wouldn't be advisable yet. It would be a red flag to any HR person who is hiring him in the near future, and that may hurt him more than help him. I had trouble getting my resume in to places and I was at my current job for a year and a half.

Re:Virtualization

[Anonymous Coward]

Actually you _can't_ play many games on a virtual machine because the video isn't accelerated. Most games require accelerated video, unfortunately.

Re:Virtualization

[DeathElk]

Actually you _can't_ play many games on a virtual machine because the video isn't accelerated

VMWare fusion allows graphics acceleration. Pretty sure VMWare Workstation for other platforms also has this feature.

Google

[A non-mouse Coward]

Back when I learned, Google was around. Turns out, it still is.

Most of the modern linux distributions have excellent package management. Most of them take care of 99% of the deploy "correctly" or "securely" issues. The only downside is that no two packages put everything in the same place on the local file system. But that's no big deal, especially if you compare/contrast to other distros.

Shoot, you can get an Ubuntu server installed as a VM in 15 minutes. (I don't see the need for dedicated server hardware, unless you're focusing on nuances of driver and hardware setup.)

Follow these steps:
1) Install base
2) Install app from package
3) Add custom content to package
4) Scan with the whole slew of freebie security scanning tools
5) Realize that at this point, you're better than most orgs already.

Re:Google

[Anonymous Coward]

Swap being 2.5-3 times the size of RAM is a very dated approach, useful only really with RAM sizes of less than 1GB.

Most systems have upwards of 2GB RAM these days and in 99.9999999% of cases you won't need a 10GB swap partition.

Re:Google

[jd]

That would be fair if (a) you were not likely running SMP and/or multi-core, and (b) most server software is horribly bloated - I swear Oracle and IBM assume you've 10 gigs of RAM and a terabyte RAID array. Because you're likely to be running a LOT of seriously heavyweight software, especially if virtualizing the machine, you really do need hefty swap.

O'Reilly Cookbook series are very hands on

[millisa]

If you want a more hands on, how do I accomplish a specific task type approach to things, I've been very happy with the books in the O'Reilly Cookbook line. They usually run 35-50 bucks depending on topic and you'll want to page through one in a store before purchasing. All the information in the books can be found online, but they usually organize them nicely in the books. Most of the topics are 1-2 pages responding to a specific "How do I do X" type question. The Linux Networking Cookbook, bash cookbook, and Linux Cookbook and Linux Security cookbook might be a good set to start with for what you are currently playing with.

Re:O'Reilly Cookbook series are very hands on

[ibmjones]

. They usually run 35-50 bucks depending on topic and you'll want to page through one in a store before purchasing.

Or for 40 bucks a month, you can get the whole O'Reilly Library at:

http://safari.oreilly.com/

Well worth the money.

Not really much to know

[jkorz]

Setting up a secured lamp server (secured from being hacked, not secure as in ssl) isn't all that difficult. First, set up your lamp server just as you need it. Then install iptables (firewall), webmin and openssh. Set webmin and openssh to use random high (>2048) ports rather than standard. Set up openssh to use public key authentication (disable password auth) and set up webmin to NOT use local user accounts to login (you will have to set up webmin users). Then use the iptables module in webmin to block all traffic but the three ports you need (80 for web and the two random ones). If you want to be extra-paranoid, block webmin as well and learn how to tunnel it through ssh.

Well, by no means...

[Anonymous Coward]

should you ask for advice on Slashdot.

Optimization

[foo fighter]

Optimization is about finding bottlenecks and then using the scientific method.

The typical bottlenecks are CPU, RAM, Disk, and Network. A little research will reveal the tools that give you insight into those subsystems on your platform.

Using those tools, you can identify which processes are stressing each subsystem. Then a little more research will reveal the tools that give you insight into that process.

Then a little-to-a-lot more research will reveal what you can do to reduce the stress or beef-up your platform.

After you do this for a bit, you'll see why LAMP is usually referred to as a stack, and not as a turn-key server. Different parts of the stack need to be optimized for different subsystems.

Another very useful bit of research would be finding or writing your own tools to stress each of the subsystems.

The FreeBSD Handbook

[psergiu]

The FreeBSD HandBook and a FreeBSD install cd.

Read-it end to end. Yes, i know it's huge. You won't regret spending the time to read it. Install FreeBSD (even in a VM) and use it. Even if you'll use other operating systems in the furture it's a good read and you'll learn a lot.

Mean bastards, aren't we?

[johndmartiniii]

Today is one of those days that I wish I had mod points.

First, the question at hand, get yourself some virtualization, and get a box that you can just plug in at home and fiddle around with when you aren't doing anything else. Trial and error will help you.

Just make sure that you do your trials and errors on a testing environment and not in production. It is alright to make mistakes until you sort stuff out, just don't bring down the house.

Second, shame on you naysayers. Let this guy learn stuff as he goes. Where did our curiosity and creativity go? You could give him advice instead of being a rude, mean, naysaying bastard. Thanks for posting as anonymous cowards too. Real nice.

Re:Mean bastards, aren't we?

[andrewbaldwin]

I also wish I had mod points as this is a constructive response rather than the mean spirited "I'm so great, you're a waste of space" answers. We all started out once (some of us some years ago!!)

Remember when trying stuff out -- take plenty of backups. There are two types of people: those who back up fervently and those who haven't yet had a disaster !

One other point of personal experience I'd add (which no-one else seems to have mentioned) is buy yourself a notebook (a paper one, not a PC) and a pen. Then whenever you make any change, install anything or otherwise fiddle with your set-up, write down WHAT you did (and why?, what were you expecting to happen).

Keep it pencil & paper (rather than as a blog or text file) then you won't lose it if your disc or network crash.

You'll be glad you did when you want to roll back to a known working state; it's also interesting to look back on events of a few months / years ago and see how things have changed and what you've learned :-) Items highlighed in red with annotations such as "NEVER, EVER DO THIS AGAIN!" bring a wry smile.

Good luck !

Some Words

[RAMMS EIN]

While many other posters give you heat for not being knowledgeable, I commend you for making the effort to learn. Keep that attitude, and you will eventually get good at it!

As for optimization, my advice to you is:

1. Know what you need to optimize
2. Measure, don't guess

It's good to read some generic "how to optimize foo" advice, but be careful you don't end up spending your time and effort optimizing something that doesn't need it. Know what things need to be fast, and focus on those. Very often, you will find that, actually, everything is fast enough, which means you don't need to optimize anything at all.

Once you have determined what, if anything, needs optimizing (by measuring, of course), the main thing is to identify the bottlenecks. If your pages take a long time to render, is it the web server that's slow, the network connection, the web browser, the code on the page, the code that generates the page, the database, the filesystem, or something else? Once you know where the slowdown is, find out what's causing it. Again, measure, don't guess.

Then, once you know the cause, you need to think about how you can solve it. In many cases, this will be clear to someone who is skilled at working with whatever technology it is. For example, a good programmer will know how to improve a program, a good DBA will know how to optimize database access, etc. In some cases, however, you will find that the performance at your bottleneck can't be improved significantly. You can have a skilled programmer spend a couple of days to squeeze the last few percents of performance out of some function, but that isn't going to help if you need things to go twice as fast. In that case, you may be able to solve the problem by using more hardware or faster hardware, or you may simply not be able to solve the problem.

You're headed the right way.

[Qbertino]

You're headed the right way. Just keep going. I'd recommend Debian over CentOS, because it's the generic professionals distro, but that's not that important.
If you're feeling overwelmed by what is required to get a webstack up and running, you're absolutely right in that respect - its a non-trivial amount of stuff. Allthough it is a tag irresponsible to take such a job without the basic knowlege, mind you.

The classic LAMP webstack is solid but has lot's of components. Start with making a list of what you *don't* know, but would like to know. Formulate these out in questions and sidenotes to your self and write then down in a simple indented list in a text editor. Notch them of as you go deeper into each issue throughout the next few weeks.

Here's a list of things from the top of my head you need to know your way around as a professional admin:

- daemons on Linux/Unix

- cron-jobs

- the cli/Bash

- cli tools: wget, mc, emacs, ssh, scp, sort, ls, less, the concept of piping, rm, chmod, chgrp (these two will help you FUBAR your LAMP-stack a few times before you get a hang of it. Don't worry, we've all been there. :-) )

- learn VI or Emacs (the "No X" versions!!!). Get a book/download the docs/print out the cheatsheets. I personally recommend Emacs. Start today. Either are a pain in the ass and you won't bare any of those longer than 2 minutes in the beginning - their handling is bizar beyond any words - but 6 weeks from now, when you know your way about the 20 basic editing actions in Emacs and are logged in via SSH and have to digg through a script or a huge Apache config you'll be very thankfull.

- Learn Apache. Start with 2.2. Get a book. Oreilly is a safe bet.

- If the P in LAMP is PHP, learn PHP and do your maintenance scripting with the CLI version of PHP, thats what I do. Copying, maintenance, cron-jobs ... all in PHP. Very neat. You swat two flies in one move, as you can look into PHP code at app-level and find your way around should that be needed in an emergency.

- Replace PHP in the above paragraph with Python or Perl if required. If Emacs is your choice of CLI editor, Elisp is a good choice for scripting aswell.

- try to understand the file system and directory standard of Linux before you implement your own little world. A lot of the dirtree in Linux is a historically grown mess and up to individual disposition, but the essential security related stuff is not(!!). So don't mess around. Plan ahead. Take notes (on paper!) and be prepared for a reinstall after a week or two when you've totally borked your system or your systems rights.

- Learn a versioning system. I recommend SVN, as the newest hype, Git, is still to unwieldy to handle in most cases (not enough tried and true 3rd party tools). Learn the CLI of your versioning system and use it too, so you get a hang of it. Put your docs, custom configs and other files like scripts into versioning and use it. I strongly recommend "Pragmatic Version Control with [fill in favorite vcs here]" from the Pragmatic Programmers Bookshelf guys. Real world versioning without the useless theoretical bullcrap. A very good line of books that finally made me understand versioning the way it was meant to be. AND USE VERSIONING! F*CKING VERSION YOUR SHIT. At every occasion. I'm dead serious. Learn to use revert, diff, etc. DO PRACTICE IT! It seperates the pros from the wannabees. You'll eventually find out why. Trust me on this one.

- MySQL. Well, it sucks just as much as any other SQL RDBMS. If you hate SQL and all that comes with it with your mind, soul and body like I do, you'll just have to bite the bullet. Get a book with a good index and keep it around for hard times. Play with a few basics of the mysql cli client so you can get up to speed when you are in a jam. Don't waste to much time with it though. It takes a strange state of mind to deal with this kind of stuff. I've never quite gotten the hang of it. A GUI-tool can take the pain out of DB admining.

Re:Slackware

[xhunter]

Interesting you pick the "commercial" linuxes as good for real-world server skills and list debian as good for desktop. My experience would say debian spanks red hat for ease of server admin, particularly if you want access to more packages to help you do your job. For instance, say you want to install shorewall as a firewall, slony1 for postgresql database replication or ntop for network monitoring. Is redhat repository going to help you with that? No, at least not in my experience. On top of that the debian package manager reliability and repository options make Red Hat smell like the armpit of Redmond. My opinion is that those who've done some sys. admin and had a choice steer away from red hat and choose or migrate to something like debian, slackware, BSD or even ubuntu server. If I had to use redhat, I'd try to migrate to CentOS, but even they are reliant upon the master as is fedora and when the master is driven by a for-profit board of directores chances are you're not going to get what's in your best interest as a sys. admin.