Slashdot Mirror
Server Optimization For Newbies?
supaneko writes "I recently took a new job as a network and server administration for a small IT company. I am absolutely shocked at how much is taking place within this company that I have little to no experience with. To help bolster my experience, I purchased a used server to use for hands-on training and practice. My ultimate goal is to have a complete, secure LAMP server available to the public running CentOS. I have been browsing the Net for various guides and tips on setup, optimization, security, and maintenance, but nothing I've found really gives me a hands-on approach to the topics I want to learn about. When you all started out, what route did you take to pick up the server setup and maintenance skills you have now? Is there anything in particular that you would recommend to someone who has excellent skills with consumer PCs and servers but is a total newbie to corporate and enterprise networking and servers?"
Learn about virtualization. Take your pick of free offerings: ESXi and Virtual Server from VMWare, Xen, Virtual Server from Microsoft, etc.
Using virtual servers that are hosted on your new physical server will allow you to set up any kind of operating system you want and any applications on that operating system again and again and again with no fear of messing anything important up. Also, you can run (depending on memory) multiple operating systems side by side.
From there, you can start diving into learning all the operating system, application server, database server, etc minutia you like!
Oh, and don't forget learning about P2V. That will allow you to do all kinds of "what if" scenarios without affecting real servers.
I'm a big tall mofo.
Back when I learned, Google was around. Turns out, it still is.
Most of the modern linux distributions have excellent package management. Most of them take care of 99% of the deploy "correctly" or "securely" issues. The only downside is that no two packages put everything in the same place on the local file system. But that's no big deal, especially if you compare/contrast to other distros.
Shoot, you can get an Ubuntu server installed as a VM in 15 minutes. (I don't see the need for dedicated server hardware, unless you're focusing on nuances of driver and hardware setup.)
Follow these steps:
1) Install base
2) Install app from package
3) Add custom content to package
4) Scan with the whole slew of freebie security scanning tools
5) Realize that at this point, you're better than most orgs already.
libertarian: (n) socially liberal, financially conservative; neither left, nor right.
If you want a more hands on, how do I accomplish a specific task type approach to things, I've been very happy with the books in the O'Reilly Cookbook line. They usually run 35-50 bucks depending on topic and you'll want to page through one in a store before purchasing. All the information in the books can be found online, but they usually organize them nicely in the books. Most of the topics are 1-2 pages responding to a specific "How do I do X" type question. The Linux Networking Cookbook, bash cookbook, and Linux Cookbook and Linux Security cookbook might be a good set to start with for what you are currently playing with.
How did you get a job as a company's sole "network and server administration" (sic) when you are a "total newbie to corporate and enterprise networking and servers"?
In every case I've experienced where someone was hired for a sysadmin job with absolutely no experience, there was a more senior person on staff there to mentor/train them. But it doesn't sound like that's the case here.
So... either (a) you were completely up front with your employer about your lack of experience and they hired you anyway, in which case there's no problem because they have limited needs, know you're learning and don't expect much; or (b) you lied to them, in which case the answer is "quit and go get a job you're actually qualified for".
Read my blog.
Slackware. Forget about Redhat and all the other GUI-fied distributions. Install Slackware and do everything yourself. It's the only way to learn.
This is good advice. I did the same back when I was in school thinking it was pre-requisite knowledge for an IT job. Then I got my first IT job and became disillusioned at all the idiots that were making more money than me that had no clue how it all worked. They kept looking for the next--> next--> finish buttons.
libertarian: (n) socially liberal, financially conservative; neither left, nor right.
Assembler. Forget about Slackware and all the other already-coded distributions. Learn assembler and code everything yourself. It's the only way to learn.
Nah, that's too extreme. Everyone knows that the best way to learn Unix is to run Gentoo.
Read my blog.
Optimization is about finding bottlenecks and then using the scientific method.
The typical bottlenecks are CPU, RAM, Disk, and Network. A little research will reveal the tools that give you insight into those subsystems on your platform.
Using those tools, you can identify which processes are stressing each subsystem. Then a little more research will reveal the tools that give you insight into that process.
Then a little-to-a-lot more research will reveal what you can do to reduce the stress or beef-up your platform.
After you do this for a bit, you'll see why LAMP is usually referred to as a stack, and not as a turn-key server. Different parts of the stack need to be optimized for different subsystems.
Another very useful bit of research would be finding or writing your own tools to stress each of the subsystems.
obviously no deficiencies vs. no obvious deficiencies
The FreeBSD HandBook and a FreeBSD install cd.
Read-it end to end. Yes, i know it's huge. You won't regret spending the time to read it. Install FreeBSD (even in a VM) and use it. Even if you'll use other operating systems in the furture it's a good read and you'll learn a lot.
1% APY, No fees, Online Bank https://captl1.co/2uIErYq Don't let your $$$ sit in a no-interest acct.
Today is one of those days that I wish I had mod points.
First, the question at hand, get yourself some virtualization, and get a box that you can just plug in at home and fiddle around with when you aren't doing anything else. Trial and error will help you.
Just make sure that you do your trials and errors on a testing environment and not in production. It is alright to make mistakes until you sort stuff out, just don't bring down the house.
Second, shame on you naysayers. Let this guy learn stuff as he goes. Where did our curiosity and creativity go? You could give him advice instead of being a rude, mean, naysaying bastard. Thanks for posting as anonymous cowards too. Real nice.
If you don't know what you're doing, you can't make mistakes.
While many other posters give you heat for not being knowledgeable, I commend you for making the effort to learn. Keep that attitude, and you will eventually get good at it!
As for optimization, my advice to you is:
1. Know what you need to optimize
2. Measure, don't guess
It's good to read some generic "how to optimize foo" advice, but be careful you don't end up spending your time and effort optimizing something that doesn't need it. Know what things need to be fast, and focus on those. Very often, you will find that, actually, everything is fast enough, which means you don't need to optimize anything at all.
Once you have determined what, if anything, needs optimizing (by measuring, of course), the main thing is to identify the bottlenecks. If your pages take a long time to render, is it the web server that's slow, the network connection, the web browser, the code on the page, the code that generates the page, the database, the filesystem, or something else? Once you know where the slowdown is, find out what's causing it. Again, measure, don't guess.
Then, once you know the cause, you need to think about how you can solve it. In many cases, this will be clear to someone who is skilled at working with whatever technology it is. For example, a good programmer will know how to improve a program, a good DBA will know how to optimize database access, etc. In some cases, however, you will find that the performance at your bottleneck can't be improved significantly. You can have a skilled programmer spend a couple of days to squeeze the last few percents of performance out of some function, but that isn't going to help if you need things to go twice as fast. In that case, you may be able to solve the problem by using more hardware or faster hardware, or you may simply not be able to solve the problem.
Please correct me if I got my facts wrong.
First off, Optimization is less important.
You can spend days, week, or even longer... trying to make your systems run better and with fewer problems... but problems will crop up. And if you spent all that time just "Optimizing," you might find yourself between a rock and a hard place...
I learned early on that Backups are ever so important. Our shop doesn't do tape backups, but we do Disk-to-Disk backups of our virtual machines, and the backups are off-site. We also do a traditional file backup as well, with versioning.
Depending on your shop, money may or may not be an issue. Whatever you want to do, it can be done for every budget. The cheaper ways just require more time/expertise on your part, and that means it might not pass the "Mack Truck Test*." If your company wants something somebody else can step in with a basic training of how things work, you'll have to go with a more expensive solution.
Once everything is working like it should, then start working on improving it.
--Pathway
*: The Mack Truck Test - If a system requires some expertise to operate, and the sysadmin is hit by a Mack Truck, how long will it take for somebody else to fill the role of sysadmin? If the amount of time is acceptable to the employer, then it passes the Mack Truck Test.
Post a link to your server on Slashdot. I guarantee you'll get a fast and furious lesson in server optimization and security.
You're headed the right way. Just keep going. I'd recommend Debian over CentOS, because it's the generic professionals distro, but that's not that important.
If you're feeling overwelmed by what is required to get a webstack up and running, you're absolutely right in that respect - its a non-trivial amount of stuff. Allthough it is a tag irresponsible to take such a job without the basic knowlege, mind you.
The classic LAMP webstack is solid but has lot's of components. Start with making a list of what you *don't* know, but would like to know. Formulate these out in questions and sidenotes to your self and write then down in a simple indented list in a text editor. Notch them of as you go deeper into each issue throughout the next few weeks.
Here's a list of things from the top of my head you need to know your way around as a professional admin:
- daemons on Linux/Unix
- cron-jobs
- the cli/Bash
- cli tools: wget, mc, emacs, ssh, scp, sort, ls, less, the concept of piping, rm, chmod, chgrp (these two will help you FUBAR your LAMP-stack a few times before you get a hang of it. Don't worry, we've all been there. :-) )
- learn VI or Emacs (the "No X" versions!!!). Get a book/download the docs/print out the cheatsheets. I personally recommend Emacs. Start today. Either are a pain in the ass and you won't bare any of those longer than 2 minutes in the beginning - their handling is bizar beyond any words - but 6 weeks from now, when you know your way about the 20 basic editing actions in Emacs and are logged in via SSH and have to digg through a script or a huge Apache config you'll be very thankfull.
- Learn Apache. Start with 2.2. Get a book. Oreilly is a safe bet.
- If the P in LAMP is PHP, learn PHP and do your maintenance scripting with the CLI version of PHP, thats what I do. Copying, maintenance, cron-jobs ... all in PHP. Very neat. You swat two flies in one move, as you can look into PHP code at app-level and find your way around should that be needed in an emergency.
- Replace PHP in the above paragraph with Python or Perl if required. If Emacs is your choice of CLI editor, Elisp is a good choice for scripting aswell.
- try to understand the file system and directory standard of Linux before you implement your own little world. A lot of the dirtree in Linux is a historically grown mess and up to individual disposition, but the essential security related stuff is not(!!). So don't mess around. Plan ahead. Take notes (on paper!) and be prepared for a reinstall after a week or two when you've totally borked your system or your systems rights.
- Learn a versioning system. I recommend SVN, as the newest hype, Git, is still to unwieldy to handle in most cases (not enough tried and true 3rd party tools). Learn the CLI of your versioning system and use it too, so you get a hang of it. Put your docs, custom configs and other files like scripts into versioning and use it. I strongly recommend "Pragmatic Version Control with [fill in favorite vcs here]" from the Pragmatic Programmers Bookshelf guys. Real world versioning without the useless theoretical bullcrap. A very good line of books that finally made me understand versioning the way it was meant to be. AND USE VERSIONING! F*CKING VERSION YOUR SHIT. At every occasion. I'm dead serious. Learn to use revert, diff, etc. DO PRACTICE IT! It seperates the pros from the wannabees. You'll eventually find out why. Trust me on this one.
- MySQL. Well, it sucks just as much as any other SQL RDBMS. If you hate SQL and all that comes with it with your mind, soul and body like I do, you'll just have to bite the bullet. Get a book with a good index and keep it around for hard times. Play with a few basics of the mysql cli client so you can get up to speed when you are in a jam. Don't waste to much time with it though. It takes a strange state of mind to deal with this kind of stuff. I've never quite gotten the hang of it. A GUI-tool can take the pain out of DB admining.
We suffer more in our imagination than in reality. - Seneca
Slackware. Forget about Redhat and all the other GUI-fied distributions. Install Slackware and do everything yourself. It's the only way to learn.
This is good advice.
Actually, it's not very good advice.
Last I checked, Red Hat/Fedora/CentOS all have the exact same command line as every other distribution, and the system is configured using the same text files that have been used for nearly 20 years. All the GUI tools do is modify the config files.
For a newbie, having the GUI there to change a config then looking at what text file got changed (and how it got changed) is an excellent learning tool.
Also, last I looked, Slackware isn't one of the distributions that make a good bullet point on a resume. Red Hat, CentOS, and SUSE are good for real-world server skills, while Ubuntu, Debian, and maybe some Fedora would be good for Linux desktop skills.
I'm actually a software developer, but I work at a place with a lot of small projects and we do our own IT...meaning that we don't get budget for a dedicated IT staff and we end up doing a lot of it ourselves.
So, the way I learned what I need to know was to mess up a lot and get yelled at a lot. :-)
In all seriousness, we have finally landed at a place where we host and run our projects on Amazon's EC2. Some projects are even sophisticated enough now to leverage the EC2 platform and third-party services such as Rightscale for truly distributed cloud computing...but this isn't absolutely necessary if all you want is a place to run your production system. Best of all, since it's all virtualized so it's foolproof to learn new tech. When you're going to make significant changes you just save a snapshot of the current system, use it to start up a new instance off to the side, and screw it up any way you want to figure out a solution, and you can always easily revert to your previous snapshot if necessary. Just make sure you keep organized on which snapshots are configured with what, and be diligent about removing old snapshots that no longer have any purpose (again, purely organizational).
We've found in our business that the cost of doing this is vastly less than maintaining a rack of servers...so even though most projects don't leverage the cloud, we still benefit. (And of course there's room to grow into the cloud, which is also very beneficial.)
Get started by reading up on EC2, S3, and get the ElasticFox plugin for Firefox.
but have you considered the following argument: shut up.
Which, unfortunately, isn't that common.
Experience is the best teacher, but unfortunately it's not a particularly fast one. Anyone on /. can point you at a few interesting things like Slackware, Google and O'Reilly's back catalogue, and plenty of people already have.
What I would advise is:
1. Learn to see past the bullshit. There's a lot of it in IT, generally being spewed by salesmen and managers who pretend they know more than they do. In my experience, the less intelligible the communication (ie. the more buzzwords), the more likely it is you're talking to someone who doesn't have a clue. The word "Enterprise" is a good barometer there - it's often used completely unnecessarily and in the IT world has almost zero meaning.
Example: A Dell 2950 with every component that can be made redundant made redundant isn't an "Enterprise Server". It's a server. If you haven't specced it with redundant power supplies and disks, I wouldn't even class it as a server. It's a PC in a very expensive case.
2. Sometimes it's worth paying for a solution. /. would have you believe that Open Source is the Answer to All Our Prayers, and that Richard Stallman is the Messiah. Not true - there are plenty of products which don't have a half-decent open source alternative. Courier is a great IMAP server but at the end of the day, Exchange is a very capable product and is fantastically hard to beat feature-wise. Zimbra comes close but who knows what kind of a future it's got as it's owned by Yahoo. And I defy you to find a F/OSS business accounts system which isn't half-arsed. You can't say to the tax authorities "Errr... about those accounts we're due to submit - yeah, we just realised that our accounts system hasn't been updated to account for the recent changes in tax law and so we're having to wait until it is. Don't know how long that will take".
3. Security, security, security. Understand the ideas rather than just mindlessly installing the patches - a hardened Apache installation with a locked down PHP configuration behind a firewall operating some fancy layer 7 intrusion prevention system is great, and will help mitigate many forms of attack - but at the end of the day if you've got a badly designed PHP application all that'll happen is that intruders will access your data through a pretty web-based user interface.
4. Look at what the business does right now, think of how things could be made better and put together a system to make things better. It doesn't necessarily have to be something that will see the light of day - it could just be feasibility checking - but it'll give you something useful to do with definite goals which will teach you a great deal and at the same time may very well benefit the business.