Slashdot Mirror
Building/Testing of a High Traffic Infrastructure?
New Breeze asks: "I'm currently working on my first web 'application', and have discovered that I know less than nothing about setting up the infrastructure to manage a high traffic system.
Where does one go to learn about setting up the infrastructure required to host something like Slashdot? Or do you just say, 'Not my area!' and help them find a consultant?"
"My experience is pretty much limited to:
I haven't a clue. The last place I worked with on something like this hired a high dollar consultant who spend a huge pile of their money setting up a load balanced, oracle parallel server redundant everything system.
How do you test it? I've worked where they actually had a room with hundreds of systems on racks that they would configured to run test transactions against different servers and software builds for stress testing, but that's not in my budget..."
1. Install the web server on one box, the database on the same box if it's a small installation or a separate box if performance seems like it will need it. Add more memory and processors based on SWAG criteria. (Scientific Wild Ass Guess)I had a potential customer ask what I would recommend if they wanted to self host, they have around 300 remote locations and would have multiple users from each location hitting the application at the same time, so saying a couple of beefy servers probably isn't the right answer.
2. Contract with a hosting company.
I haven't a clue. The last place I worked with on something like this hired a high dollar consultant who spend a huge pile of their money setting up a load balanced, oracle parallel server redundant everything system.
How do you test it? I've worked where they actually had a room with hundreds of systems on racks that they would configured to run test transactions against different servers and software builds for stress testing, but that's not in my budget..."
These are very obvious links to a shock site, ignore them and mod parent down. Seriously, AC, don't you get tired of this?
it really depends on what you need.
In my experience though hardware (especially memory) and bandwidth come before a superoptimized software front-end & database.
A good introduction I can recommend is called "Developing IP-Based Services: Solutions for Service Providers and Vendors" - I forget who wrote it. But definatly worth reading on the subject.
If you want to throw some serious load at your equipment, get a few other systems saturating your network with Apache Benchmark (ab) requests. It gives lots of useful data, like response times, etc. . And you're best off toppling the application and trying to find the cause that it failed and working on that as someone already suggested. The rinse and repeat.
Looks like Apache has updated their tools since the last time I had to do this...
http://httpd.apache.org/test/
Check out what those guys do at Wikipedia. Don't forget to look at their useful links at the bottom.
Or maybe it's overkill.
First off, if this is a "must succeed with no problems" project, all bets are off -- hire an experienced consultant so you have someone to blame. Also, this technique only works when you have the type of site which will *build up* to expected load -- not get turned on instantly.
This is tough to generalize without knowing specifics, but here goes:
1. Make sure your application can work correctly when load balanced across multiple boxes
2. Keep webserving and DB work on different machines
3. Make sure your application can work with another database without much work (this gives you the option to hire, say, an Oracle DBA and buy an Oracle license if MySQL can't keep up.. does it even support row-locking yet?)
4. Have extra hardware handy, in the rack. Do NOT turn it on yet.
5. Observe the application running; determine bottlenecks, tune
6. If you can't tune it to perform adequately, NOW is the time to break out the extra hardware while re-evaluating the implementation.
If you throw all your hardware at the problem at once, you get very little warning when the shit starts to hit the fan, and no response scenario. Do NOT make that mistake. Load, test, tune, repair, repeat.
Do daemons dream of electric sleep()?
These are some very basic thoughts on the subject. They may not be 100% right for you, but will get you thinking in the right way:
Rule 1 - Three tier archictecture is popular for a reason - it works. Offload user interface (web) to dedicated boxes, make application itself run on separate boxes and make database separate
Rule 2 - When possible, scale horizontaly not vertically. Make sure your application is as stateless as possible and is capable of you just dropping in an extra server when needed without a lot of reconfiguring. Make sure you can survive a loss of a server without loss of data. Lots of cheap servers will most always work out better (and cheaper) than one big ass box.
Rule 3 - Make as much of your application as static as possible. Even pseudostatic data (something that updates every minute or so) should be made static and have a process re-generating it every minute or so. Not wasting your CPU time to render a menu or something on every hit will add up fast under heavy stress.
Rule 4 - Strip your HTML. For example, some crappier web languages (think ColdFusion) have a tendancy of inserting spaces for every line of code etc. A large application running CF (dont ask) would insert enough spaces to make a simple page hundreds of kb in size. Just turning on "the write to output only on demand" option will drop size of the page to next to nothing. So know what it is that you are producing on output and make sure it is lean. Turning on server side compression solves this better, however adds to CPU requirements. On trully stateless web servers this just mean you need more web servers. So MAKE YOUR WEB SERVERS STATELESS.
Rule 5 - Know how many users your upstream connection can handle (in simplest terms - average size of HTML communication * number of users) and make sure you do not exceed it. Limit your connectivity at load ballancer. Having some users not be able to access your site is better than having ALL users not be able to access your site. Make sure you get plenty of bandwidth to spare. If you are setting up a multi-site presence, make sure your intersite communication is a - not going over same line as incoming and b - has sufficient bandwith and latency to serve the traffic.
Rule 6 - Professional load testing tools cost big bucks. But if you are carefull you can fake it with some open source software. Google it. When testing remember to take into consideration the limitation of your tester system and bandwidth.
-Em
RelevantElephants: A Somatic WebComic...
Okay, let's put it to a real test. Post some good pr0n links and we'll try to slashdot them!
Having worked for porn sites, I will tell you this: They, more than anybody, will rise to the challenge. Porn = Traffic = Ad Click Throughs = Money. If a porn site sees a sudden rise in traffic, they will drop more servers into their delicately load balanced system without blinking an eye. Porns sites aren't slow. There is a reason why.
It may or may not be a great idea depending on your situation. For one - the cost of SSL card for F5 is so high, it may be easier to just get extra servers. For another, I work with some banking applications and having data sent cleartext, even on an inside network directly connected to load balancers is NOT a valid option.
However if local security can be ignored and you have the money to spend, F5's offer a nice offload of encryption processing. But then again, so do hardware cards for individual servers.
-Em
RelevantElephants: A Somatic WebComic...
It's also a good opportunity for people to learn from other's experiences. Christ, man, I don't see why people have to hoard their knowledge. What kind of example does that set?
There is a reason why this is a specialty. There isn't a clear answer.
The answer depends on many factors such as:
- how heavy are the pages (many pictures?)
- what's the platform (Lamp/J2EE/etc....)
- how is the usage?, if someone gives you a figure for concurrent users, ask yourself what they mean by that. Some apps have users contstantly submitting, others once in a few minutes
- how are they connected? Reverse proxy can really help for slow connections!
- if you have performance problems, investigate where the pain really is. Is it the (R)DBMS, or the app server, memory IO.
- etc. etc.
Most of all: test! Get something like grinder, or opensta and put some serious load and stress on the setup. See where it hurts.
Make sure that if you have a problem, you actually fix the right problem. It is ok to add hardware, but you have to know what hardware to get.
Also many problems can be handled by configuration, such as preventing the system to come to crashing halt by limiting the amount of connections to the amount you can handle.
Look overhere Perl strategy doc It has some good advice that will help you also in non perl environments.
---
Accommodating "high traffic" that is mostly bandwidth intensive is quite a different problem than accommodating traffic that is database intensive.
Here's what I do: Bitty Browser & Andromeda
There's no shortcut or substitute for good profiling and benchmarking of your application. If you're doing anything mission critical, SWAG flat out isn't going to cut it. You need to PROFILE to figure out what resources your app uses so you can (a) tune and (b) allocate appropriately. For instance, if your app is making a lot of database queries you can look at ways to cut those down (such as caching responses where possible). And you know you'll need fairly beefy database servers (or conversely, that you can get away hosting the database on the same box handling the web front end).
BENCHMARKING allows you to size the hardware apropriately. This needs to be done scientifically. Set up whatever architechture - benchmark - if it doesn't meet the expected load, plus reasonable headroom for future growth, plus reasonable slop for load spikes, you can use your profiling results to help spot bottlenecks. Consversely, if you're getting "too good" performance (e.g. some servers are staying idle) you'll know where you can safely cut. The key here is to handle it scientifically. Measure, vary only one variable, then measure again. Rinse. Repeat.
Even if your company goes with a consultant, you need to be deeply involved in the process. Web application performance is too deeply tied to the application (duh) to allow independant evaluation of the infrastructure requirements. You need top to bottom approach, from application architechture, to implementation, to deployment, to get it right. All steps of the process are going to impact performance.
Jesus what an asshole this parent poster is. Someone asks for advice and this arrogant guy calls them incompetent for not being born with the knowledge. Someone please mod him troll; this is exactly why non-techies think we're all arrogant.
This is one of those areas that is there is no set answer. There are lots of articles on the topic, but usually on systems larger than you plan to do. Go to user groups, but many in user groups are doing smaller site, but some might be doing what you are.
Main thing is define what you call a lot of traffic. A lot to one person isn't a lot to another.
Then nail down your budget that will be your most defining factor.
Then when designing use a design that is easy to scale. That way if you are off you can scale with little pain.
Personally I would put money into the database server, they can be real pain to scale. The web side design as a farm even if only two web servers to start with. Decide how you plan to load balance. A couple web boxes DNS round robin will do, but bigger you have to look to real load balancing options. Also what is your SLA that will determine how big your farm needs to be or if to keep hot or cold spare boxes around. IF a farm how are you going to keep content in sync? Then power, cooling, Security, and on and on. Its a lot of work, but when done and everyone is happy you can't wait for a even bigger project.
I work for a website that does alot of traffic (it's a specialized industy, and no it's not pr0n). The site pushes about 10Mbit/s from 9-5 during the week through 6 webservers. There are a couple things you need to look at as far as making a site like that work.
The first thing you should do is look at your system and determine what your resource drains are. Do you have a database? Is it read-write or read-only? What are your replication and growth options for that app? That affects your scalability at that point and similarly applies to applications like EJB or other app servers. Do you use sessions? Do you have some sort of session aggregator available so sessions could be accessibly from multiple webservers? There are lots of things like this you need to find. I for example, setup seperate webservers (tux & apache) to handle static and dynamic content so that my DB connections would not be held by processes not using them.
The next thing you need to do, is know how your system is used. You should be able to statistically break this out from your logs by looking at a small set of users during testing. I found that 60% of my hits were to one page, and I knew I had to really optimize that (someone mentioned apache bench, which can work very well for testing single pages). Also, you need to know how parts of your site use your resources. If you have a single DB server and multiple webservers, you don't want anything slowing your DB because that cascades back to your webservers. We have pretty strict performance testing guidelines whenever a part of the site is updated, and I recommend doing your performance testing as you go.
The final thing you need to do, is have a growth plan. Do you know how to setup load balancing for your webservers? Can your DB/app servers be replicated, or do you just need to buy faster hardware as you grow? Do you know your capacity thresholds from your performance testing? If your system is going to grow, you're going to need to be able to answer these questions.
If you make sure you've got your scalability issues known, and you don't lock your self into something that can't grow, you should be ok. Beyond that test for speed under load, and track how your performance changes over time. That will help you know when you need to grow your hardware. HTH.
I've done the porn gig for a while, and I can tell you a "secret" that won't work all that well for a web application. Anything on a porn site that can be served up statically, usually is. If it's a "page of the week" it usually is generated once a week, so that you don't need to pull from the database on every single hit. At least, that's they way it works/worked for us.
If you do need to hit the database with almost every page load, there's a few simple tricks. These are what I use with PHP, which is the only language we use on our sites. If there's common information that will be called upon a lot, store it in a session variable, as it saves database transactions (not much, but every bit counts). Take a look at how you're database transactions are setup. If you have a table with 30 columns, and you only need 8, then select those 8 instead of *. If there's a lot of traffic on a particular page, try persistent connections. Look at an object oriented way of writing all of your db transactions, makes big applications a little smaller and easier to work with, and if you find a way to fix something you're doing it, just fix the class instead of every script in the application. There's plenty of other little "tricks" and what i've said doesn't really qualify as tricks, but there's plenty you can do before you start talking multiple load balanced servers.
Knowledge comes from Experience, and experience comes from Doing.
Mistakes will be made, They key is in mitigating the effects of those mistakes. Redundancy and Manageability are your two biggest buzzwords here. A good load test and utilization projections are definitely key, but no matter what you think your userbase will be, if it's a public application, you'll almost certainly be wrong. Try to prepare for the most traffic possible.
Redundancy on every level, including switching infrastructure is a very good plan. Any decent server sold can use multiple bonded NICs for reduncancy, if possible design your network such that if a switch fails, your network will fail over to another switch, etc.
I would suggest going to many local datacenters and interviewing each with probing questions relating to your situation. You will find that they are all relatively equal in terms of Standard DC items:
Diversity of route (physical entrance of cabling into the building) and redundant carriers.
Cooling
Power and backup gens
The things they differ on will be the readiness of their NOC team (do you have to fill out a web-form or call a call-center in East St. Louis to get a problem fixed in San Jose, or can you just "call the NOC and somene goes to your cage"), the monitoring/alerting they provide their customers for issues on the datacenter network. Infrastructure-wise, most DC's can provide you with Ping/Power/Pipe, but the service and SLAs are where they get points.
Do a LOT of reading. Depending on your platform, you have many choices. Linux vendors and Microsoft both have good platforms WRT building redundant networks, provided you do your homework.
Which brings you to manageability. Make sure that you have a deployment framework you can live with right from the start. Deploying code by hand is alright when you have 2 sites in IIS x 3 or 4 machines, but it gets hairy when you have 15 sites x 20 webservers. Make sure you can deploy web content, mid-tier apps, etc, with the "click of a button". This helps to ease the possibility of repetitive mistakes being made. Depending on the app, you may have to roll-your-own, but it's worth it.
Scalability. Make sure you pick a DC that can grow with you. If you plan to start out with 4 1u rackmount webservers and maybe a 7u DB, plus some storage array, make sure there is "room to move" in the DC without needing to cross-connect all over their facility with a cage here and a couple cabinets on the other end. Scalability testing by your engineers would be a great plan also. During load testing, if you're planning on using 2 mid-tier servers to process "Project X" from the web-users, set up 6 or 8 and load them up with bogus traffic. See how long it takes to kill your DB server.
Monitoring/analysis. Make sure you have a monitoring system into which you can hook custom monitors and alerts. Of your installation, those parts with the lowest levels of monitoring will be the ones most prone to breakage. Good packages here are NetCool and HP Openview. Expensive though. It's something you can probably write in-house until you need to spend the big bucks for an enterprise package.
Look to do a lot of reading, but break it into chunks. There is (I hope) no book called "Building and Maintaining High Traffic Enterprise Networks, for dummies, vol2". Every network will be different. But if you componentize your search, you will yeild great results. If you look to build your own monitoring or code deployment system, read up on WMI, read Cisco related newsgroups for network layer redundancy, etc.
Consultant is NOT a dirty word. Make sure you hire one for the right reasons. You do not want someone to come in and "make it so". You want someone with more experience than you have to work WITH you to design a network that you understand, can maintain, and which will scale. There's an art to it, hire Chris van Allsburg, not Picasso, Dali or certainly not Poll
I like music
What does it mean to not scale "vertically"? When I read that, the only thing that comes to mind is to put the boxes next to each other, not on top of each other. From context I gather that horizontally means extra machines, but what does vertically mean?
Horizontal scaling - adding more machine
Vertical scaling - adding more CPU/Memory/etc to existing machines.
For example, a horizontally scaled application may have 20 1u 1cpu servers, a vertically scaled one has a Sun E15k heating up the room.
For "dropping in an extra server when needed without a lot of reconfiguring", what do you mean by "a lot of reconfiguring"? Obviously you need to get the machine, install the os, set up networking, install the web server, setup the web application, point it at the database, etc. How does the application being "stateless" help? I guess, what are some examples of state that an application can have that will make configuring an additional web server difficult?
Reconfiguring the application not the servers. A stateless web server does not store any user state. Meaning that if a user hits web server A for one request, and web server B for another, the user will not know the difference. Also meaning that if you add another server, you do not need to worry about conflicts, sharing data, etc. Stateless servers can be taken offline or brought online without any fuss. They become a commodity appliance and if you need more, you just get more. In realistic terms this means that if you need state for the application (login, etc) you either store the state on the client's machine in a cookie (BAD, all sorts of abuse is possible) or better store an temporary ID in a cookie (or in URL) and store state in App server or (better) DB. A lot of web servers and app servers offer clustering to solve the state issue. While this may or may not work, most of the time it is a marketing hype that rarely lives up to expectations and add extra load. It also violates KISS principle (Keep It Simple Stupid) and will give you more headache than it is worth.
Concerning the pseudo static data regeneration, what if the thing that was being updated was only accessed once every half-hour on average? I am assuming then that generating the page on demand would be better?
Use your brain. The idea is to lower CPU requirement and potential risk from overloading, not just to use a cool trick. Do whatever works best.
I don't really know what you mean by "MAKE YOUR WEB SERVERS STATELESS". I mean, they have to know if a request just came in, where the data is, what time it is etc, and that stuff gives it state. I am assuming you mean something else by stateless but I cannot figure it out.
State implies retained state across MULTIPLE connections/hits. Most application require state, however state does not need to be kept on the web servers and sometimes not even on app servers.
HTH
-Em
RelevantElephants: A Somatic WebComic...
A good alternative is the book by OReilly - Web Performance Tuning (http://www.website-owner.com/books/servers/webtun ing.asp).
Dw.
I have some experience with administration of web sites with very high traffic. My previous experience was with p0rn sites (lots of sites, lots of concurrent accesses). My current job is at Skyrock / Skyblog, that serves about 25 million pages every day.
In both jobs, the infrastructure was extremely similar.
The entry point is one (or more) load balancer.
A load balancer will not only blindly allow you to have multiple backends. It will also accept client connections, buffer the request, get the data from already established (keepalive) sessions, buffer it, and transmit it though large chunks to the client. This, alone, really helps to reduce the number of Apache processes that are taking resources (especially memory) for nothing.
The load balancer can also do other things, like protecting the servers against some attacks, plotting the current workload of every backend, compress HTML pages, etc.
At my previous job, we were using Foundry Serverirons. Now, we are using Zeus ZXTM http://www.zeus.co.uk/ with great success. Although it's very expensive software, it's way cheaper than Foundries, way more configurable, way more user-friendly and we are very pleased with it so far. A single PC handle 300 Mb/s (Linux 2.6 is needed for epoll).
The load balancer can also be configured to send the requests to this or that server according to the request.
Thus, servers are dedicated to specific tasks.
We have a bunch of static servers for static HTML, CSS, images, etc. They run minimal Apache servers, designed for speed, with NPTL and the worker MPM. Non-forking servers like thttpd or lighttpd is also an option. The static servers are mainly old P3 machines, with only 512 Mb RAM.
Then, we have servers for PHP. The Apache they are running is huge (our web sites need a lot of modules), the hosts are dual 3 Ghz Xeon with 2 Gb RAM and there are some other specific tweaks.
Content differentiation is important. It's a waste to spawn huge Apache process to serve static stuff, just because the same host should also be able to serve PHP. Also, tuning (esp. NFS) is very different for static and dynamic content. And as a specialized server often serves the same files, caching is more efficient.
We run Gentoo Linux on all web servers, plus one DragonFlyBSD (mostly for testing).
The same content differentiation is made for SQL server. One SQL server serves one sort of thing, so that caching is efficient. Also don't forget that on x86, Linux and MySQL can hardly use more than 2 Gb of RAM. So with big tables, this is really annoying. We are switching SQL servers to Transtec Opteron-based servers for that.
On high traffic infrastructures, the I/O is often the bottleneck especially if you serve a lot of different content.
For our blog service, we had to buy a Storagetek disk array with 56 disks (fiber channel, 15k) in RAID 10. As NFS would introduce too much delay, we directly plugged two web servers to the controller of the disk array. These web servers are the NFS servers for the PHP servers, but they also directly serve the static content.
The access time of hard disk is really annoying. For shared data, but also for databases. We found that RAID 5 was way too slow (even with the high-end Storagetek/LSI controller) since we have about 1 write for 5 reads. So we had to switch everything to RAID 10. It really performs better, but it's obviously more expensive.
Another bottleneck was the share of PHP sessions between all load-balanced PHP server. We first used a MySQL/InnoDB-based solution, but it poorly scaled. That's why I had to write specific software : Sharedance http://sharedance.pureftpd.org/
In a high-traffic infrastructure, my hint would be to use many modest, but specialized servers over one huge mega-fast server that does everything. This is way more scalable. And easier to manage, even from a financial point of view. You can b
{{.sig}}
You don't mention if you're on the applications side of the world or the network, so I'll cover a little about both.
1. If you're on the app side, make friends with the network side and vice versa. To understand web site management and acceleration, you will need to know about both parts. Making peace with the other team is crucial to a successful site.
2. If you are on the app side, start thinking about concurrency from the start. You're going to have not 2-3 users at the same time, but more like hundreds if not thousands. This means that you can't do things like lock up tables and the like in the database. If at all possible write your application so that users don't need to come back to the same server to track their session information. Make sure each request is tracked quickly and easily. Also, differentiate your static content from the dynamic content -- you'll eventually want to cache the static content and life will be easier with static objects being served out of a known location. And please... please, please, please... make sure your app generates clean HTTP headers. Set your cache controls correctly, don't duplicate headers, don't be a smart-ass with your headers. Just use clean headers. ASSUME that there will be proxies between you and the client. ASSUME that you will not be able to control all of them.
3. Don't forget about megaproxies. Depending on the nature of your site, you're going to have a ton of your users coming from a small handful of addresses. (e.g., AOL) While some megaproxies have fixed the issue of a single user coming out of multiple proxy servers, all have not. This means anything that you use for client IP persistence is broken.
4. Client IP addresses... don't assume you have them. Don't assume they represent a unique user. They don't. Many load balancers/web accelerators also need to act as proxy and will replace the client IP address anyway. (Don't stress about logging -- any reasonable one will insert the client IP address in a HTTP header that you can extract like X-Forwarded-For:)
5. Peak load on your web servers. Apache can go fast, scale, blah blah blah... my ass. It's not the web server or operating system that is going to determine your peak performance. It is your application itself. Be prepared to fess up to the reality that your application peak performance is not going to be hundreds or thousands of requests per second unless you go insane with the optimization. (e.g., write your application into the web server and embed the whole thing into the kernel, etc.) Assume you're more likely going to get a few dozen requests/sec per app server. Keep that in mind as you plan server purchases and scaling.
6. HTTP request does not equal TCP connection. Don't assume that. With HTTP multiplexing like the stuff that Netscaler does (web accelerator), you're going to see most of your requests coming out of a small handful of TCP connections. Make sure your application supports that. Even if you don't use a web accelerator, browsers will do that do. Don't cheat and force the connection closed on every HTTP request, your web server will crap.
7. This is related to 6, but don't forget that web connections are very short lived compared to what the original designers of TCP were thinking about. As a result, you're going to run into cases where you run out of epheral ports (netstat -an will show a ton of ports in TIME_WAIT) even though your machine is idle. This is why HTTP Multiplexing is important -- you don't want a lot of connection churn. Yes, you can tweak your OS settings so that TIME_WAIT expires quickly, but that isn't going to help your overall performance. (TCP connection setup/teardown is a huge burden on a HTTP request that may only span a few packets...)
8. Look into HTTP acceleration technology from the get go. I've used several different brands and I've found Netscaler's to be the best. They are crazy fast and capable boxes that have a ton of features (like the HTTP multiplexing, SSL acceleration, HTTP compression, web
better than ab is siege, which can deal with HTTP/1.1 Keep-Alives, and give more regression-style stats. it's at joedog.org.
better than siege would be something like httperf, and autobench, which will give you some indication whether or not your client generating the requests is still healthy. autobench also allows you to run multiple instances of httperf on different machines, and then aggregate the numbers after the test.
remember folks, there are only 65535 (minus 1024) ports that any machine can be using with one IP...that has to be considered as well, including at the load balancer layer.
300 sites, between 12 and 200 concurrent users at a site.
It's a CRM system, i.e. some basic data entry, some portions are transaction processing. i.e. the workflow portion for the base part of the app is very simply:
Search for customer by various criteria.
No customer found, add one.
Retrieve customer information.
Add current order information being stored for this customer.
Process loyalty/discount programs to see if customer qualifies for an award.
Return award to order entry system for processing.
There's a lot more to it, but that's the meat of it. It's fairly data intensive, there is a great deal of information stored for customers for use in data mining the collected information. It's primarly web service based, but there is a fairly extensive management and reporting tool that is all HTML based.
My guess is going to be that the bottleneck is going to the the database, but we've done extensive testing with a million customer sample database running multiple instances of test applications from 10 other boxes, but that doesn't exactly prove much as it's too predictable.
I think there was a misunderstanding. I know how the application works in great detail. I know that it can be scaled up across multiple machines. It will scale up.
What I don't know is how to judge what hardware to reccomend to someone wanting to self host.
I'm pretty damn sure that only comes from 1) Testing, and I'm not buying $10's of thousands in hardware to test with and 2) Experience, which I don't have.
The last place I worked at had a rack of nice quad zeon processor systems in their lab that they had to take back from a customer when they didn't withstand the onslaught. I'm leaning towards letting a consultant roll the dice, because I 'd lose my shirt if that happened.
Aight, I'll take a stab...
:-) Check out pound:
You'll need a hardware SSL loadbalancer, with redundancy:
http://www.coyotepoint.com/e450.htm
(Two of those).
You'll need at lease two web servers with CPU and RAM. The requirements on these boxes really depend on the app. I'd make them dualies at least, with fast Xeon processors (Good bang for the buck). A couple gigs of ram each. You can add servers to the load balancer later if you need to. Disk doesn't really matter, but I'd use a SCSI mirrored root volume for reliability.
The database needs to be redundant, and since you think it'll be the bottleneck, an Oracle RAC setup would seem to fit your needs. I really don't like Oracle from a developer stand-point, but two big servers with Oracle running on raw disks for performance is a tough combo to beat. Expensive and long on the setup and install, though.
With all of that, you'd have a large initial investment, but something that would grow with your needs. You could add new apps to this setup later on, as well. However, if you needed to run this on the cheap, the Linux Virtual Server project does a lot of this with simple Linux boxes.
If this is too expensive, the first thing to take out is the hardware SSL. I included it because I want them, not because I have 'em.
http://www.apsis.ch/pound/
A couple Linux boxes with failover setup and you've saved a good 40-70 grand. Requires some expertise.
1. Foundry ServerIrons at the front-end layer.
2. Front-end proxying/caching. Not just static content either, take dynamic content that need not be updated often and put it on the front-end in a fashion that does not require over-weight httpds (i.e. no mod_perl). Use session affinity tricks on the front end (such as mod_rewrite with cookies). squid for caching as necessary.
3. Back-end heavy servers should have a maximum amount of memory, and obviously lower maxclients.
4. NetApp storage on the back-end, scaled as needed.
5. http://www.backhand.org/mod_log_spread/
6. Well designed network topology and aggressive switch partitioning: hint, use vlans and minimize trunking.
yes, i Know you asked how can find how to setup a high traffic architecture. I think you came at the right place on Slashdot.
:10s of millions of pages views a day, most of them dynamic.
Although I have never seen really many documentation online, I have setup many architecures in the past, and still able to handle very high volume traffic
It really all depends on ONE factor: money.
I will give 2 choices, I have implemented both:
Appropriate budget:
Frontends/Load Balancing: We had a pair of of Big/IPs with SSL accelerator, configured for redundancy, that rocks.
behind them, we had a clustered NetApp F840, with gigabit interfaces, on a gigabit networks.
Frontends: We were running Apache, with all the binaries, config, webpages, perl scrips located on the shared filesystem. Each machine was a dual CPU, 2GB memory, 2x36GB scsi drive, we had 26 of them, double the capacity really needed so if a machine or two were to go down during the night, no need to worry and it would wait for next day, business hours, great for peaks as well.
As a database backup we had an Oracle Cluster on a SUN 6650, 14CPUs, 14GB of Memory, connected on an EMC storage. One machine was configured as the master, the other as a standby with the possibility to take down the primary and mount it's filesystem directly from the SAN. Pretty much all the config was on the SAN, on different volumes, and could be mounted on either machine. Each volumes had a copy and an hourly update in case of failure of the primary volume.
Now for a more realistic scenario with low budget:
- Load Balancer: Get 2 Linux machines, I'd suggest machines with 2GB Memory, 2x36GB Disk, 2x3Ghz CPUs, with Linux Virtual Server. (http://www.linuxvirtualserver.org/)
- Build 2 Linux machines that you would use as NFS Server (If you are short in budget also could use them as Oracle or Mysql Server), configure them with 2 external scsi arrays that can be mounted on either machine. If you are really short in budget, don't use external array, but big enough internal drives, and for example rsync to replicate the data between the 2 of them. (I would personnally use LVM, establish a snapshot copy on the master and do a rsync of this snapshot. If you have a database on it, put it in quiet (hot-backup) mode while you do the snapshot
).
- FrontEnds: Get a couples of machines with 2 CPUs, 2GB memory for example, 2x36Gb drives. Configure them to mount the filesystem from the NFS servers.
- Database, it is budget, use Mysql (or Oracle this would work), configure one machine as Master, the other as read-only. Have all your machines interrogating either machines for read-only requests, and going to the master only for write requests.
If you need more power: configure more frontends, configure more read-only slave database server. Now if you are write intensive, more than reads, on the database, then it becomes a bit more complicated.
if you want to know more, contact me off-list.