Slashdot Mirror
Best Solution For HA and Network Load Balancing?
supaneko writes "I am working with a non-profit that will eventually host a massive online self-help archive and community (using FTP and HTTP services). We are expecting 1,000+ unique visitors / day. I know that having only one server to serve this number of people is not a great idea, so I began to look into clusters. After a bit of reading I determined that I am looking for high availability, in case of hardware fault, and network load balancing, which will allow the load to be shared among the two to six servers that we hope to purchase. What I have not been able to determine is the 'perfect' solution that would offer efficiency, ease-of-use, simple maintenance, enjoyable performance, and a notably better experience when compared to other setups. Reading about Windows 2003 Clustering makes the whole process sounds easy, while Linux and FreeBSD just seem overly complicated. But is this truly the case? What have you all done for clustering solutions that worked out well? What key features should I be aware for successful cluster setup (hubs, wiring, hardware, software, same servers across the board, etc.)?"
1000+ unique visitors is nothing. Even if they all hit the site at lunchtime (1 hour window), and look at 30 pages each (very high estimate for a normal site) that's only 8 requests a second. That isn't a lot. A single server could cope easily, especially if it's mostly static content. As an example, a forum I run gets a sustained 1000+ users an hour and runs fine on one server.
As for "high availability", that depends on your definition of "high". If the site being down for a morning is a big problem then you'll need a redundant failover server. If it being down for 15 minutes is a problem then you'll need a couple of them. You won't need a load balancer for that because the redundant servers will be sitting there doing nothing most of the time (hopefully). You'll need something that detects the primary server is offline and switches to the backup automatically. You might also want to have a separate database server that mirrors the primary DB if you're storing a lot of user content, plus a backup for it (though the backup DB server could always be the same physical machine as one of the backup webservers).
Whoever told you that you'll need as many as 6 servers is just plain wrong. That would be a waste of money. Either that or you're just seeing this as an opportunity to buy lots of servers to play with, in which case buy whatever your budget will allow! :)
http://twitter.com/onion2k
If the site goes down do you lose truck loads of money or does anyone die? Load balancing and HA sounds a little overboard for a site with a thousand visitors a day. A hundred thousand and you can probably justify the expense. I would probably just be looking at a hosted dedicated server somewhere for now.
there is also more of them than you can poke a stick at and prices are very reasonable. places like rackspace for this kind of thing for $100/mo.
the other advantage is you don't need to pony up for the hardware.
If you mod me down, I will become more powerful than you can imagine....
At work we have a pretty good experience with Pound - it's easy to set up & it load balances and will detect when one of your servers is down and stop sending traffic there. You can get hardware load balancing from people like F5 too.
If you're just starting out you'll probably want to start with software and then, if the load demands it, move to hardware
Machine-wise, we use cheap & not overly powerful 250 GBP, 1u servers with a RAID1; they'll die after a few years (but servers will need to be refreshed anyway) and they provide us with lots of options. They're all plugged into 2 gigabit switches
Global symbol "$deity" requires explicit package name at line 2. - If only $scripture started "use strict;"
Why are you purchasing six or so servers before you even have one online?
You say that you expect "1,000+ a day" visitors which frankly is nothing. A single home PC with Apache would handle that.
This entire posts strikes me as either bad planning or no planning. You're flirting with vague "out of thin air" projections that are likely impossible to make at this stage.
Have a plan in place for how you will scale your service *if* it becomes popular or as it becomes popular but don't go wasting the charities money just in case your load jumps from 0 to 30,000+ in 24 hours.
Lets get more blunt. Depending on what you are doing and if you want to worry about failover then 1000 a day is bugger all. Simple set up of Apache and Tomcat (if using Java) with running round-robin load-balancing will give you pretty much what you need.
If however you really are worried about scale up and scale down then have a look at Amazon Web Services as that will probably more cost effective to cope with a peak IF it occurs rather than buying 6 servers to do bugger all most of the time.
2 boxes for hardware failover will do you fine, if you are worried about HA the its the COST of downtime that you are worried about (i.e. down for an hour exceeds $1000 in lost revenue) which will justify the solution. Don't just drive availability to five nines because you feel its cool, do it because the business requires it.
An Eye for an Eye will make the whole world blind - Gandhi
I want to give you some more information. Based on your visitor estimates I think you do not have a lot of knowledge about it. Because for this number of visitors you do not really need a cluster.
But now to the other stuff. Yes, Windows clustering is (up to Win Server 2003 [1]) a lot easier. But this is because it is not really a cluster. The only thing you can do is having the software running on one server, then you stop it and start it on the new server. This is what Windows Cluster is doing for you. But you can not have the software running on both servers at the same time.
If you really want to have a cluster then you need probably some sort of shared storage (FibreChannel, iSCSI, etc.). Or you are going to use something like DRDB [2]. You will need something like this too if you want to have a real cluster on Windows.
I recommend you to read some more on the Linux HA website [3]. Then you get a better idea what components (shared storage, load balancer, etc.) you will need within your cluster.
If you only want high availability and not load balancing then I recommend you to not use Windows Cluster. Better set-up two VMware servers with one virtual machine and then copy a snapshot of your virtual machine every few hours over to the second machine.
[1] I don't know about Win Server 2008
[2] http://www.drbd.org/
[3] http://www.linux-ha.org/
16GB? Are you mad? Anything beyond 1GB should be enough to handle 1000 unique visitors per day. If you want to virtualize the system and have a separate web- and database server, 2GB should be enough already, if you ant to go further and have a separate virtual mail server in there, 2GB is still sufficient and 3GB is plenty.
Haproxy is better than Pound, IMO. It's lightweight, but handles immense load just as well as layer 3 load balancing (LVS), with the advantages of layer 5 proxying. It uses the latest Linux APIs (epoll, vmsplice) to reduce context switching and copying to a minimum. It has a nice, concise stats module. Its logs are terse yet complete. It redirects traffic to a working server if one is down / overloaded.
Hi! we run a non-profit website that gets 100 million visitors a day on ~350 servers. we don't even use any "clustering" technology, just replication for databases, and software (LVS) load balancer in front of both app (PHP) and squids at the edge. but oh well, you can always waste money on expensive hardware and clustering technology. and you can always check how we build things
I'm sorry, but I have to say that. Don't be offended, please - sooner or later you will look at your submission and laugh really hard, but for now you need to realise that you said something very, very silly. A few people already politely pointed out that 1000 visitors a day is nothing - but seriously, it's such a great magnitude of nothingness that, if you make such a gross misintepretation of your expected traffic, you need to reconsider if you really are the right person for the job *right now* and maybe gain some more experience before trying to spend other people's money on a ton of hardware that will just sit there, idle and consume huge amounts electricity (also paid by other people's money).
I'm serving a 6k/day website (scripting, database, some custom daemons etc.) from a Celeron 1.5GHz with 1GB RAM, and it's still doing almost nothing. If you really have to have some load balancing, get two of those for $100 each.
This is Slashdot. Common sense is futile. You will be modded down.
1000 users a day? So what? That's less than one user a minute. Even if you assume they stay on the website for 20 or so minutes each, you're never looking at more than about 20 users at a time browsing content (there will be peaks and troughs, obviously). Now picture a computer that can only send out, say, 20 x 20 pages a minute (assuming you're visitors can visit a full page every 3 seconds) - we're talking "out of the Ark". Unless they are downloading about half a gig of video each, this is hardly a problem for a modern machine.
I do the technical side for a large website which sees nearly ten times that (as far as you can trust web stats) and it runs off an ordinary shared host in an ordinary mom-n-pop webhosting facility and doesn't cost anywhere near the Earth to run. We often ask for more disk space, we've never had to ask for more bandwidth, or more CPU, or got told off for killing their systems. Admittedly, we don't do a lot of dynamic or flashy content but this is an ordinary shared server which we pay for out of our own pockets (and it costs less than our ISP subscriptions for the year, and the Google ad's make more than enough to cover that even at 0.3% clickthrough). We don't have any other servers helping us keep that site online (we have cold-spares at other hosting facilities should something go wrong, but that's because we're highly pedantic, not because we need them or that our users would miss us) - one shared server does the PHP, MySQL, serves dozens of Gigabytes per month of content for the entire site, generates the statistics etc. and doesn't even take a hit. I could probably serve that website off my old Linux router over ADSL and I doubt many people would notice except at peak times because of the bandwidth.
Define "massive" too... this site I'm talking about does multiple dozens of Gigabytes of data transfer every month, and contains about 10Gb of data on the disk (our backup is now *three* DVD-R's... :-) ). That's *tiny* in terms of a lot of websites, but equally puts 99% of the websites out there to shame.
Clustering is for when you have more than two or three servers already and primitive load-balancing (i.e. databases on one machine, video/images on another, or even just encoding half the URL's with "server2.domain.com" etc.) can't cope. In your case, I'd just have a hot-spare at a host somewhere, if I thought I needed it, with the data rsync'd every half-hour or so. For such a tiny thing, I probably wouldn't worry about the "switchover" between systems (because it would be rare and the users probably don't give a damn) and would just use DNS updates if it came to it. If I was being *really* pedantic, I might colo a server or two in a rack somewhere with the capability for one to steal the other's IP address if necessary, or have DNS with two A records, but I'd have to have a damn good reason for spending that amount of money regularly. If I was hosting in-house and the bandwidth was "free", I'd do the same.
Seriously - this isn't cluster territory, unless you see those servers struggling heavily on their load. And if I saw that, I'd be more inclined to think the computers were just crap, the website was unnecessarily dynamic, or I had dozens-of-Gigabytes databases and tens or hundreds of thousands of daily visitors.
You're in "basic hosting" territory. I doubt you'd hit 1Gb/month traffic unless the data you're serving is large.
Obviously has shares in Kingston.
(16Gb RAM for 1k visitors? What kind of pages are you serving?)
"It doesn't cost enough, and it makes too much sense."
First, I suggest you read and think deeply about Moens Nogood's essay "So Few Really Need Uptime".
Key quote:
And that corresponds pretty well to my experience: the more effort people make to duplicate hardware and build redundant failover environments the more failures and downtime they experience. Consider as well the concept of ETOPS and why the 777 has only two engines.
sPh
If you're looking for a lightwheight open source loadbalancer with a lot of features, go for HAProxy. In my company we work with F5 Big IPs, Alteon, Cisco CSS which are the leading load balancers from the industry, they are really expensive and depending on the licence you buy, you won't have all the features (HTTP level load balancing, cookie insertion/rewriting). We first used HAProxy for POC and now we're installing it in production environnements, works like a charm on a linux box (debian and RHEL5) with around 600 users.
There is no way to be fully redundant unless you have independent power sources, which usually requires your backup systems to be geographically separated. In my experience, loss of power is the single most common reason for a system failure in a well designed system (after human error that is).
It's fairly trivial to install RedHat/CentOS based clusters, especially for web serving purposes.
There are a few components involved:
1) A heartbeat to let each node know if the other goes out.
2) Some form of shared storage if you need to write to the filesystem.
3) Some methood of bringing up services when it fails over.
A web server with a backend database is one of the canonical examples. You'd install the heartbeaat service on both nodes. Next, install DRBD (distributed replicated block device). Finally, configure the services to bring up during a failure. The whole process takes about an hour following instructions on places like HOWTOFORGE.
But 1000 visitors a day is not much. It's small enough that you could consider virtualizing the nodes and just using virtualization failover.
HAproxy (which is the one I use) has the ability to define "backup" servers which can be used in the event of a complete failure of all servers in the pool, even if there is only one server in the main pool. If you're trying to do this on the cheap, that may help. It also has embedded builds for things like the NSLU2, so it may be easy to run on an embedded device you already have.
"He may look like an idiot, and talk like an idiot, but don't let that fool you. He really is an idiot." - Duck Soup
Windows clustering allows for Active/Active clusters, so you CAN run the same service on two cluster nodes at the same time (with the exception of Exchange).
Setting up two servers to host VMWare guests and copying is not a good idea either - the HA tools for VMWare are expensive, and totally unneccessary for the proposed deployment. Without these HA tools, he would have to down his primary guest every time he wanted to make a snapshot.
We're talking about a very simple deployment here - HTTP and FTP. You don't even need clustering or a dedicated load balancer - instead, try using round-robin DNS records to do some simple load balancing, and then use a shared storage area as your FTP root (could be a DFS share for Windows or an NFS mount in Linux). This would give you a solid two-server solution that works well for what you're trying to accomplish, and adding servers would be trivial (just deploy more nodes, and add DNS records to the list).
If it grows much larger than 2 nodes, you might consider an inexpensive load-balancer; Barracuda sells one that works well and will detect a downed node.
Clustering for this job is totally unnecessary though. You're wasting your time by looking into it.
My favorite (the name seals the deal for me) is http://www.ultramonkey.org/
It's probably more complicated and overkill for what the poster needs, but it worked great for us. We used this years ago for transaction processing (~100,000 transactions an hour, not too busy) on a couple old HP NetServers with 1GB RAM each.
At night I drink myself to sleep and pretend I don't care that you're not here with me
Use Google. Why spend all that money buying up equipment for a non-profit that could be spent on your REAL mission.
Do it in Google sites and dump the data center. I even think google offers google apps for free to non-profits.
Everybody keeps saying that 1000 unique visitors is peanuts and starts talking about Apache, etc. The OP mentions FTP as well, and didn't say if those 1000 users will all be regularly FTP'ing megabyte files or if they will be almost exclusively using HTTP with the occasional FTP download. If the former is the case, without analyzing it too much, it seems like this would be too much traffic for a single server to handle, no?
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
OpenBSD PF or hoststated seem like the simplest solutions to me.
Stop Computers/Cars Analogies on S
2 boxes for hardware failover will do you fine, if you are worried about HA the its the COST of downtime that you are worried about (i.e. down for an hour exceeds $1000 in lost revenue) which will justify the solution. Don't just drive availability to five nines because you feel its cool, do it because the business requires it.
This is something that is rampant: techies tend to overestimate the value of uptime.
Sure, it's sexy to have high availability this and redundant that, but unless your company is pulling down at least $1,000,000 per year or more in gross revenues, it's hard to beat the 3 to 4 nines or so uptime delivered by a good quality, whitebox server running Linux. Something like this unit would deliver excellent performance and excellent reliability at a very low cost.
How much does an hour of downtime actually cost your company? Be honest. If you had to tell your customers: "we were down for 2 hours because a software update caused us to have to ..." what would it actually cost your company? Especially if it only happened every year or so? In my experience, even in fairly stiff production environments, there has been no cost at all. We've maintained about 99.95% uptime for the past 3 years, with 1 "incident" every year or so, with no cost at all. In fact, our company has a good reputation for availability and support!
So don't spend money on sexy hardware with lots of blinkie lights and cross-connects, which often decrease your reliability by introducing unnecessary complexity.
Instead, spend money on your hosting. Don't *ever* host it in-house. Ever. Get a first-tier hosting facility, with redundant network feeds, power, and staff who give a damn. Don't be afraid to pay for it, because it will probably save you money, anyway. You'd be amazed at how price-competitive top-notch hosting farms can be!
Make sure to get to know the on-site techies on a first-name basis, give 'em a six-pack of their favorite beverage, and thank them profusely when they do anything for you. The goodwill these types of things can bring will work wonders for you down the road.
And remember:
2 nines is 3.65 days of downtime per year. .365 days of downtime per year (~ 8 hours) .0365 days of downtime per year (~ 45 minutes)
3 nines is
4 nines is
It's a very, very rare case indeed where 3-4 nines of uptime isn't completely sufficient.
And 1,000 unique visits per day? Pssht. Unless you are doing some pretty ferocious database stuff, (EG: joins across 12 tables with combined inner/outer/composite joins) the aforementioned server should do the job just wonderfully.
DON'T FORGET BACKUPS! And backup your backups, because backups fail, too.
I have no problem with your religion until you decide it's reason to deprive others of the truth.