Slashdot Mirror
Building Big Sites on a Budget
Joe Mamma writes: "There is a good article on Anandtech.com about how they upgraded their backend. They are running a bunch of AMD chips in their servers and make good use of the Linux Virtual Server Project software for their load balancers. Anyway its a good read for those who are looking to expand their web backend on a budget."
Interesting read. I'm sure some of you have read about this before, but the new article talks about budget constraints, among other topics. The project was tightly budgeted to begin with, but they still managed to bring it in way under budget. Some of the strategies Microsoft used would apply to a small, independent website as well.
I think my backend is expanded enough. After all, /. all day and night
all I do is sit here are read
:-)
I must be dreaming, then.
But as you're discovering, on the back end -- the database machine, where failure is much less tolerable -- it is going to be much, much harder to spec out a machine. Your example of hardware RAID is the most suprising area where Linux simply falls down. Even though cards like Compaq's SmartArray RAID controller has working Linux drivers, there are no Linux utilities.
I am sitting 15' away from a Linux machine running with a DPT Century UW2 RAID controller and I've got no problem running the Unix utilities. I can shut drives down, tune both the RAID and cache performance, silence alarms, rebuild RAID volumes, you name it. (It looks like DPT was bought by Adaptec so that may not be the exact card, mine is a hardware cache/RAID controller)
Don't spew FUD. Hell even my ancient P90 had a DPT controller in it with working utilities and that was 5 years ago!
Sorry, I missed some of your trolling.
For far too many of the RAID controllers out there, reconfiguring or rebuilding the arrays either involves shutting down the machine and rebooting into a stand-alone utility, or "echo"ing pretty much undocumented commands into your /proc/scsi filesystem. If a drive fails, you will have downtime. The Linux RAID solutions might protect your data integrity, but they will not protect your uptime. I couldn't even imagine trying to hot-plug some new drives into the array, and then resizing the live filesystem when I needed more room. (Someone will try to refute this, and will use the word "ReiserFS" in their post, but all I have to say is that there are a lot of possible things I couldn't imagine actually doing).
Hmmm. Reiserfs will support this but I'd want to layer LVM on it before Reiserfs. As far as I know ext2 won't handle online resizing. With LVM and Reiser though it will. I've been playing with it on my home system. In a month or two I'll upgrade the office fileserver to support it. We've already got the hotswap drives in and the hardware RAID/cache, so this will just complete the upgrade path.
Things like hot plugable CPUs, or network cards, or drive controllers, are simply non-existant on Linux. Stuff like monitoring utilities that will page you in the middle of the night when one of your redundant power supplies fail, or when one of your CPUs burns out, or when one of the drives in your storage arrays dies, simply don't exist. There's a lot of stuff that you need if you want to run 24x7 that just doesn't work in Linux yet.
Interesting. I get paged if my server goes down or the internal case temp gets too high. I don't see it as much of a stretch to page if a CPU temp gets too high. Watching for casefans and whatnot is just a matter of temperature monitoring. And I do this on an external server, how about that... The main fileserver could explode and I'd still get paged that something was wrong.
A big part of that is because of posts like yours, of course -- you don't want to spend more than $2,000.00. It's pretty damned hard to build a well supported server for $2,000.00, and the people who are willing to spend the money for real support often don't care as much about Linux. You're left with a lot of after-thought solutions -- stuff that got built for something else, and just happens to work on Linux by an almost happy accident.
I'll agree here: Five 9.1G UW2 SCA drives aren't (weren't) cheap. 64M of ECC cache memory weren't cheap, and neither was the hardware RAID/cache controller it goes in to. The 20G DAT wasn't cheap, either. I don't have redundant power supplies but the server isn't that critical that I can't bring it down for 15 minutes to swap out a power supply. I've never had a power supply die on me in 5 years of network administration but then again we have a 300lb online UPS which keeps most of the nasties away from the computers. What was absolutely mission critical was that the data was safe in case of drive failure, and that it could be expanded as needed.
In short, go away troll. I"ve got a perfectly maintainable Linux server which has worked superb over the last 6 years. Had a drive fail, had a CPU cook, never lost data. I don't remember arcane commands which are echoed to /proc, I run dptutil. All I seem to do on the server is keep an eye on the logs.
You are right about one thing though: You need to consider what you're doing this for before you put a price on it. I was able to get away with about $3000 to $3500 after all the bills came in. It hasn't got hot-swappable CPUs or power supplies, but that wasn't mission critical for me. The datastore is solid and that's what mattered.
K5, for example, has been able to take several direct Slashdottings on 1 VA Fullon box.
Now I like K5 as much as the next guy but I do NOT consider not being able to serve up pages as "taking a slashdotting."
I think it was last week or the week before when the last slashdotting took place over rusty's culture post. I couldn't get so much as a squeak out of K5 until hours later. Sure the server might have stayed up but it sure as hell wasn't serving up too many pages...
Enjoy! :-)
"We have the right to believe at our own risk any hypothesis that is live enough to tempt our will."
http://gabrielcain.com/
onboard video for x86 server boards is actually a Good Thing. It means I don't have to bother looking for the cheapest possible AGP card at a computer show because it would kill me to spend $$$ for features that a server box will never use.
onboard NICs aren't so bad either, though they tend to be intel which is sad. You'll need both onboard NICs and video if you want to go 1U..
Your Working Boy,
- Otis (GAIM: OtisWild)
Just check this page: Advertising. They wouldn't lie on that page (they claim >40 million page views monthly), it would give them a bad reputation with advertisers if they found out.
"I love my job, but I hate talking to people like you" (Freddie Mercury)
We serve between 200.000 and 250.000 dynamic page views / day from 1 single-cpu front-end box and 1 dual-cpu mod_perl box and have room for approx. 3 times higher traffic. In the end it's all down to programming, page size and cacheability ...
For a good example of an very scaleable configuration look at Google - their software must be extremely well designed.
"I love my job, but I hate talking to people like you" (Freddie Mercury)
So, how does a high-traffic NT-site such as anandtech compare to a high-traffic Linux-site such as slashdot? Both are serving dynamic content from database backends, both have the ability to add user comments to stories.
Does anyone have any numbers on the amount of traffic and the amount of servers both sites have?
It is my impression anandtech is running more and faster servers than slashdot (slashdot has around ten webservers, IIRC). I'm not sure about the amount of traffic they generate.
Anyone?
This is your sig. There are thousands more, but this one is yours.
Love the fact that the 905C's get their features used in Linux, just what I was interested in hearing. It is curious that even the linux vendors seem to like using the eepro.
zero-copy networking yum, just wish we could wait on these servers till Apache 2 shakes out with PHP support.
For 5 frontend webservers looking at 1Ghz AMD Athlons, 512MB RAM, with an ASUS KT133A motherboard. Looks like the 3COM 905C for a NIC and an IBM Deskstar rounds out that package nicely. $900/each with the 2U case.
The backend database server is tricker. The FastTrak 100 RAID controller, a nice IDE raid solution is not supported under Linux. What are the good (and cheap) alternatives for RAID 1 or 10? Will Dual 1GHz pentiums really beat out a 1.3 GHz athlon? How about a nice NIC that works under Linux? $2000 would be fine...
I suspect I'm not alone in wishing their were some good solid sites that had some recommeneded systems for Linux. As is, I end up wading through a lot of Windows tech sites to find that something is not supported under Linux? The hardware compatability lists don't descriminate between worthwhile products and overpriced junk. And it's be great to know this products works great and these manufactures actively track kernel development etc...
Pointer, comments and experiences welcome.
And now they want to test their upgrade with a good old front page Slahdot link.
This is kinda useless. Yes, they tell us that they are running 15 servers total all on 1Ghz PCs, but they do not tell you what kinda hits they take on it.
K5, for example, has been able to take several direct Slashdottings on 1 VA Fullon box. 1 box which does MySQL, Apache w/ Mod_perl, and plain image serving Apache. (DNS is handled by other boxen). We handle about 65,000 to 70,000 hits a day (on average, mod_perl only.. no images traffic) with that one dual processor box. Vs the two dedicated dual proc DB servers, 11 web servers, two load balancers, etc of Anandtech. And we're at 8 months uptime with our single server. Sounds a bit better than requiring a load balancer which has to remove downed NT servers from the pool..
I could theorize on how well their Cold Fusion/NT solution stacks up against my Slackware/Apache/mod_perl/MySQL solution IF they were so kind as to give info on hits. Without that, this is just another point-and-drool at some RAQmount stuff which performs a job somewhere, somehow.
--
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
It's unfortunate that the article doesn't go into further technical details, but it's an interesting and useful read all the same. In particular the round robin DNS good and bad sides and the linux virtual server project.
My personal preference would be to drop OpenBSD on the web servers. The security by default makes tweaking it to your secure settings pretty simple. Yeah, the performance isn't up there with FreeBSD or Linux, but to me it is adequate. If I'm throwing beige x86s at the problem anyway, I can just throw one more if I want to compensate for the speed.
:)
For the database, what are you doing? Are you deploying administrator entered data (in which case transactions are meaningless, go throw MySQL up there), or are you really getting and processing data.
One compromise (for added complexity) is to run two database servers. Put a FreeBSD or Linux Server with MySQL (I prefer PostgreSQL for development, but MySQL DOES Scream, Postgres is starting to annoy me on one of my platforms) hidden from the Public network. Load it with RAM and RAID 0 drive system. Just run the backup after you make changes, and you're good to go.
Then, for the live, interactive data, buy a real database server. Design it, pick your platform (DB2 or Oracle are both fine), and then pick your platform (IBM, Sun, and HP all make nice machines) and put it there. Hire someone who knows what they are doing to put this machine together and configure it, then DO NOT touch it.
This way you don't need to shell out big bucks for an Oracle Server for the content that MySQL is fine with. As you pay for the #Processors*MHz*RISC with the big boys for the database license, if MOST of your content is going to be "static dynamic" data (where you update once an hour/day/week, if that), there is no need for Oracle to touch it. Let Oracle process your transactions, and stick your "content" on MySQL.
Consider something, most database backed websites aren't really database systems. Most use a database as a convenient place to store info and retrieve it. With a "database system" I'm using relations to develop complex systems that can be cross-referenced with my keys, monitoring transactions, real-time changes, etc. If I am writing articles to dump on a server, yeah the DB is convenient, but Oracle isn't necessary.
As all of us here can admin a PHP/MySQL or PHP/Postgres solution, keeping the stuff that we work with there makes sense, then pay someone else for the real iron.
Moving from Solaris to NT just to overcome some problems with ColdFusion??? Why didn't they just dumped ColdFusion and moved to a better technology like PHP or JSP/Servlets? I know changing the scripting language needs a lot more effort than changing the O/S, but in the long run it pays back...
Donate free food to the hungry at The Hunger site.
Rather than upgrading all their hardware ($$$), they could've just switched to apache/php and upped the efficiency of their existing hardware. It is hardly surprising that they were having trouble with solaris Cold Fusion implementations. The engine is flaky enough on it's native platform!
;)
IMO, php, perl, or perhaps python (I have no experience with it, though) would all make better alternatives to Cold Fusion. (aka Server Side Scripting For Dummies, not that it doesn't have it's place)
I suppose they could also install a BSD or Linux to cut down on useless cruft (like GUI, etc) running on their web servers, (flame-retardant suit on) but then again I suppose many would argue that Linux contains a huge amount of cruft in the first place
Almost every car has a water pump. Any schmuck can design a $400 water pump that does it's job. A real engineer can design one that does it's job for 20 bucks.
As a network engineer, coder, or architect, your job is to make your client's project a success and deliver the most value possible. Your job is not to cover your ass, or to to get the project done with as little of your sweat as possible.
That's the unstated contract that drives tech jobs to get the salaries they do, and the "Let's just throw Solaris at it," or "Microsoft is easier - they say so!" attitude is undermining the tech inustry.
For web hosting, networking and network servers, Linux or a BSD on X86/AMD gives more bang for the buck than any other offering right now. Even though they have less froofy tools and don't have as much in the clicky GUI department. Yes, you have to be careful with hardware choices - You don't with NT? Yes, you need to know what you're doing - why do you have this job if you don't know what you're doing?
MySQL does a great job, no matter how much it may suck "theoretically", or how it may fail the ACID test. Compare the number of MySQL horror stories to the number of MySQL positive anecdotes you hear. It works, it's fast, it does the job.
Building solid, cheap boxen and deploying them in a stable configuration at low cost is an art. Call them 'frankenstien boxes' if you want, but they get the job done, and they let me come in with quotes at half of what my competiton does, and the stability and value get me return business.
Buying Solaris and having them do all the heavy lifting for you generally means you haven't done the value calculation for your clients. Solaris is good, and Sun hardware is good - but neither is *that* good.
Buying Microsoft - I hate to bash MS for the sake of bashing MS. Win2k *is* thier best offering yet. But it still bluescreens, DLL Hell still exists, it still needs to be reformatted every few months, it costs too much, it eats RAM, and not being able to remotely maintain it or to see the source so I can figure out what's causing strange behavior, coupled with MS' predatory business practices just makes it a non-option for me. YMMV.
The days of the Mercedes dot-bombs is over, and tossing cash at problems isn't the way things are being done today. Look around you and start gearing up for the Toyota world of working fast, smart and as cheaply as possible to get the job done.
From what the article mentions, this isn't a matter of improved stability over linux, rather that Coldfusion for windows was more stable than the less mature Linux port.
<rant>
I love comparative terms like "more stable"...they do a great job of avoiding the messy details, like how ColdFusion (my experience is on NT) will take your server DOWN if you try to perform just about any kind of simple regular expression matching. To say nothing of the fact that you're not going to get it to return the matches...no, you get an offset of the starting point of the first match. How bloody useful!
Functions you say? What are those???
</rant>
Can you tell I currently have to "code" in CF for my day job? Thankfully our next project there will be done in PHP or Perl. The (new) lead programmer finally got the suits to realize that ColdFusion, and originally hiring a guy who didn't know the first thing about programming to put it together, is why their site sucks.
I'll see you in hell, ColdFusion...
"Sweet creeping zombie Jesus!"
"Sweet creeping zombie Jesus!"
-The Professor, Futurama
under different brand name, Microsoft proudly present Clustering on a Budget - and the meaning of the term on a Budget in Microsoft's dictionary($28,000 for two clustering PC).
Nice little chunk of money saved by using Linux virtual servers over Arrowpoint, however I would like to know how a high content site would hold up with a lot of those perl scripts running to cache, one of the possible problems you won't find with Arrowpoint, Alteon Ad Directors, Netapps, not to say they're better, but the article did mention "Big Budget", aside from that some information on traffic handling would have been nice to show, e.g. amount of data passed into the network would give insight as to why they may have chosen to go via certain routes (not routers, or routing protocols.. choices) versus others.
;) but then again this was a semi "Big Budget" article, not a Poor Man's Network which in my case would be my Cisco 1xxx series running Zebra and GBGP (what you know about that.. Ghettotized BGP werd), 400mhz i386 running OpenBSD for the website, my spanking U1 for db stuff, ghettotized rj45's I found, with stolen bandwidth running out "Moving Day to Day Networks" run from my garage, and a C64 for DNS (fear)
I remember some of the guys where I'm at did some overhauling, and when we were doing the firewalls, instead of ordering 4-5 Nokia's or looking into other fw's, we ended up getting one Nokia 650, and since we were running FreeBSD we threw on ipf on all the boxes and created rules to eliminate the load of ACL's, and the FW load which was actually cheaper than buying x amount of new firewalls, and since we jumpstarted most of the machines, we had a slew of tightened security scripts for Sun, and BSD's to have an auto locked down network no matter how much shit was upgraded.
One of the things I wonder about though is the "dual processor" factors, which has many people going gah-gah over. Dual 700mhz's may sound nice, but to only serve up web content I wonder how is that better than just 1 700mhz chip or a 1ghz Athlon for that matter (anyone care to comment?)
As for switching from Oracle to SQL7, sounds like a good move, however again there's no mention of how much data goes into their database, so while it may suit them, what about mega sites like Yahoo, I wonder how they would stand up to SQL over BSd's, Linux versus a nice Sun E10k running Oracle?
Well they certainly have a pretty cool network, I wish they would have included actual network information as well such as router info, traffic stats, etc., now they would have blew my mind had they said, they're running strictly Zebra on a Nix box versus a Cisco or Juniper
Blackbox Themes
360 degrees of Karma
Actually they didn't switch to NT from linux. They switched from Sun Solaris to NT. Furthermore, they had switch over to NT because the Cold Fusion port to Solaris was unstable, not because of the operating system.
As far as I can recall from reading their previous server article a year ago the load balancing nodes are the first time Anandtech has used Linux.
Load balancing via IP routing tricks is kind of nice. Mosix goes one step further and allows live processes to migrate across a cluster. Experimental add-ons also will do socket migration and havedistributed file system support. I think that's the kind of approach to clustering you are going to see in the long run.