Slashdot Mirror
Introducing The New Slashdot Setup
the original version of this document was written by Andover.Net Alpha Geek Kurt Grey. The funny jokes are his. The stupid jokes are mine.
The Backstory
We realized soon that our setup at Digital Nation was
very flawed. We were having great difficulty administering the machines and
making changes. But the real problem was that all the SQL traffic was flowing
over the same switch. The decision was made to move to Exodus to solve these
problems, as well as to go to a provider that would allow us to scatter
multiple data centers around the world when we were ready to do so.
Meanwhile Slashcode kicked and screamed its way to v1.0 at the iron fists of Pudge (Chris Nandor) and CaptTofu (Patrick Galbraith). The list of bugfixes stretches many miles, and the world rejoiced, although Slashdot itself continued to run the old code until we made the move.
The Co-Loc
Slashdot's new co-location site is now at Andover.Net's own (pinky
finger to the mouth) $1 million dedicated datacenter at the Exodus
network facility in Waltham, Mass, which has the added advantage of being
less than 30 minute drive for most of our network admins -- so they don't have to fly
cross-country to install machines.
We have some racks sitting at Exodus. All boxes are networked together through a Cisco 6509 w/ 2 MSFCs and a Cisco 3500 so we can rearrange our internal
network topology just by reconfiguring the switch. Internet connectivity
to/from the outside world all flows through an Arrowpoint CS-800 (which
replaced the CS-100 that blew up last week) switch which acts as both a
firewall load balancer for the front end Web servers. It also so happens that
the Arrowpoint shares the same office building with Andover.Net in Acton so
whenever we need Arrowpoint tech support we just walk upstairs and talk to the
engineers. Like, say, last week when the 100 blew up ;)
The Hardware
- 5 load balanced Web servers dedicated to pages
- 3 load balanced Web servers dedicated to images
- 1 SQL server
- 1 NFS Server
All the boxes are VA Linux Systems FullOns running Debian (except for the SQL box). Each box (except for the SQL box) has LVD SCSI w/ 10,000 RPM drives. And they all have 2 Intel EtherExpress 100 LAN adapters.
The Software
Slashdot itself is finally running the latest release of Slashcode (it was pretty amusing being out of
date with our own code: for nearly a year the code release lagged behind
Slashdot, but my how the tables have turned).
Slashcode itself is based on Apache, mod_perl and MySQL. The MySQL and Apache configs are still being tweaked -- part of the trick is to keep the MaxClients setting in httpd.conf on each web server low enough to not overwhelm the connection limits of database, which in turn depends on the process limits of the kernel, which can all be tweaked until a state of perfect zen balance has been achieved ... this is one of the trickier parts. Run 'ab' (the apache bench tool) with a few different settings, then tweak SQL a bit. Repeat. Tweak httpd a bit. Repeat. Drink coffee. Repeat until dead. And every time you add or change hardware, you start over!
The Adfu ad system has been replaced with a small Apache module written in C for better performance, and that too will be open sourced When It's Ready (tm). This was done to make things consistant across all of Andover.Net (I personally prefer Adfu, but since I'm not the one who has to read the reports and maintain the list of ads, I don't really care what Slashdot runs).
Fault tolerance was a big issue. We've started by load balancing anything that could easily be balanced, but balancing MySQL is harder. We're funding development efforts with the MySQL team to add database replication and rollback capabilities to MySQL (these improvements will of course be rolled into the normal MySQL release as well).
We're also developing some in-house software (code named "Oddessey") that will keep each Slashdot box sychronized with a hot-spare box, so in case a box suddenly dies it will automatically be replaced with a hot-spare box -- kind of a RAID-for-servers solution (imagine... a Beuwolf cluster of these? *rimshot*) Yes, when it'll also be released as open source when its functional.
Security Measures
The Matrix sits behind a firewalling BSD box and an
Arrowpoint Load balancer. Each filters certain kinds of attacks and frees up
the httpd boxes to concentrate on just serving httpd and allows the dedicated
hardware to do what it does best. All administrative access is made through a
VPN (which is just another box).
Hardware Details
Type I (web server)
VA Full On 2x2
Debian Linux frozen
PIII/600 Mhz 512K cache
1 GB RAM
9.1GB LVD SCSI w/ hot swap backplane
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter
Type II (kernel NFS w/ kernel locking)
VA Full On 2x2
Debian Linux frozen
Dual PIII/600 Mhz
2 GB RAM
(2) 9.1GB LVD SCSI w/ hot swap backplane
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter
Type III (SQL)
VA Research 3500
Red Hat Linux 6.2 (final release + tweaks)
Quad Xeon 550 Mhz, 1MB cache
2 GB RAM
6 LVD disks, 10000 RPM (1 system disk, 5 disks for RAID5)
Mylex Extreme RAID controller 16 MB cache
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter
while () {
s/KURT/MARTIN/;
s/Andover/Adam/;
}
ROBLIMO: Not after we demonstrate the power of this station. In a way,
you have determined the choice of the web site that'll be slashdotted
first. Since you are reluctant to provide us with a URL, I have chosen
to test this station's slashdotting power...
on your home page on iVillage!
AC: No! iVillage is peaceful. We don't flame Linux on iVillage.
We only discuss travel and mystery novels. You can't possibly...
ROBLIMO: You would prefer another target? A commercial target? Then name the URL!
Roblimo waves menacingly toward AC.
ROBLIMO: I grow tired of asking this. So it'll be the last time. What is the URL?
AC: (softly) pcweek.com.
AC lowers her head.
AC: The FUD piece was posted on pcweek.com.
ROBLIMO: There. You see Lord Taco, she can be reasonable. (addressing
Hemos) Continue with the operation. You may post when ready.
INT EXODUS - MAIN SERVER ROOM
[MOFF KURT, a tall, confident technocrat, strides through the assembled geeks to the base of the shuttle ramp. The geeks snap to attention; many are uneasy about the new arrival. But Moff Kurt stands arrogantly tall.]
[The exit hatch of the shuttle opens with a WHOOSH, revealing only darkness. Then, heavy FOOTSTEPS AND MECHANICAL BREATHING. From this black void appears DARTH TACO, LORD OF THE SITH. Taco looks over the assemblage as he walks down the ramp.]
MOFF KURT:
"Lord Taco, this is an unexpected pleasure.
We're honored by your presence."
DARTH TACO:
"You may dispense with the pleasantries, Commander. I'm here to put you back on schedule."
[The commander turns ashen and begins to shake.]
MOFF KURT:
"I assure you, Lord Taco, my men are working as fast as they can."
DARTH TACO:
"Perhaps I can find new ways to motivate them."
MOFF KURT:
"I assure you, this station will be operational
as planned."
DARTH TACO:
"Andover does not share your optimistic appraisal of the situation."
MOFF KURT:
"But he asks the impossible. I need more geeks."
DARTH TACO:
"Then perhaps you can tell them when they arrive."
MOFF KURT: [aghast]
Andover's coming here?
DARTH TACO:
"That is correct, Commander. And they are most displeased with your apparent lack of progress."
MOFF KURT:
"We shall double our efforts."
DARTH TACO:
"I hope so, Commander, for your sake. Andover is not as forgiving as I am."
"This server is now the ultimate power in the universe. I suggest we use it!"
In MySQL you can decide per table if you want the table to be fast or take the speed penalty of making it transaction safe.
In MySQL, you do not have the choice turning on transactions and atomicity.
You have the choice of turning on features that they mistakenly label transactions and atomicity, but let's call a spade a spade here.
You use MySQL if you care about speed a lot, and don't care much about data integrity. That's a perfectly valid position, but let's not pretend it's some other position.
If you do care about data integrity, you use something other than MySQL, and find another way to achieve the speed.
--
One reason might be that mySQL is much faster than Oracle when it comes to building up and dropping connections.
This is really important for the web, because a typical web program will start by opening a connection and end by closing it. So you effectively have one connection for every hit that occurs.
Unless you do some fancy sharing of connections, this is going to be a big problem when you use Oracle. This forces Philip Greenspun to use TCL/AOLServer for his work, since it allows connections to stay up between CGI invocations.
In the mean time, I can open and close as many mySQL connections as I need to.
In addition, as I said in another post, he would probably have to rewrite the Slash engine to use another database; it's most likely very dependent on the mySQL API (as my programs are as well). We get a big payoff from this - far greater speed - so we pay the penalty of being stuck on one database unless we want to make a herculean effort to convert all the software we've already written.
D
----
We have several Fullons ourselves, and under heavy load these cards cease to function. I have to log in via an internal network interface and ifconfig the outside IP down then back up.
A sentence you'll never see on an Internet discussion board: "You know what? You're right."
Webservers: $22525
NFS servers: $21120
Database server: $25739
Being THE place for Natalie Portman and Hot Grits on the Web: priceless
There's some things money can't buy. For everything else, there's Slashdot.
---
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
I wish that the old slashdot server was still up...it would be interesting to visit every now and then, like going to a museum. :o)
Got Rhinos?
Be careful of MySQL documentation, and web site ... over the past couple of years, the PostgreSQL developers have tried to work with the MySQL folks to improve their various pages, especially their 'comparison' page ... we recently sent them a list of changes (the number of them eludes me, but it was alot) on just one section of the comparison page that listed a bunch of their types as 'SQL-Compliant' that have nothing to do with the standard, and a bunch of 'SQL Non-Compliant' that are required by the Standard ... As for 'a magnitude speed difference' ... check out PostgreSQL now. I would imagine that their 'tests' were based on several releases behind ... v7.0, that was just released, is several times faster then our last release, which was noticeably faster then v6.4 ... The point I'm making is that if you are going to judge something, judge it on your experiences, not on a competing vendors experiences... you might just be pleasantly surprised ... - scrappy@hub.org
Well, a typical Exodus facility isn't nuke-proof, but it's pretty damn close. I've toured one (in Herndon, VA) because our company is about to co-loc at it. Here's a brief rundown of the physical security:
You run into all this before you even see anything resembling a computer, apart from the terminals in the receptionist's enclosure. In the actual computer pens, you have the cages, and for the really paranoid, you can get a steel box with a biometric lock instead of a conventional cage.
To sum up...it would take a truly concerted effort to physically breach one of these facilities.
Aero
We can believe in you for 3 minutes, but beyond that, even the King of All Cosmos can't be expected to wait.
... the amount of relevant information has remained the same ever since /.'s been on a dusty ol' 386 with a 14.4kb/s pipe...
/. ...)
(that is not to say the trollers and such are not fun, which they are.. they're just not useful outside the context of
Forgive me if this has been asked elsewhere, but why did y'all choose those distributions for those servers? I'm genuinely curious; I'm unfamiliar with the large-scale differences between distributions. (My computer runs Mandrake... that decision was based on the single factor that my friend happened to have a Mandrake CD on him.)
>the licesnse that MySQL is under does not come anywhere close to meeting the open source definition
Exactly. So why not move to a product that has it, like Oracle/Informix/...., or, if you are going to spend the $, why not invest the $-time in PostgreSQL, a database that IS opensource?
Is there any reason beyond: MySQL is what we have been using, so now we will continue to use it?
MySQL has said:
On Roll-Back
"MySQL has made a conscious decision to support another paradigm for data integrity, "
Ok, fine, that is a design choice. If they wanted it(rollback), they would have designed it in.
PostgreSQL has rollback, and just needs database replication, and they would LOVE to see that feature.
So, why work with MySQL, other than "it is what we have always done" or "We didn't think of another option"? Are you hoping to have them change the licence?
If it was said on slashdot, it MUST be true!
Why was the choice made to use one beefy-as-hell SQL server instead of multiple lesser powered systems?
Scott Ruttencutter
We Apprentice Developers and Designers
I looked over the setup for those servers and frankly, they seems to be quite an overkill, for this site itself. Maybe I am wrong, but if this site handles over that many people, since I have no clue on how many been hitting the site, then the setup is not bad. But Quad Xeon Processors for MySQL? What wrong with Dual Pentium II/III or single Athlon MySQL server?
:)
I can understand the need for failover for the MySQL which is a major requirement, but the computer itself is quite a major overkill. I do adminstration for over 16 different servers over 3 different clients and we uses Mandrake Distros on those and only problem I usually finds is the MySQL can be really stressful when there is many people using it on the same server with the apache server. I am in process of moving the MySQL over to new delicated server so it can handle that among several servers, and maybe that might call for beefy processor to handle the load.
If the site is setup to handle... let say.... 100,000 hits a day, what server configuration is needed for this? With MySQL on it own server, of course.
-- Amazing how the Internet still humms along.... -- Dispite all the flaws of Micro$oft in their software!
Everybody Chant ..
:)
"We Want Pictures.. We want Pictures.. We want Pictures"
Seriously though: it would be nice to actually see this setup. Don't forget to have CowboyNeal give us an oh so sexy pose near the almost as sexy VA SQL server...
This series is very useful for people just starting to work in linux admin environments, even if it's just a personal server.
Lots of varying methods are discussed of how to properly protect or run a server, and now we get a real life scenario of what happens when the shit hits the fan.
Don't just publish the Hellfire series, package up this one too.
I wonder why there is a red hat box in the mix? what is the reason? Now I am a Debian bigot, but my guess is that so are you guys, is there somthing specific about Redhat and mysql that I don't know?
Second why not mylex cards in all the box's? mylex's new DAC110 SCSI cards are simply the fastest I have ever seen.
Why not Gigabit? I use it with Linux it works, it makes all that heavy duty hardware sing, 100mbit is just a passe :>
I am proud that you chose Potato for most of your box's
"think of it as evolution in action"
Just out of curiosity, wouldn't it be easier to use something like PostgreSQL (which is just as freely available) that already has rollback & atomicity than to pay the MySQL people to develop it? Didn't y'all read the article on here a few weeks ago, "Why not MySQL?"
__________________________________________________ ___
rooooar
I'd tell you, but I have a personal policy against helping lazy luddites who think they're taking some kind of principled stand because they don't visit sites that use <img> tags. At least read the damn FAQ linked to on their home page.
Cheers,
ZicoKnows@hotmail.com
Well, a 386 can't fill a 10meg pipe - a 486 could, but not a 386. Also, a 486 could only do it if it was only static content (i.e. pre-written HTML and images.) Slashdot isn't static content by any stretch of the imagination. The pages are dynamically generated for each user using data from MySQL. MySQL is very CPU/IO intensive, as is mod_perl, which generates the HTML.
--
I am sure everyone wants to know how much everything here costs. Here are the calculations for the linux boxes(info is off of the va linux custom configuration program):
webservers (type 1) = $4505 each
NFS server (type 2) = $7040
database server (type 3) = $25739
So the grand total is $68819. I haven't found the prices for the switches and firewall. I would suspect that the BSD box is close in price to the webserver (prob. a bit less).
Come play Heroes of Might and Magic Mini online.
Do you really need 1 Gig of ram on the web servers? Wouldn't the extra memory be better used on the DB server? You're probably not using more than 150MB for the httpd's.
Not true at all. I'm running a slash server which doesn't get very many hits (~3000 in the last two days, chump change compared to Slashdot) and right now httpd is using 270MB of RAM.
--
Who/what exactly is Exodus?
Exodus is one of the world's biggest (in terms of service capacity available) Internet Service Providers.
"We're going to need bandwidth. Lots of bandwidth."
Exodus specializes in having more bandwidth then most of the third world. They've got NAPs (Network Access Points, i.e., backbone connections) all over the continental United States, and a few outside the US as well. They link this all together using both external and internal networks. The end result is, most anywhere on the net that has a good connection, has a good connection to Exodus.
They provide servers. Do you need to host downloads for ten million users? Exodus can give you servers to do so.
They provide co-location space. If their standard server packages just won't cut it -- bring your own. They'll give you a rack, a dedicated co-loc cage, or a dedicated high security vault.
Their web page has a lot of graphics because they have a lot of pictures of their equipment and graphs of their capacity. It is actually justified. You may want to make a return trip.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Exodus is a very high-class co-location provider. They have lots of datacenters across the world which are all 1000's of square feet each and house sites like Hotmail, Lycos, etc. In each of their datacenters they bring in many pipes to all the major internet backbone providers as well as AOL, etc.
:)
They have, literally, 1000's of racks and TONS of machines. At the Sterling, Virginia facility which we moved into (which, BTW, is one of their newest and flagship facilities) I saw SGI Origin machines, countless Sun enterprise-level machines, mainframes, small machines, etc, etc, etc. They run OC-12, 48, or more between each of their centers. They offer "Datacenters within the Datacenter" which are little rooms constructed on the raised floor which offer a secure environment that companies can pay (lots!) for. They have fibre coming in to multiple points in most buildings, many generators, huge UPS's, fingerprint readers to get into the network rooms, etc, etc, etc.
It's pretty phat. Their service is top notch as well. I had to fly up there from Dallas and it was an immense pleasure (even though I had to work.
They're pretty awesome (and no, I don't work for them).
Jeff
I remember back when there was a cachedot.slashdot.org that maintained a replica in case the server was having trouble. Is there any chance that this could be implemented again but in a different location?
Of course such a thing would not need to be as powerful as the main slashdot systems, but would provide some additional backup in case of another DDOS or a network outage of some sort.
Sort of a "battle bridge" for those of you who remember the days when Star Trek was good. (startrek.version = ST:TNG)
Mycroft-X
There has been a small talk about this story in the debian-devel channel on IRC just some minutes ago, and of course the great question was:
Why isn't the SQL server Debian as well?
If there's any problem with Potato's MySQL, I think Debian would be pleased to hear, whether it's a bug report in the BTS or whatever.
Thanks
Posted by BSD-Pat:
;)
The web servers are running mod_perl, each process takes up alot of RAM (hrrrm...something to streamline with mod_perl/slash?)
So as a result, the machines that need the most amount of RAM are the webservers and MySQL machines.
essentially we need enough RAM to run up to the MaxClients set in apache *and* have file cache
-Pat
Others have said it well, but I'll add this: Exodus hosts Yahoo. 'Nuff said.
--
Sometimes it's best to just let stupid people be stupid.
You're using Win2000, IIS, and Active Server Pages. We all know it. Quit making stuff up.
I don't know about the other slashdottes, but I for one would love to see how the Slashdot network is configured topologically.
GNOME or KDE on the RH6.2 box?
What? servers don't need a GUI? Don't let
redmond find that out...