Slashdot's Setup, Part 1- Hardware

As part of our 10-Year anniversary coverage, we intend to update our insanely dated FAQ entry that describes our system setup. Today is Part 1 where we talk mostly about the hardware that powers Slashdot. Next week we'll run Part 2 where we'll talk mostly about Software. Read on to learn about our routers, our databases, our webservers and more. And as a reminder, don't forget to bid on our charity auction for the EFF and if you are in Ann Arbor, our anniversary party is tomorrow night.

CT:Most of the following was written by Uriah Welcome, famed sysadmin extraordinaire, responsible for our corporate intertubes. He Writes...

Many of you have asked about the infrastructure that supports your favorite time sink... err news site. The question even reached the top ten questions to ask CmdrTaco. So I've been asked to share our secrets on how we keep the site up and running, as well as a look towards the future of Slashdot's infrastructure. Please keep in mind that this infrastructure not only runs Slashdot, but also all the other sites owned by SourceForge, Inc.: SourceForge.net, Thinkgeek.com, Freshmeat.net, Linux.com, Newsforge.com, et al.

Well, let's begin with the most boring and basic details. We're hosted at a Savvis data center in the Bay Area. Our data center is pretty much like every other one. Raised floors, UPSs, giant diesel generators, 24x7 security, man traps, the works. Really, once you've seen one class A data center, you've seen them all. (CT: I've still never seen one. And they won't let us take pictures. Boo savvis.)

Next, our bandwidth and network. We currently have two Active-Active Gigabit uplinks; again nothing unique here, no crazy routing, just symmetric, equal cost uplinks. The uplinks terminate in our cage at a pair of Cisco 7301s that we use as our gateway/border routers. We do some basic filtering here, but nothing too outrageous; we tier our filtering to try to spread the load. From the border routers, the bits hit our core switches/routers, a pair of Foundry BigIron 8000s. They have been our workhorses throughout the years. The BigIron 8000s have been in production since we built this data center in 2002 and actually, having just looked at it... haven't been rebooted since. These guys used to be our border routers, but alas... their CPUs just weren't up to the task after all these years and growth. Many machines plug directly into these core switches, however for certain self contained racks we branch off to Foundry FastIron 9604s. They are basically switches and do nothing but save us ports on the cores.

Now onto the meat: the actual systems. We've gone through many vendors over the years. Some good, some...not so much. We've had our share of problems with everyone. Currently in production we have the following: HP, Dell, IBM, Rackable, and I kid you not, VA Linux Systems. Since this article is about Slashdot, I'll stick to their hardware. The first hop on the way to Slashdot is the load balancing firewalls, which are a pair of Rackable Systems 1Us; P4 Xeon 2.66Gz, 2G RAM, 2x80GB IDE, running CentOS and LVS. These guys distribute the traffic to the next hop, which are the web servers.

Slashdot currently has 16 web servers all of which are running Red Hat 9. Two serve static content: javascript, images, and the front page for non logged-in users. Four serve the front page to logged in users. And the remaining ten handle comment pages. All web servers are Rackable 1U servers with 2 Xeon 2.66Ghz processors, 2GB of RAM, and 2x80GB IDE hard drives. The web servers all NFS mount the NFS server, which is a Rackable 2U with 2 Xeon 2.4Ghz processors, 2GB of RAM, and 4x36GB 15K RPM SCSI drives. (CT: Just as a note, we frequently shuffle these 16 servers from one task to another to handle changes in load or performance. Next week's software story will explain in much more detail exactly what we do with those machines. Also as a note- the NFS is read-only, which was really the only safe way to use NFS around 1999 when we started doing it this way.)

Besides the 16 web servers, we have 7 databases. They currently are all running CentOS 4. They breakdown as follows: 2 Dual Opteron 270's with 16GB RAM, 4x36GB 15K RPM SCSI Drives These are doing multiple-master replication, with one acting as Slashdot's single write-only DB, and the other acting as a reader. We have the ability to swap their functions dynamically at any time, providing an acceptable level of failover.

2 Dual Opteron 270's with 8GB RAM, 4x36GB 15K RPM SCSI Drives These are Slashdot's reader DBs. Each derives data from a specific master database (listed above). The idea is that we can add more reader databases as we need to scale. These boxes are barely a year old now — and still are plenty fast for our needs.

Lastly, we have 3 Quad P3 Xeon 700Mhz with 4GB RAM, 8x36GB 10K RPM SCSI Drives which are sort of our miscellaneous 'other' boxes. They are used to host our accesslog writer, an accesslog reader, and Slashdot's search database. We need this much for accesslogs because moderation and stats require a lot of CPU time for computation.

And that is basically it, in a nutshell. There isn't anything too terribly crazy about the infrastructure. We like to keep things as simple as possible. This design is also very similar to what all the other SourceForge, Inc. sites use, and has proved to scale quite well.

CT: Thanks to Uriah and Chris Brown for the report. Now if only we remember to update the FAQ entry...

Comments

the powers that be

[ebolaZaireRules]

The hardware that powers slashdot?

I wanna know about the power that powers slashdot... are you really as green as the default colour scheme?

Redhat 9?

[eli pabst]

It'll be interesting to read the software section. It was surprising to see that they use an EOL'd version of Redhat (RH 9) that is no longer supported by Redhat. Granted, they're just webservers, but you'd think that would still require a lot of manually updating to keep things patched.

Possibly obtuse question

[athloi]

What determines why you run Red Hat 9 on some systems, and CentOS on others? Was BSD even considered? (You wouldn't run on Macs, would you?)

Re:Possibly obtuse question

[bogie]

Just because you don't have an ISO to download doesn't mean it isn't Free. How do you think CentOS gets made?

Reference Materials

[Ided]

This may be slightly off but I was wondering if anyone could recommend some good reading materials for setting up clustered sites or how to spread out work loads like they're doing with their systems.

Re:Reference Materials

[the_tsi]

http://www.linuxvirtualserver.org/ or anything about F5 BigIPs. Most of understanding load balancing is about understanding (a) how to fool layers above you in the OSI stack (switching on layer 4 through 7 -- particularly 7 -- can take a while to wrap your head around) and (b) the algorithms to pick which physical server gets the next connection (round robin, least connections, predictive, whatever).

bandwidth usage and cost?

[TwoWheelTomy]

wonder how much bandwidth slashdot is using and how much it costs.

Re:bandwidth usage and cost?

[anticypher]

For a 100Mbit/sec commit on a GigE connection for a full internet feed, I've been getting quotes from California of around US$10-$16 per Mbit, depending on the data centre and provider. For a 1 Gbps commit, the price drops to around US$6/Mbit. Those are prices from sales droids without any attempt at negotiating a better price.

For a site like /., a 95th percentile bandwidth of 50 Mbps for the month would cost between US$500 and $800, less if the total commit for all of OSDN's traffic was much higher. Add in hosting costs estimated around US$2,500 per month for a cage with room for 6 racks and matching electricity and cooling, and you can calculate how much operating costs /. runs.

After that, you have to count up all the amazing 7 figure salaries of Rob and the gang who keep things running :-)

the AC

Re:Windows?

[truthsearch]

Probably the same number of people that were elated to find out that Microsoft's site is cached using Linux. I found it deliciously amusing.

Why CentOS?

[bogaboga]

I am not saying that CentOS is any inferior at all but wonder why they chose it over all the possible serious systems in the Linux world. Is there anything CentOS does better than say OpenSUSE or Ubuntu/Debian and the rest?

Multiple master DBs

[atomic777]

"These are doing multiple-master replication, with one acting as Slashdot's single write-only DB, and the other acting as a reader."

Isn't that a contradiction? If you have only one write DB, why do you need multiple masters, aren't the other 6 just slaves at that point? Or are there separate master/slave pairs (I'm assuming these are MySQL databases)

Considered a CDN?

[xmpcray]

I was wondering if you ever considered using a CDN service like Akamai to serve content? Most of the big sites (Apple/MS etc) use it.

Re:Considered a CDN?

[Anonymous Coward]

Would not be good for me. I block Akamai in my hosts file. Sure some sites render badly as the CSS is not pulled from the Akamai server but that is my choice to make and I have work arounds for that.

I made this choice after coming across an odd occurrence on an order form. With Akamai blocked the order would not submit. With Akamai not blocked the order would submit. I looked at the form used and it sent the order to an Akamai server first instead of directly to the site I was ordering from. Up until that point the only thing Akamai was serving for this site was the CSS. In fact all the other pages I clicked next on during my information entry and item selection did not do this. I thought it very odd. I started blocking Akamai on my main machine ever since.

Akamai is nice and all. Very useful for a great number of things. However, seeing that my order was being sent to an Akamai server before being sent to the store server, I decided that Akamai did not need that much information about me while using that IP address.

The site in question was Best Buy online. I was ordering a laptop on a web only special. I ended up calling the order in.

backup?

[nido]

Well, let's begin with the most boring and basic details. We're hosted at a Savvis data center in the Bay Area. Do you ever worry that a big earthquake will hit and your datacenter goes offline? Do you at least keep an offsite backup?

Thanks

[debrain]

To the editors:

Thanks for this. It's really very interesting.

-B

Comment woes

[theantipop]

Any chance that with all that iron you can loosen up the crazy restrictions on comment posting? It can get rather ridiculous sometimes.

Re:Redhat 9

[Shakrai]

and is therefore completely unsupported.

I've been in the mission critical provider business (ISP, then an insurance agency, now in health care) for almost eight years now and I'd take community support (websites, forums, and *gasp* USENET) over corporate support any day of the week.

Yes, it's nice to be able to sell your project to the PHB/MBA by saying "We'll always have support if we have problems!", but anybody that has spent a few hours on the phone with their vendor and who realizes that they know the product better then the Level Four support guys do, comes to realize that support isn't the sliver bullet that it's presented as.

Have my vendors helped me out in the past? Absolutely. But will I upgrade away from a rock-solid system that has served me well for years merely to gain support with a newer version? No way in hell......

root passwd != externally vulnerable

[Steve Hamlin]

I can't wait for "Slashdot's Setup, Part 8 - Root Passwords"

And what would you do with them? Knowing the root password shouldn't get you into a properly configured and patched system.

I even remember one cracking contest where the owner of the machine gave out the root password to the target machine. (quick google: nope)

You could attack the bandwidth, or try to get physical access. But if Cmdr. Taco can't get in....