Slashdot's Setup, Part 1- Hardware

As part of our 10-Year anniversary coverage, we intend to update our insanely dated FAQ entry that describes our system setup. Today is Part 1 where we talk mostly about the hardware that powers Slashdot. Next week we'll run Part 2 where we'll talk mostly about Software. Read on to learn about our routers, our databases, our webservers and more. And as a reminder, don't forget to bid on our charity auction for the EFF and if you are in Ann Arbor, our anniversary party is tomorrow night.

CT:Most of the following was written by Uriah Welcome, famed sysadmin extraordinaire, responsible for our corporate intertubes. He Writes...

Many of you have asked about the infrastructure that supports your favorite time sink... err news site. The question even reached the top ten questions to ask CmdrTaco. So I've been asked to share our secrets on how we keep the site up and running, as well as a look towards the future of Slashdot's infrastructure. Please keep in mind that this infrastructure not only runs Slashdot, but also all the other sites owned by SourceForge, Inc.: SourceForge.net, Thinkgeek.com, Freshmeat.net, Linux.com, Newsforge.com, et al.

Well, let's begin with the most boring and basic details. We're hosted at a Savvis data center in the Bay Area. Our data center is pretty much like every other one. Raised floors, UPSs, giant diesel generators, 24x7 security, man traps, the works. Really, once you've seen one class A data center, you've seen them all. (CT: I've still never seen one. And they won't let us take pictures. Boo savvis.)

Next, our bandwidth and network. We currently have two Active-Active Gigabit uplinks; again nothing unique here, no crazy routing, just symmetric, equal cost uplinks. The uplinks terminate in our cage at a pair of Cisco 7301s that we use as our gateway/border routers. We do some basic filtering here, but nothing too outrageous; we tier our filtering to try to spread the load. From the border routers, the bits hit our core switches/routers, a pair of Foundry BigIron 8000s. They have been our workhorses throughout the years. The BigIron 8000s have been in production since we built this data center in 2002 and actually, having just looked at it... haven't been rebooted since. These guys used to be our border routers, but alas... their CPUs just weren't up to the task after all these years and growth. Many machines plug directly into these core switches, however for certain self contained racks we branch off to Foundry FastIron 9604s. They are basically switches and do nothing but save us ports on the cores.

Now onto the meat: the actual systems. We've gone through many vendors over the years. Some good, some...not so much. We've had our share of problems with everyone. Currently in production we have the following: HP, Dell, IBM, Rackable, and I kid you not, VA Linux Systems. Since this article is about Slashdot, I'll stick to their hardware. The first hop on the way to Slashdot is the load balancing firewalls, which are a pair of Rackable Systems 1Us; P4 Xeon 2.66Gz, 2G RAM, 2x80GB IDE, running CentOS and LVS. These guys distribute the traffic to the next hop, which are the web servers.

Slashdot currently has 16 web servers all of which are running Red Hat 9. Two serve static content: javascript, images, and the front page for non logged-in users. Four serve the front page to logged in users. And the remaining ten handle comment pages. All web servers are Rackable 1U servers with 2 Xeon 2.66Ghz processors, 2GB of RAM, and 2x80GB IDE hard drives. The web servers all NFS mount the NFS server, which is a Rackable 2U with 2 Xeon 2.4Ghz processors, 2GB of RAM, and 4x36GB 15K RPM SCSI drives. (CT: Just as a note, we frequently shuffle these 16 servers from one task to another to handle changes in load or performance. Next week's software story will explain in much more detail exactly what we do with those machines. Also as a note- the NFS is read-only, which was really the only safe way to use NFS around 1999 when we started doing it this way.)

Besides the 16 web servers, we have 7 databases. They currently are all running CentOS 4. They breakdown as follows: 2 Dual Opteron 270's with 16GB RAM, 4x36GB 15K RPM SCSI Drives These are doing multiple-master replication, with one acting as Slashdot's single write-only DB, and the other acting as a reader. We have the ability to swap their functions dynamically at any time, providing an acceptable level of failover.

2 Dual Opteron 270's with 8GB RAM, 4x36GB 15K RPM SCSI Drives These are Slashdot's reader DBs. Each derives data from a specific master database (listed above). The idea is that we can add more reader databases as we need to scale. These boxes are barely a year old now — and still are plenty fast for our needs.

Lastly, we have 3 Quad P3 Xeon 700Mhz with 4GB RAM, 8x36GB 10K RPM SCSI Drives which are sort of our miscellaneous 'other' boxes. They are used to host our accesslog writer, an accesslog reader, and Slashdot's search database. We need this much for accesslogs because moderation and stats require a lot of CPU time for computation.

And that is basically it, in a nutshell. There isn't anything too terribly crazy about the infrastructure. We like to keep things as simple as possible. This design is also very similar to what all the other SourceForge, Inc. sites use, and has proved to scale quite well.

CT: Thanks to Uriah and Chris Brown for the report. Now if only we remember to update the FAQ entry...

Comments

Windows?

[mseidl]

I'm like sooooooooo surprised you guys aren't running nt4 boxes. IIS was this sh!t back in the day

Re:Windows?

[QBasicer]

Do you know how many people would be heartbroken if they found out Slashdot was run off windows?

Re:Windows?

[truthsearch]

Probably the same number of people that were elated to find out that Microsoft's site is cached using Linux. I found it deliciously amusing.

Re:Windows?

[ObsessiveMathsFreak]

Not as many people who were traumatized to hear that Slashdot is run on a Dell.

Re:Windows?

[Shakrai]

<lame joke>But dude, your getting a Dell!</lame joke>

Re:Windows?

[Anonymous Coward]

But dude, your getting a Dell!

Re:Windows?

[LingNoi]

You can't come out with a comment like that and not give me more reading material to indulge in!

Re:Windows?

[BadMackTuck]

Microsoft isn't running linux servers, they're using Akamai who does.

http://news.netcraft.com/archives/2003/08/17/wwwmicrosoftcom_runs_linux_up_to_a_point_.html

Re:Windows?

[CCFreak2K]

Believe it or not, Dell's server hardware and support is worlds better than their "consumer-grade" products and support.

Re:Windows?

[TClevenger]

Yeah, but they have their problems, too. Like a Dell Poweredge we retired recently. The BIOS reports that the main system fan isn't working (though it obviously is), and you can't get past the "Press F1 to continue" unless you're there at the console. It NEVER times out. Even the latest BIOS release still has this bug.

Sucks that you can't reboot the thing remotely--it sits at that screen until you drive down there and hit "F1"

Re:Windows?

[bofkentucky]

Shell out for the drac card if you need/want remote reboot/console

Re:Windows?

[Anonymous Coward]

your getting a Dell

"you're".

Re:Windows?

[Kadin2048]

Yeah, but that's just a pretty crappy design, that a server -- which is designed to be stuck in a rack somewhere -- would ever require you to physically hit a button the console just to get past a boot prompt, when the option to continue booting existed. I can understanding hanging at the console prompt if there's a fatal error (no system, memory failure, whatever), but if the option to continue booting exists, it should always time out to that eventually. In a server that can't be accessed physically without a lot of trouble, booting up is the 'softer' of available failure modes. Waiting stupidly for someone to press a key on a local PS/2 keyboard is not.

Re:Windows?

[Anonymous Coward]

...was? ... what if it still is?

Re:Windows?

[bofkentucky]

Personally I feel dell should get with the program an provide drac on all systems. In reality the customers with big clusters with very small nodes, don't really could care less about remote management of the individual nodes and dell makes their money off those high volume sales.

Re:Windows?

[SnowZero]

What the GP said "Microsoft's site is cached using Linux" is completely consistent with what you said. He didn't say anything about serving, just the caching, which is what Akamai does (using Linux).

Re:Windows?

[Anonymous Coward]

Anyone running servers in Remote data centers without a maintanance contract with the Vendor (Dell) deserves everything they get. Dell allows you to buy 6 years of warranty on ANY poweredge server so your inability to shell out the dough for either a service contract or the DRAC card means your the fool not Dell.

Re:Windows?

[that this is not und]

The server obviously has a hardware fault in it somewhere. A good server fails the way that one does, which is to break down and require someone to do something physical about it. A bad server design is one where it just blunders forward and burns out, or worse, it continues to operate in a broken fashion, corrupting large databases as it runs.

Booting up is NOT the 'softer' of available failure modes. You're supposed to fix it, ya know, not just drive in and hit a key on it.

Savvis

[garethwi]

Nice to see you're hosted by a Microsoft Gold Partner. That's a benchmark of quality.

Re:Savvis

[Anonymous Coward]

They've changed hands several times and names even more times since we moved in.

Re:Savvis

[pdm]

Who's your quality hosting provider? How many racks do you rent for your high traffic needs?

Re:Savvis

[analogueblue]

Yeah, Savvis has been a total nightmare whenever I've had to deal with them. Although I could just be unlucky.

Re:Savvis

[bunco]

Twice, actually. Slashdot is hosted in an Exodus legacy data center. Exodus was bought by Cable & Wireless who then sold their US network assets to Savvis.

Depending on who you talk to, you'll get different responses about Savvis. This is mainly due to the heritage of various customers. i.e. Savvis/Bridge/Intel vs Exodus reputation.

Savvis is actually the conglomeration of _many_ companies.

Exodus == (Exodus, AIS, Arca, Cohesive, Network-1, Global Center)
C&W US == (MCI (IP backbone), Exodus, Digital Island)
Savvis == (C&W US, Intel Hosting, Bridge Networks)

Redhat 9

[Anonymous Coward]

Tell me that's a hilarious joke...

Re:Redhat 9

[Anonymous Coward]

i always thought they were running some archaic variant of bsd. im kinda sad now.

Re:Redhat 9

[Anonymous Coward]

Second only to slashcode.

Re:Redhat 9

[MisterFuRR]

If it works, and theres no need to change -- why introduce unknown incompatibility...its a production network -- not your home box.

Re:Redhat 9

[eln]

I wouldn't use it for a production system because it was end-of-lifed like 3 years ago and is therefore completely unsupported. I don't think I'd want to run a website that (presumably) generates quite a bit of revenue on ancient unsupported software.

Re:Redhat 9

[idontgno]

Amen.

Although, in the words of a wise man, "The singular of data is not 'anecdote'", I'll cite my own little war story data point. It's trivial, of course, as every story told by a 6-digit /. ID is going to be, but still...

My little bitty household server had been running RH variants from 5.something onwards, surviving numerous hardware transplants and live upgrades all the way up through RH 9.

It was running pretty smooth but I felt like it was getting harder to find RPMs packaged for it, and security updates seemed to be coming more rarely. So I went for one more live update: to Fedora Core 5.

Huge mistake. Damn near bricked the machine. I eventually got minimal useful service running, after $DIETY-only-knows how many RPM removals and replacements to fix broken dependencies and horked up library versions and the like.

As soon as I got the beasty running in limp-along mode, I started scrounging the bits and bobs to replace the box. The new server is live, running slightly newer hardware (most of it from this century, even) and CentOS 5. So far, so good, but no more live upgrades for me.

Re:Redhat 9

[cyphercell]

seems to me like slashdot has been supporting it just fine.

Re:Redhat 9

[Anonymous Coward]

The boxes have been up for 3 years. What sort of support do you think they need, that they don't have?

Rule #1 of a mission-critical application is "if it ain't broke, don't fix it."

Re:Redhat 9

[Shakrai]

and is therefore completely unsupported.

I've been in the mission critical provider business (ISP, then an insurance agency, now in health care) for almost eight years now and I'd take community support (websites, forums, and *gasp* USENET) over corporate support any day of the week.

Yes, it's nice to be able to sell your project to the PHB/MBA by saying "We'll always have support if we have problems!", but anybody that has spent a few hours on the phone with their vendor and who realizes that they know the product better then the Level Four support guys do, comes to realize that support isn't the sliver bullet that it's presented as.

Have my vendors helped me out in the past? Absolutely. But will I upgrade away from a rock-solid system that has served me well for years merely to gain support with a newer version? No way in hell......

Re:Redhat 9

[Shakrai]

Rule #1 of a mission-critical application is "if it ain't broke, don't fix it."

Slashdot is mission-critical? That's what I'll tell my boss the next time he asks why I'm spending so much time on it ;)

Kidding aside though, I'm rather fond of the old joke that I found in 'fortune -o'. "Working computer hardware is a lot like an erect penis. It stays up as long as you don't fuck with it."

Re:Redhat 9

[Shakrai]

I'll cite my own little war story data point. It's trivial, of course, as every story told by a 6-digit /. ID is going to be, but still...

You know what's worse then having a six digit UID? Having an old five digit UID in the 20,000 range that you don't remember the password or e-mail address on :( I even had a running e-mail exchange with CmdrTaco about two years ago trying to guess the e-mail address out of the shitload of old ones that I used to use. No go :(

Hell, wait for the 25th anniversary and our six digit UIDs will be worth something ;)

Huge mistake. Damn near bricked the machine. I eventually got minimal useful service running, after $DIETY-only-knows how many RPM removals and replacements to fix broken dependencies and horked up library versions and the like.

That's why I'm still running Slackware for both my desktop and servers. Sure, it's more work, but it's raw unfiltered Linux, and if I fuck it up I know exactly who to blame ;)

Re:Redhat 9

[mrsteveman1]

I'll second that, if bugs are found in the various software available over IP, there's no updates coming from RH, ever.

That means if something significant is found the site owners must backport it or leave it as is.

Re:Redhat 9

[Anonymous Coward]

if you think that's funny... we're still running rh7.3 ... talk about a nightmare.

Re:Redhat 9

[WhatAmIDoingHere]

How do you BRICK a computer short of taking an axe to it? Boot from install media and reinstall. If the hard drive is shot, the hard drive is shot. But a dead HDD doesn't mean it's bricked. I can see maybe fucking up a BIOS upgrade but even with that there are ways to undo the damage.

You people keep using the word "brick" to refer to "broken software that can easily be reinstalled."

Re:Redhat 9

[Anonymous Coward]

It is to the people whose mission it is to run Slashdot, and who derive a paycheck therefrom.

Re:Redhat 9

[Macrosoft0]

you can brick a computer if a bios update screws up. you cant even re-flash the bios without a working bios already on the mobo.

Re:Redhat 9

[Anonymous Coward]

How do you BRICK a computer short of taking an axe to it? ... You people keep using the word "brick" to refer to "broken software that can easily be reinstalled."

I'm thinking it was the realization that a modern version of an OS would barely run on his dated hardware (thus rendering it "useless" for his needs). I think he just wanted to get a supported (ie, newer) version limping along (and not unusable) until he could upgrade hardware??

Re:Redhat 9

[Anonymous Coward]

Obviously you weren't a valued customer. I work for one of the big news companies. When we call IBM, SUN, Rackable, Novell, Oracle etc... we have someone at our site in most cases in about an hour. Of course our setup is very custom, so occasionally dev teams for a product will visit our our site for weeks on end to fix a problem in their code. It's such a regular thing that we sometimes joke we should be paid for research and development for several of the products we use.

Of course that doesn't mean we don't make massive use of the open-source community and contribute to it in fairly major ways in some cases. But to say that enterprise support is useless is tantamount to saying you've never really gotten top-tier support.

Re:Redhat 9

[dissy]

Yes, it's nice to be able to sell your project to the PHB/MBA by saying "We'll always have support if we have problems!" I've always said:
If your goal is to get your system back up fairly fast, and fix the problem well, community support is best.
If your goal is to have someone to drag into court when it breaks, corporate support is the only game in town.

People and companys do have different goals in that area.

As silly as it might sound, some people/companys don't really care if their system is fixed fast (or at all) as long as they can recover the costs in court.

Re:Redhat 9

[Anonymous Coward]

Tell me that's a hilarious joke... I think Linux is choice for enterprise for another reason. There is no actual "EOL" like Microsoft/Apple dictates. Apache got performance issue? Compile the newer version and install. Not like Apache autoconfigure will say "You got EOL, I am not getting compiled" :)

Re:Redhat 9

[Znork]

"If your goal is to have someone to drag into court when it breaks"

And throw even more good money after the bad, you mean. Obtaining actual damages representing fictional costs of downtime is, as far as I know, a mirage. I dont think I've ever heard of a corporate support contract ever actually resulting in any damages (and I suspect that any such contract that would actually be enforcable would cost exactly the max penalty sum plus costs). Usually you'll just get a best-effort out of the vendor, so it pretty much amounts to hand holding.

So the moral of the story is, if it costs you more than you can afford to have downtime, then you'd better design sufficient redundancy into your systems that you dont have it. Or you should have alternate or manual fallbacks to mitigate the damages.

"As silly as it might sound, some people/companys don't really care if their system is fixed fast (or at all) as long as they can recover the costs in court."

Yep, and that's probably part of why courts frown upon punitive damages. If the company really cared about the system they'd do what they could to avoid the problem/resolve it asap. If they come to the court whining about a contract, the issue apparently wasnt that important to them...

Re:Redhat 9

[Shakrai]

Of course that doesn't mean we don't make massive use of the open-source community and contribute to it in fairly major ways in some cases. But to say that enterprise support is useless is tantamount to saying you've never really gotten top-tier support.

I don't recall saying it was useless. I've had "top-tier" support from Cisco when I worked at the ISP. I've had "top-tier" support from my vendor for the agency management system we used at the insurance agency.

What I do recall saying is that "it's not a sliver bullet" and "I wouldn't upgrade away from a rock-solid system just to gain support"

Re:Redhat 9

[WhatAmIDoingHere]

Right but he didn't "brick" the computer. A slow computer isn't "bricked."

Re:Redhat 9

[bogie]

I know the feeling. I lost my old 4 digit UID many years ago.

Re:Redhat 9

[Kalriath]

In reality, virtually every motherboard has the Award BootBlock BIOS (or some equivalent) which loads code to do nothing but drive I/O on the removable media and flash the BIOS. Bricking a machine nowadays with a failed BIOS update just isn't possible. The more modern machines will actually notice the failed BIOS and flash it back from the second BIOS with no downtime, and a single reboot.

Re:Redhat 9

[Chainsaw76]

Not recommended, but I have had to do it...

There is the old Hot Bios swap.. Pop out the Bios chip, pop in a donor Bios chip that works, but don't seat it all the way.

Boot.. Then with the machine still on remove the donor Bios, Swap in the bad bios in, and reflash it...

Power off, Reseat properly, and power on.. No more brick.

-Jason

Re:Redhat 9

[Anonymous Coward]

Redhat 9 kicks arse. So much so that it's probably the reason they dropped it, people would keep using it! And only it!

Also, you can compile the latest greatest vanilla 2.4 kernel on it and all is well if you use redhats config.

I chose to continue to use it in a new deployemnt of 20 plus server in-house (at a 200 million dollar public company) well after end-of-support was announced.

It just works.

Re:Redhat 9

[hostyle]

Thank you Captain Pedant for saving all of us and people everywhere from needless wasted words, unnecessary sentences and ceaseless repetitive re-reading.

Re:Redhat 9

[iphayd]

Keep taking the little blue pill and you can believe that.

Re:Redhat 9

[iphayd]

There used to be a nice bug in one of the iMac revisions (the bug was in the driver code from NVidea, but Apple didn't fix it.)

If you had a second monitor plugged in and changed resolution, it would forevermore think there was a second monitor plugged in, and force the nonexistent monitor to be the default. Resolution changes affect firmware. This one effected firmware in a way that a firmware reset wouldn't touch. The _only_ recourse was the swap the logic board.

This is my one and only personal example of commercial software causing permanent failure of commercial hardware.

(This machine _was_ replaced by Apple out of warranty because of the error.)

Re:Redhat 9

[apsmith]

Ha, my uid is less than your uid :-)

And I started with Linux kernel 0.99, so there!

Is there a Godwin's law for computer nostalgia discussions?

Re:Redhat 9

[JayAEU]

So far, so good, but no more live upgrades for me. You really should give Debian or Ubuntu Server a try.

Re:Redhat 9

[that this is not und]

It's a certainty that it was simply an eeprom or NVRAM setting somewhere in the machine that got changed and disabled the hardware. But, ya dance with closed proprietary hardware vendors like Apple, and when little bits inside their magic mystery hardware get toggled, you dance to the tune and wave more plastic at the Apple Store. On any reliable, properly documented and more open hardware, there would be a way to reset the relevant bits and carry on.

Re:Redhat 9

[that this is not und]

You can also always just reprogram the flash chip in your PROM programmer. Hotswapping the chip is risky and a programmer to burn your own flash is a $40 purchase on eBay.

But I've 'hotswapped' the NVRAM chip on a Sun Ultra 1 to break past a password in the hardware. I'm sure I could have put the NVRAM into my PROM burner socket and reprogrammed it. But didn't have the relevant details to do so. The flash code for any modern generic motherboard is just a binary block that you blow onto the chip, then plug the chip into the motherboard.

Can I...?

[Kranfer]

can I play on that awesome hardware? Or perhaps run SETI on it and make it a huge waste of processing power? oh oh, please please!!!

Re:Can I...?

[Anonymous Coward]

I would hardly call there hardware "awesome" Maybe It was awesome couple years ago.

Re:Can I...?

[Gonarat]

If it does the job well (and I have no problems with /.), then why keep running on the upgrade treadmill? Get the most life you can out of your equipment and software, only upgrade when needed. This is what most businesses do -- it saves money and (usually) allows a more stable environment.

Re:Can I...?

[Anonymous Coward]

I'd be more impressed if it could run Unreal Tournament 3 at a decent framerate with this hardware.

Re:Can I...?

[jlarocco]

Where did the GP say they needed to upgrade? He simply said the hardware specs aren't that great, and they aren't.

Re:Can I...?

[RockDoctor]

can I play on that awesome hardware?

Awesome?
It's quite a substantial set of kit i'll grant you. But I was keeping a (pretty crude) processor count as I RTFA. There's the equivalent of around 50 recent commodity processors in there, and a relatively modest amount of hard drive. If I counted up the onshore computing hardware in my work (development, beancounting and admin) and the active offshore hardware (the revenue earning stuff, apart from software sales) then I'd probably come to a similar total amount of processor cycles running on a week day. We're a medium-size enterprise - with current recruitment, we're probably sustaining 50 full-time people on the payroll at the moment when last year would have been in the low 40s, doing nothing awesome.

What obviously makes the difference in the hardware described is quality of componentry and sustained workload. Much of our productive work goes on inside people's heads while they absorb and interpret information on screen in front of them. During those times, the machines could switch the processor down to a 20Hz monitor on keyboard and mouse and no-one would notice any change. Terminal server territory?
My biggest bitch about hardware is that the development and testing people get the newest, fastest hardware, and code to that. which leaves us grunts in the field twiddling our thumbs at the hourglass icon on our vintage laptops. The obvious flow path for hardware to be retired is for front-line systems to be retired to development, then to testing, so that the testing people have the oldest, crankiest equipment in the company. If new developments work adequately there, then they should be OK on the newer hardware in the rest of company. Of course, the testing guy doesn't see it like that.

Re:Can I...?

[sparkeyjames]

I think you have this all backward. The newest stuff should go to the testers you know new code testing on new hardware to TEST for flaws. Stable code and hardware belong in the production department. The OLD crap goes to the development department to keep them from using new features in hardware that would break on current stable systems.

Re:Can I...?

[Anonymous Coward]

call there hardware

"their".

Re:Can I...?

[RockDoctor]

The newest stuff should go to the testers you know new code testing on new hardware to TEST for flaws.

What changes about hardware?
Hard drives get faster, but that's not a concern to application land.
Video cards change, but monitors stay at 1024x768 (OK, some laptops are going up to something like 1400x1050 or whatever the numbers are for widescreen. But since our documents are typically 10^5 pixels high by 4500 pixels wide, the size of screen they're displayed on is pretty unimportant. Unless you know of a vendor of 100000-pixel high monitors.
Printers change. But we print to a propriatory bitmap format (cough! spit!) at the work site, and the client (where ever they are in the world takes that to their local reprographics company, or the corporate flat-bed plotter and prints the bitmap on expensive archive-quality rolls of drafting film. We're trying to drag the industry kicking and screaming to 300dpi, but it's a hard struggle, since the product has to be legible after 4 generations of copying by dyeline copier. (That's the ones that stink of ammonia and gave the name to "blueprints".)
Keyboards change. While we're typing, if some muppet spill drilling mud into them. Who cares?

This isn't gaming, this is industry. There have been no significant changes in computing hardware in the last decade. Sure things get faster, but that's made up for by the developers throwing more bells and whistles onto the program. In practice, things are maybe ... (thinks) 1/3 faster now than in the early 1990s.

Sorry - one exception : cryptographic USB dongles have disposed of the problems of printing *through* parallel dongles (which was a significant part of the motivation to develop the propriatory (cough! spit!) bitmap image format as an intermediate output stage). We did that when some of our machines used DOS 4 (PC-DOS or MS-DOS, who cared?) At ~£100 each, the dongles are a significant cost. But by enabling us to charge per month of use of our software, the result is well worth it.

I don't know which country you're working in, but here the tax man requires us to keep hardware on the register for the full 5 years of it's tax-write-off lifetime. What that means is that there's 5-year-old machines still in the warehouse. And they're the ones that get sent offshore, while the office staff negotiate for new goodies. The office staff see themselves as the company, but in reality the revenue comes from us in the field. With the exception of the coding people, the office are a cost, not an asset. But they get the good stuff.

the powers that be

[ebolaZaireRules]

The hardware that powers slashdot?

I wanna know about the power that powers slashdot... are you really as green as the default colour scheme?

Re:the powers that be

[the_tsi]

As green as the power in San Jose is. That is to say, not very.

However, the webservers all have little green LCDs. Does that count?

Re:the powers that be

[Cctoide]

Are they greener than red LCDs?

Re:the powers that be

[the_tsi]

Only the bad guys have red LCDs. That's how you tell them apart from the good guys, who always have blue or green. ... or purple, I guess, if you're a bad ass motherfucker.

Re:the powers that be

[alta]

You two and your green/red LCD's. I've seen Green CRT's, and Amber CRT's... And Green, Red, and Even BLUE LED's. But've never seen a green or red LCD.

Actually, that's wrong. The original HP calc had a red LCD, and that damn bright alarm clock of mine that I put a shirt over every night has a green LCD.

Shit, I'm even wronger, I'm looking at my Dell 2850 server over there and it has a BLUE LCD scrolling the name of the machien back and forth. Nevermind.

Re:the powers that be

[josecanuc]

Actually, that's wrong. The original HP calc had a red LCD, and that damn bright alarm clock of mine that I put a shirt over every night has a green LCD.

You mean LED, not LCD.

Re:the powers that be

[alta]

Well, actually MEANT LCD. Ala http://www.phys.uwosh.edu/mike/calcs/hp-35.html

Unfortunatly I was wrong, and they are still LED. I classified them as LCD due to the ability o create numbers like a digital watch. Apparently the fact that they EMIT LIGHT wasn't enough to convience me that they were made from LED. Add that to the fact I thought LED's were very rare back then.

Stupid me.

Re:the powers that be

[makomk]

I hear the early digital watches used LEDs and LCD displays were a later improvement, though I've never seen one myself.

Re:the powers that be

[JustOK]

I still think digital watches are a good idea.

Re:the powers that be

[Mr. Slippery]

I hear the early digital watches used LEDs and LCD displays were a later improvement, though I've never seen one myself.

Feh. You kids today. Back in the late 70s and early 80s, a digital watch meant the cheerful red glow of LEDs - you pushed a button to make it show the time, since otherwise your battery would die really quick. If you have retro sensibilities you can own a modern replica

I also had a TI programmable calculator with LED display, might have been a TI-58 C though I don't recall for sure.

Re:the powers that be

[that this is not und]

The early digital watches used LEDs, which means that they sucked big amounts of power any time they were displaying the time. So to see what time it was, you had to hit a button on the watch, and the digits lit up. The more often you hit that button, the sooner the battery had to be replaced.

I remember the first LCD digital watch I ever saw, in a display case in a store. It just sat there and showed the time steady and continuously. I wanted it, badly, but it was priced in the mid three figures, which was a LOT of money in the 70's... Hell, a brand new Volkswagen Beetle off the showroom floor in the early 70's was an $800 or so purchase.

Re:the powers that be

[that this is not und]

I have test equipment with Nixie tube displays. There you're talking high voltage vacuum tubes. They make LEDs seem like micropower devices. But then I also have a random noise generator that is vacuum tube based. Have thought that I should put it online to make it available as a random source to anybody needing it. But oh well. My wheatstone bridge doesn't have ANY active electronics in it. Just resistors and the meter movement.

Re:the powers that be

[Mr. Slippery]

I have test equipment with Nixie tube displays. There you're talking high voltage vacuum tubes.

Sweet. Have you seen this Nixie tube wristwatch?

Load Balancing

[lymond01]

We have a hard enough time using CARP never mind specifying servers that just read or just write. I need to take a class. ;-)

Re:Load Balancing

[Grey_14]

thats usually done in application logic, not through packet shaping/redirecting,

Interesting

[lord_rob the only on]

Interesting read about Slashdot server farm. I'm somewhat surprised to see that Slashdot subscribers have two dedicated servers to read the main page, that's as many servers dedicated to a minority of users as to the rest of the users. But well, that's good for them, they help our best thrustworthy news site so they diserve to be rewarded :-p

Re:Interesting

[brunascle]

where are you seeing that? all i see is 2 for static content and 4 for logged-in users for the home page.

Re:Interesting

[CatPieMan]

Non-logged in user see the same page, so its basically a static page that gets updated every couple of minutes.

Logged in users can have a bunch of customization options on the front-end, which would take more resources.

I find it just as interesting that the logged-in readers use up that much more CPU.

Re:Interesting

[ZachPruckowski]

Who said subscribers have two dedicated servers to read the main page? The article/summary says that two servers serve for ACs reading the main page, and 4 for logged-in users. I saw no subscriber/non-subscriber distinction.

Re:Interesting

[jamie]

Yeah, I wasn't sure what he meant either. We have 2 webheads serving static pages (like the non-logged-in homepage), and 4 serving specifically the dynamically-generated homepage for all logged-in users. Plus 1 that serves all SSL traffic, which subscribers can use.

People often say "subscriber" when they mean "logged-in Slashdot user," not specifically a paying subscriber.

Re:Interesting

[avronius]

best thrustworthy news site I do not think that word means what you think it means...

Re:Interesting

[Anonymous Coward]

Never know, as pointed out, Savvis don't allow pictures to be taken, maybe this guy works there now.

Hope no one checks my Google search keywords, took a bit to find that link and finally just recalled what the article was with it.

Re:Interesting

[Provocateur]

Yes but did you see where all this hardware is wired to a Palm Pilot??

The Palm, so the legend goes, provides the necessary horsepower for the few that actually do read TFA's.

Re:Interesting

[inKubus]

Between "thrustworthy" and "master db" I'm starting to think this really is geek porn.

Re:Interesting

[that this is not und]

Even the earliest Palm Pilots have a Dragonball processor in them, which is a derivative version of the 68000 processor. The 68000 processor is nothing to scoff at, it's a damn powerful machine for many purposes. If you want to talk about tiny processors, you need to look at things like the PIC10F200 that I've been writing some code for at work. It has 16 bytes of R/W memory and 256 words of program memory. You're not gonna use a C compiler to write your code for a little critter like that, though you can if you really want (don't use a lot of recursion, the stack is two levels deep.) I just noticed that the above linked page says it has 0K of program memory. I suppose rounding to the nearest K, that is true. Really cool little chips, though. Priced at only about $0.40, you can deploy your code just about anywhere you need really cheap.

Re:Interesting

[EzraM]

I'm still not sure what was meant. How does the LVS setup distinguish logged-in users from non-logged in users?

Re:Interesting

[jamie]

Pound, our reverse proxy, lets you redirect traffic based on header text. If it sees one of our user cookies, it directs you to a dynamic webhead even if the URL you're hitting is static by default. One of the relevant parts of our config file reads:

UrlGroup "/$"
HeadRequire Cookie ".*user.*"

More on software

Redhat 9?

[eli pabst]

It'll be interesting to read the software section. It was surprising to see that they use an EOL'd version of Redhat (RH 9) that is no longer supported by Redhat. Granted, they're just webservers, but you'd think that would still require a lot of manually updating to keep things patched.

Re:Redhat 9?

[Roadkills-R-Us]

Not really, because as they're just web servers, you have a fairly minimal OS install footprint. There aren't that many things to keep up with. The odd kernel or basic library update, httpd update, and probably ssh-related stuff.

I should know; my web server is on 7.3. 8^)

Re:Redhat 9?

[mrsteveman1]

Thats true, but updates have to be backported, it just means increased effort with no company to blame if something goes wrong.

Re:Redhat 9?

[jimicus]

90+% of Apache updates have been for modules - and the RH9 servers are serving static pages which probably don't need many modules installed.

Write-only database?

[jolyonr]

That sounds useful! I use /dev/null as a write-only database. Very efficient.

Jolyon

Re:Write-only database?

[Ron Harwood]

I'm trying to assume that's humour... but that said...

If you have a farm of replicated mysql servers (which are read only - as replication is one way here) you need a db to write to.... not reading from it reduces the load on that server.

So, assuming that your read-mostly - it's actually a nice way to balance the load across multiple systems.

Re:Write-only database?

[Anonymous Coward]

But if it's truely ready-only, how to you propagate the data written out to the read-only nodes?

Re:Write-only database?

[cbiltcliffe]

Magic.

Re:Write-only database?

[Anonymous Coward]

Think of a large organization - like a government.

How often do you have the opportunity to make inquiries directly of the leader? I'm guessing that the further from the center of control that you get, the closer to zero that number approaches.

Instead, all of the data that goes up, gets fed back down to the "circle of influence" - the handful of diseminators. These people are the ones that are more available for inquiry. The larger the organization, the greater the number of layers of segregation.

As you make a simple request for you "foe list", it's not a question that needs to go to the master server - one of the plebian 2nd tier servers will be more than happy to answer that question for you.

Of course, I could be over-complexing this...

Re:Write-only database?

[cbiltcliffe]

I use /dev/null as a write-only database.

That does sound fast, certainly, but I prefer the idea of using /dev/urandom as a read-only database. Or maybe /dev/zero as a source for your encryption keys......

Re:Write-only database?

[grub]

As you make a simple request for you "foe list", it's not a question that needs to go to the master server

My foe list has its own dedicated Cray XT4 for processing.

Re:Write-only database?

[Anonymous Coward]

One of the Master db's handles all writes to the database. The other machine handles all reads from the database. This way the load is split between the two machines.

Re:Write-only database?

[Dante]

I never can figure out how people decide foes/friends. Now of my list of friends you have both friends and foes, go figure.

Re:Write-only database?

[that this is not und]

I have mozilla set to block graphics.slashdot.org so I have no idea about any of this friend/foe stuff.

There's a blank zone up at the top corner of most slashdot pages where I know from memory that there's a slashdot logo. I click up there sometimes to refresh the main page.

Possibly obtuse question

[athloi]

What determines why you run Red Hat 9 on some systems, and CentOS on others? Was BSD even considered? (You wouldn't run on Macs, would you?)

Re:Possibly obtuse question

[Precision]

Deployment date. The Redhat 9 machines were deployed 3 years ago and just haven't needed to be reinstalled yet. BSD, not so much.. we have a team of great linux admins, introducing another variable isn't likely to happen.

Re:Possibly obtuse question

[athloi]

Thank you for clearing up that puzzler. Seems like a nice, stable network setup.

Re:Possibly obtuse question

[Anonymous Coward]

We run macs on our desks, we've considered them for servers from time to time, but they always had some showstopper that didn't meet requirements.

Also, have you ever tried to get support from Apple for their commercial offerings? It's like getting support for an iPod. "Sure, we'll send you a replacement disk, just give us a credit card number in case you don't send the failed one back..."

Re:Possibly obtuse question

[saterdaies]

Usually these decisions are made based on familiarity, availability, and the like. If you're staff and you are all really familiar with RedHat, why would you force them to run BSD or Debian? Each system has pros and cons, but to be honest, the largest pro or con is usually familiarity. It's really easy to get familiar enough with any *nix to get Apache running. The issue is whether you have the knowledge to deal with it when your live webserver suddenly stops responding to requests.

Stability and familiarity are more important than the latest cool distro. Is there a reason that they should have picked BSD over RedHat? Of course there are some. There are others to pick RedHat over a BSD. In the end, you have to go with what you're comfortable and familiar with in order to ensure that you can deal with sudden, unexpected problems.

Re:Possibly obtuse question

[LWATCDR]

I have to give you high marks for using CentOS. It is a great server distro.
CentOS doesn't get the publicity that it should IMHO.

Re:Possibly obtuse question

[BokLM]

CentOS is just the same as RedHat

Re:Possibly obtuse question

[LWATCDR]

Yes but it is free. RES costs money.

Re:Possibly obtuse question

[byolinux]

RES is free too. You pay for support.

Re:Possibly obtuse question

[LWATCDR]

Really? Can you give me a link to the ISOs so I can download it?

Re:Possibly obtuse question

[bogie]

Just because you don't have an ISO to download doesn't mean it isn't Free. How do you think CentOS gets made?

Re:Possibly obtuse question

[LWATCDR]

Yep it means you have to download all the parts and do the work that the CentOS team does for you.
CentOS is free as in beer. CentOS is free as in speech and beer, and it is easy.
So why wouldn't you use CentOS?

Re:Possibly obtuse question

[that this is not und]

They were considering NetBSD running on a Macintosh SE/30 (a 16 MHz 68030) to serve all pages to paid subscribers, but then they decided the cost of electricty to run the system wouldn't justify the little usage it would see.

Re:Possibly obtuse question

[that this is not und]

I'm pretty sure you can buy Pink Tie Advanced Server from cheapbytes.

You have a point, of course. I asked the head of Red Hat marketing at a traveling presentation they were doing back in the days right after Red Hat 5.0 had come out about such matters. I asked her 'Is it okay if I use copies of the Red Hat 5.0 CD set that my brother in law bought?' in the q/a period after the dog/pony part of the show. She just acted like I was a troll.

Still can't believe..

[doomicon]

It's been 10 freakin years!!! I can remember going to Rob's page for his E apps. An amazing ride!

Reference Materials

[Ided]

This may be slightly off but I was wondering if anyone could recommend some good reading materials for setting up clustered sites or how to spread out work loads like they're doing with their systems.

Re:Reference Materials

[Anonymous Coward]

Umm.. the next article in this series?

Re:Reference Materials

[the_tsi]

http://www.linuxvirtualserver.org/ or anything about F5 BigIPs. Most of understanding load balancing is about understanding (a) how to fool layers above you in the OSI stack (switching on layer 4 through 7 -- particularly 7 -- can take a while to wrap your head around) and (b) the algorithms to pick which physical server gets the next connection (round robin, least connections, predictive, whatever).

Re:Reference Materials

[truthsearch]

The complexity is as much in the software as the hardware. I've had some experience with this from the programming side and started a DocForge wiki page.

Re:Reference Materials

[womenwantmefishfearm]

Try here: http://highscalability.com/

Finally some adult stories

[Jack Malmostoso]

Oh yes, geek pornography finally appears on /. :)
Thanks for the report, looking forward to the software part!

Re:Finally some adult stories

[goodtim]

Oh yes, geek pornography finally appears on /. :)

I agree. But lets get some pics up! I'm sure everyone here would enjoy a few pictures filled with racks of servers and blinking lights.

On a serious note, are you currently running in a SAN environment? If so, what vendor?

Re:Finally some adult stories

[avronius]

The article mentions NFS - available via most NAS offerings - and local disk. No mention of a fiber infrastructure. You'd think that would deserve it's own paragraph, at least...

Re:Finally some adult stories

[the_tsi]

There isn't any.


... yet. That's what I'm for.

Re:Finally some adult stories

[the_tsi]

I sent some cellphone pics to Taco. I dunno what he's gonna do with them. For the general idea, see http://meta.slashdot.org/comments.pl?sid=323935&cid=20928895 from the previous history posts.

Re:Finally some adult stories

[Anonymous Coward]

Shouldn't they have started with software and then done hardware?

Just sayin'

*ahem*

[zsouthboy]

Imagine a Beowulf cluster of those.

*cough*

Re:*ahem*

[Anonymous Coward]

In Soviet Russia Beowulf clusters you!

Which DB?

[umrguy76]

The database hardware was discussed, but what database software does Slashdot use today? MySQL?

bandwidth usage and cost?

[TwoWheelTomy]

wonder how much bandwidth slashdot is using and how much it costs.

Re:bandwidth usage and cost?

[Anonymous Coward]

I just got a quote for a point to point 100 Mb circuit that spans about a mile for just over $3,000. I imagine an internet connection an order of magnitude faster probably doesn't scale linearly.

Ouch.

Re:bandwidth usage and cost?

[Lumpy]

That's crazy. just lease a dark fiber. WE do that for a point to point that is 12 miles and pay $1500.00 a month. bring my own gear and I'm running 1000Mb happily.

The savings pays for the gear in less than 2 years plus we have 10X the band width as well as full control over the connection.

Re:bandwidth usage and cost?

[Anonymous Coward]

The value proposition for dark fiber is dependent on the distance involved, not to mention in many markets dark fiber is becoming scarce.

Re:bandwidth usage and cost?

[Precision]

The average monthly bandwidth usage for /. is around 40-50mbit/sec, which is relatively small. As for cost, you can contact your local ISP for a guesstimate, we get fairly deep discounts since we push quite a bit more with all the sites consolidated.

Re:bandwidth usage and cost?

[anticypher]

For a 100Mbit/sec commit on a GigE connection for a full internet feed, I've been getting quotes from California of around US$10-$16 per Mbit, depending on the data centre and provider. For a 1 Gbps commit, the price drops to around US$6/Mbit. Those are prices from sales droids without any attempt at negotiating a better price.

For a site like /., a 95th percentile bandwidth of 50 Mbps for the month would cost between US$500 and $800, less if the total commit for all of OSDN's traffic was much higher. Add in hosting costs estimated around US$2,500 per month for a cage with room for 6 racks and matching electricity and cooling, and you can calculate how much operating costs /. runs.

After that, you have to count up all the amazing 7 figure salaries of Rob and the gang who keep things running :-)

the AC

Re:bandwidth usage and cost?

[grub]

We do that for a point to point that is 12 miles and pay $1500.00 a month.

We pay CA$100 a month for a few kilometer run at a city on the east coast in .ca. The transceivers were something like $900 each but that's a one-time hit.

Re:bandwidth usage and cost?

[Percy_Blakeney]

That's interesting; I expected Slashdot to use a lot more bandwidth than 40-50 over the month. However, that statistic doesn't say much about what you actually push in the middle of the day; I would be much more interested in knowing what your typical bandwidth usage is from 8-5 and what your highest spike has been. As a point of comparison, I'm involved with a group of sites that pushes 60-70 on average over a month, but the typical 8-5 traffic is more like 120, with occasional spikes up to 260 or so.

Re:bandwidth usage and cost?

[Percy_Blakeney]

For a site like /., a 95th percentile bandwidth of 50 Mbps for the month would cost between US$500 and $800

Actually, if you're referring to the earlier comment about Slashdot's bandwidth, that 50 Mbps figure is supposedly the monthly average; the 95th-percentile billing that most service providers do is more about bursting time than about averages over large periods of time.

Re:bandwidth usage and cost?

[methos1142]

How are you possibly getting 50Mbps from a datacenter for that price? Every data center (Tier 1 centers) I've ever been with are typically about $600-700/mo for 1 Mbps, but once your at about 10 Mbps, it drops down to betwene $200-300/Mbps. This is always on a 95 percentile. So even at 50 Mbps at 95, I would imagine you'd be paying at least $100/Mbps -- which times 50, would be like $5000/mo. If you're getting it that cheap, let me know!

Re:bandwidth usage and cost?

[Percy_Blakeney]

How are you possibly getting 50Mbps from a datacenter for that price?

Slashdot is part of a larger hosting agreement. So, they're not paying for 50 mbps; that's just what Slashdot's portion is.

Re:bandwidth usage and cost?

[anticypher]

I was shopping for transit in the U.S. this summer, and those were the reasonable prices from companies that I also work with here in Europe. I don't know of any tier-1 who will bother with 1Mbps, most tier-2's won't either. My "standardized" quote is for 100Mbps commit on a GigEthernet port, that can handle sustained traffic of 800Mbps. This lets me compare without giving away details of my clients before contracts and NDAs can be signed.

However, I had a strange split in quotes I received. Some were in the range I expected, from about US$10 for a 100 Mbps commit (minimum bill about US$1000) up to around US$20-$25/Mbps. Then there was a huge jump up to the $500/Mbps range you speak of. Companies that were obviously not one of the tier-1 or 2 players, just resellers of tier-2 bandwith, but who didn't seem capable of competing.

Quite a few places seemed to think they could obfuscate the quote by refusing to deal in Mbps/month, and instead would offer traffic totals of 100 Gigabytes for inbound+outbound together. There were others who offered peak+offpeak or other ways to hide the usual Mbps/month quote.

One place was offering GigE ports, but I discovered later their internet transit was just a pair of 100M copper links. They sold their traffic as a package but when you calculate out 50 Gigabytes in one month into a traffic figure, you come up with something like 1-2 Mbps, for the low price of US$500. This may be where you are getting your quotes from.

As a very general rule of thumb, the tier-1s don't want to deal with a monthly bill of less than US$10,000, the tier-2s don't want anything less than US$1,000, and the tiny resellers will try to sell you everything they can (rackspace, metered electricity, port costs, traffic) to try to keep the bill upwards of $300-$500/month.

Just for comparison, even with the US dollar in free fall this summer, US prices were well over twice what we pay in Europe for internet transit.

the AC

Re:bandwidth usage and cost?

[Anonymous Coward]

But that's Canadian fiber in Canadian dollars.

$1.00CA = $15.00US right now.

Re:bandwidth usage and cost?

[mcrbids]

Yes, I know. One of the things that is no end of surprise to me is that the very best quality hosting is also the cheapest.

Starting at around $100-ish/month, you can get first class, type-A hosting with redundant power, redundant network, air-conditioned hosting, and lots and lots of bandwidth. What's expensive is the administration, which is why Linux is so great.

If you your administration is planned well, and you use quality hardware, you can cut it to the bone and still maintain excellent availability!

Re:the fark guys have a much better setup

[the_tsi]

I'd say doing more pageviews with less (and older) hardware says more about a "better setup" than having shinier boxes.

Why CentOS?

[bogaboga]

I am not saying that CentOS is any inferior at all but wonder why they chose it over all the possible serious systems in the Linux world. Is there anything CentOS does better than say OpenSUSE or Ubuntu/Debian and the rest?

Re:Why CentOS?

[jeffmeden]

CentOS is redhatish for the sake of redhatishness, as opposed to the current RedHat, which we all know is redhatish for profit, and Fedora, which is redhatish for testing purposes only. Ubuntu can't be taken seriously; you should have asked about Debian. Debian is redhatish to a fault.

Re:Why CentOS?

[pak9rabid]

CentOS is RHEL, minus the support. CentOS is 100% binary compatible w/RHEL as well, meaning the RPMSs you'd get from RHEL would work just fine in CentOS and vice-versa.

Re:Why CentOS?

[Precision]

We use a combination of CentOS and RHEL. The reason we chose CentOS over say debian is because it is basically identical to RHEL, we end up with a "single" platform that we have to deploy, test, and build packages for regardless of support. Depending on the system we will deploy either RHEL or CentOS accordingly based on support requirements.

Re:Why CentOS?

[LWATCDR]

Well I can not answer for Slashdot but I can think of a few reasons I would.
CentOS based off Redhat Enterprise. It favors stability over "new hotness". But unlike Debian it keeps pretty up to date without going to "Testing" or "Unstable". Yes I have used Debian and I am not a big fan. It may have changed so try it for yourself.
I also use OpenSuse daily. Yast is a mixed blessing. I find it very slow and too gui like for a server. I use it on my desktop and several servers in my office. I have years of experience with it. I bet I could set up Yum on Suse but just have not. Yast works in it's own way. The other thing is servercentric packages will be available for RedHat Enterprise first. I have never seen a major server package come out for OpenSuse before RedHat Enterprise. They may come out at the same time but not before. RedHat RPMs will work for CentOS so you benefit from RedHat popularity.
I set up Ubuntu server as a test. I found it to be a big pain. A lot of packages I wanted where not in the repositories so I spent a lot of time compiling stuff and chasing libraries.
For a server I would have to say that when it comes to CentOS the question really isn't "Why" but "why not".

Re:Why CentOS?

[AragornSonOfArathorn]

CentOS is redhatish for the sake of redhatishness

CentOS is redhattish because they use RedHat's freely-available source code.

Re:Why CentOS?

[saleenS281]

stability and enterprise hardware vendor support (by way of RHEL).

Re:Why CentOS?

[idontgno]

Yes indeed; RedHat with the serial numbers filed off.

My GNU/Linux server experience is pretty much all in the RH family tree, except for a brief fling with SUSE for SPARC in the early '00s. For desktop, though, I have to admit that Kubuntu is pretty sweet. I just haven't had to (or had the chance to) learn the guts like I've had to (or had the opportunity to) learn the ugly white underbelly of server OSs.

But Hell, I come from a SunOS and AIX background, so almost nothing in the GNU/Linux universe can really scare me much.

Re:Why CentOS?

[trandism]

>> I set up Ubuntu server as a test. I found it to be a big pain. A lot of packages I wanted where not in the repositories so I spent a lot of time compiling stuff and chasing libraries.

As someone who administers around 80 ubuntu servers, i'd like to hear more on that.. Which packages werent in the repos??

Re:Why CentOS?

[Wdomburg]

Seven year maintenance cycle. Wide ISV support, since it's RHEL compatible. Backported patches and bugfixes.

Re:Why CentOS?

[LWATCDR]

The two big ones off the top of my head where Webmin and PostgreSQL.
PostgreSQL was there but it wasn't the latest stable release. Which is typical of all distros but they also didn't have a deb on the PostgreSQL. The other was Webmin which which shocked me. That took a lot of time to get all the dependencies in this case perl modules working.
Of course things change so both those problems my have changed. Ubuntu Server doesn't suck. I just think that anyone that is setting up a server from scratch should take a look at CentOS. I will say one thing for Ubuntu Server, I liked Ubuntu Sever better than Debian even even though they share a lot of common heritage.

Re:Why CentOS?

[TClevenger]

I use CentOS and RHEL, depending on how critical the machine is. I used Fedora for a couple of years, but its lifetime is too short for me. For instance, FC5 came out in March '06, and has already been EOL'd. On the other hand, CentOS 2.1, which was released in 2004, will continue to receive security bug patches through 2009.

Re:Why CentOS?

[Suicyco]

CentOS is a clone of RHEL, a rock solid heavily supported OS. I've never seen OpenSUSE or Debian in a production/critical environment. If you dont think CentOS is a serious Linux OS, you have your facts out of whack.

Re:Why CentOS?

[trandism]

PostgreSQL?? solved
Webmin?? hmmm...

anyways, me would NOT leave APT heaven for RPM hell..
To be honest though, the only CentOS server we have here - an Asterisk server - is rock stable. So i'm on the CentOS++ side myself.

Saavis? Oh Noes.

[Chicken04GTO]

As someone with lovely personal experience wtih Saavis, im sorry.

Multiple master DBs

[atomic777]

"These are doing multiple-master replication, with one acting as Slashdot's single write-only DB, and the other acting as a reader."

Isn't that a contradiction? If you have only one write DB, why do you need multiple masters, aren't the other 6 just slaves at that point? Or are there separate master/slave pairs (I'm assuming these are MySQL databases)

Re:Multiple master DBs

[Bellum Aeternus]

master-master allows really fast fail over because you don't need to down the system to re-cofig a slave as a master. I've actually worked with companies that have master-master-master clusters.

Re:Multiple master DBs

[Unoti]

Perhaps it meant the write-only database has only 1 machine that is a reader for it, and the rest of the slaves all use that one reader as their source for replication. So the master writing database only has a single client, and all the other readers read from the "master" reader.

Re:Multiple master DBs

[Anonymous Coward]

Ahhh... we use master-blaster over here

Re:Multiple master DBs

[Anonymous Coward]

What is the con to running master-master then? Just curious why you wouldn't always run that way...

One way it's better

[Synn]

It's familiar to people who are used to working with Red Hat.

Re:One way it's better

[MrP-(at work)]

Yea we've migrated off RHEL to CentOS on some of our servers where I work.

Re:One way it's better

[mikek2]

Sadly, despite my objections, my new boss insisted on the exact opposite. I know RedHat's not that expensive, but it really irks my inner geek. He loves the comfort of being able to point fingers and say "it's not my fault, it's RedHat/Cisco/Verizon..."

Re:One way it's better

[sulfur]

Remember that without Red Hat there would be no CentOS. Supporting commercial Linux vendors is important, so if your company wants to pay money to Red Hat (for "support" or having somebody to point fingers at), let it be so.

Everybody wins with this model: Companies/PHBs who think that a product has to have a vendor and has to cost something are happy to be able to pay for Linux and have somebody to blame; commercial Linux vendors make a profit and increase market penetration/awareness; small businesses and individuals take advantage of free-as-in-beer builds (such as CentOS).

Uriah? That wouldn't happen to be U R Welcome?

[potscott]

CT: Thanks to Uriah and Chris Brown for the report. Now if only we remember to update the FAQ entry... Not a lot of dudes named Uriah out there. Could it possibly be Precision? The running Enlightenment and Gnome on a dual Celeron 300?

Re:Uriah? That wouldn't happen to be U R Welcome?

[Precision]

Indeed it is, sir. And it must have been my Dual PII 400, I've never owned a Celeron.

Re:Uriah? That wouldn't happen to be U R Welcome?

[potscott]

Ah, my bad. Thought it was overclocked celerons for some reason... 'Suppose I owe you a bit of thanks for that Debian Potato CD back in 99. =D

Considered a CDN?

[xmpcray]

I was wondering if you ever considered using a CDN service like Akamai to serve content? Most of the big sites (Apple/MS etc) use it.

Re:Considered a CDN?

[Anonymous Coward]

Would not be good for me. I block Akamai in my hosts file. Sure some sites render badly as the CSS is not pulled from the Akamai server but that is my choice to make and I have work arounds for that.

I made this choice after coming across an odd occurrence on an order form. With Akamai blocked the order would not submit. With Akamai not blocked the order would submit. I looked at the form used and it sent the order to an Akamai server first instead of directly to the site I was ordering from. Up until that point the only thing Akamai was serving for this site was the CSS. In fact all the other pages I clicked next on during my information entry and item selection did not do this. I thought it very odd. I started blocking Akamai on my main machine ever since.

Akamai is nice and all. Very useful for a great number of things. However, seeing that my order was being sent to an Akamai server before being sent to the store server, I decided that Akamai did not need that much information about me while using that IP address.

The site in question was Best Buy online. I was ordering a laptop on a web only special. I ended up calling the order in.

Re:Considered a CDN?

[mdm-adph]

And admit they can't handle it themselves? Never!

Personally, I think /. does just fine without a CDN -- never experienced much lag here.

Re:Considered a CDN?

[Precision]

Actually many of our sites do use a CDN, however the /. devs long ago decided against it for some reason or another. Heck it's still even all setup for them.

I want my sense of childlike wonder back!

[foo fighter]

I always imagined slashdot ran on hundreds (perhaps thousands) of modded Dreamcast consoles powered by lucky, randomly selected registered users running in hamster wheels who were lured by blocks of Wisconsin cheese dangling just out of reach.

Thanks for destroying my sense of childlike wonder, you insensitive clods!

Re:I want my sense of childlike wonder back!

[An ominous Cow art]

I always imagined slashdot ran on hundreds (perhaps thousands) of modded Dreamcast consoles Are you seriously saying that you imagined a Beowulf Cluster of them?

Re:I want my sense of childlike wonder back!

[Eberlin]

Wait...you mean...

then WTF am I doing running on this hamster wheel with my dreamcast plugged in? Damn you, Digg -- fooled me again!

Thanks for giving me the red pill, I guess.

Re:I want my sense of childlike wonder back!

[maxume]

You know what would be funny? If someone made a joke about a Beowulf cluster of something, and then somebody else said "Hey, did you make a joke about a Beowulf Cluster?" That would be funny.

Re:I want my sense of childlike wonder back!

[An ominous Cow art]

You're right! It could become the Next Big Thing on Slashdot. And to think, you and I were right there to witness the beginning... Definitely something for the autobiography, or to tell the great-grandchildren.

(I forgot to click the 'post anonymously' box for that one, as I usually do for my weaker efforts :-))

backup?

[nido]

Well, let's begin with the most boring and basic details. We're hosted at a Savvis data center in the Bay Area. Do you ever worry that a big earthquake will hit and your datacenter goes offline? Do you at least keep an offsite backup?

Re:backup?

[statikuz]

From the website: "SAVVIS has done extensive engineering to ensure that any Datacenter located in a region prone to seismic activity is braced for such events. Design elements include, seismic isolation equipment to cushion facilities against movement as well as seismic bracing earthquake bracing on all equipment racks. All SAVVIS Datacenters have racks anchored to the concrete slab below the raised floor."

Re:backup?

[Eponymous Bastard]

Well, let's begin with the most boring and basic details. We're hosted at a Savvis data center in the Bay Area. Do you ever worry that a big earthquake will hit and your datacenter goes offline? Do you at least keep an offsite backup? First rule of offsite backups: Never talk about your offsite backups.
Second rule of offsite backups: Never talk about where you keep your offsite backups.

You thought I was going somewhere else with that didn't you?

In all seriousness, that sounds like it would be in the software article instead.

Re:backup?

[Precision]

Of course we do offsite backups, but also we're currently preparing building a new primary data center in Chicago away from Earthquake land.

Re:backup?

[brarrr]

The rest of the world might be surprised to know that earthquakes are not a daily concern to californians. There is no 4pm shake. Sorry to disappoint.

Re:backup?

[DrgnDancer]

Which sounds great until the fiber link is cut by extraneous damage half a mile from the data center, or the slightly less hardened ISP data center is taken offline. The lengths they've gone to are primarily to ensure that the customers machines and data stay undamaged. Very little is going to prevent the site from going down in a major disaster other than backups and an alternative facility.

Re:backup?

[Anonymous Coward]

There is a 4PM shake, but it's no earthquake, and it's far from a disaster. Why do you think everybody from California is so laid-back?

I never quite understood the point of the "California King" sized bed until i moved out here...

Re:backup?

[Anonymous Coward]

That's actually exactly what we were hoping for. We don't want you to actually be PREPARED when the big one hits! Losing slashdot would be a small price to pay for being able to rid ourselves of you for once and for good, but based on Uriah's response above, we wouldn't even have to worry about that!

--Looks at watch to see how long 'til 4pm--

Re:backup?

[darkcmd]

For some reason I was thinking they were talking about the Bay City area (which is in Michigan) though California makes more sense.

Re:backup?

[Anonymous Coward]

Um, that would be 4:20

Re:backup?

[Anonymous Coward]

I guess you never met my wife!

Re:backup?

[GaryOlson]

I hear you can get cheap space in New Orleans -- no earthquakes there either.

Re:backup?

[shokk]

At which point, Slashdot is the least of people's worries. This is a news entertainment site, not a critical care facility.

Re:backup?

[DrgnDancer]

Two or three days of all of Andover/VA's sites being down would be a significant revenue hit. It's a concern for them, not for the world in general. Lots of sites where page views == revenue either spread their stuff amongst multiple data centers or have a lower power backup site able to take over in the event of catastrophic failure.

Re:backup?

[Anonymous Coward]

Ummm, you do know that Chicago is about 300 or so miles North of the New Madrid fault. It hasn't been terribly active (quakes are about a 2-4 on the scale...), but in years past it has been off the charts. That fault changed the course of the Mississippi for a while and completely redid the landscape down there... If it goes, we're f'd up here in Chitown...

There's also some in NW illinois about 90 miles outside of Chicago in LaSalle, IL which have had some quakes register 4 on the scale...

So we might not have big quakes, but we still got some stuff goin on here...

Re:backup?

[Anonymous Coward]

The Missouri River used to flow east from near St Louis (and out to the Atlantic via the New River) but thanks to millions of years of the New Madrid Fault, it now flows south.

Thanks

[debrain]

To the editors:

Thanks for this. It's really very interesting.

-B

read the entire series

[clem]

I can't wait for "Slashdot's Setup, Part 8 - Root Passwords".

Re:read the entire series

[Anonymous Coward]

Webservers: Yur@wank3r

DB servers: G3tfuk3D1

Routers: CwByN34Lr00lz!

Re:read the entire series

[maxume]

You're going to have to wait, it's scheduled for the 50th anniversary celebration.

Re:read the entire series

[the_tsi]

root:*x:0:0:root:/root:/bin/bash

SSH keys everywhere.

Re:read the entire series

[bteeter]

Some server farm admin at Savvis is scratching his head right now. "Why did Slashdot get hit with 10000 attempts to login, strangely all using the same wrong password?"

Re:read the entire series

[Oliver Defacszio]

... when they'll STILL be running RedHat 9.

Re:read the entire series

[moosesocks]

I have this funny vision of what would happen if /. got hacked, and how it would have been done:

The admins would wake up the next day to discover that the site was running perfectly normally, but was performing slightly faster than normal.

After closer inspection, they'd find that their datacenter had been emptied, and replaced by a single Apple ][ that had been hacked to run the latest version of Ubuntu, and that slashcode had been rewritten so that it would perform all of the same functions as the previous slashcode, but ran at twice the speed... on the Apple ][.

A post-it would be found stuck to the screen, stating that all of slashdot's old and now unnecessary hardware had been sold, with the proceeds being donated to the EFF. The message would likely include or be in the form of a Soviet Russia joke. Additionally, a miniaturized plastic Gnu would be left behind as a calling-card.

The news of this would be regarded as insignificant by the editors, until over a year later, it gets posted four times in the span of two days.

Comment woes

[theantipop]

Any chance that with all that iron you can loosen up the crazy restrictions on comment posting? It can get rather ridiculous sometimes.

Re:Comment woes

[Soko]

I'm guessing that's an anti-spam feature, not a limitation of the hardware. I think the time between posts is less for posters with better karma, too - not positive on that though.

Soko

Re:Comment woes

[theantipop]

Yes, I was being facetious. Usually it balks if you try to post twice in two minutes, although random strange behavior like this 10 minute thing happen every once in a while. And I'm not sure it's a karma thing, I don't think mine can go any higher (or at least it hasn't budged in 2 years). It's a much better topic of discussion for the software behind slashdot article anyway.

Re:Comment woes

[bill_mcgonigle]

I think the time between posts is less for posters with better karma, too - not positive on that though.

Yeah, but I still hate that "it's been 2 seconds since you last posted" message.

Seriously, though, it should be two minutes per post per story, not global. The rules predate tabbed browsing and Cable modems.

Re:Comment woes

[Anonymous Coward]

10 minutes? I've seen it as high as 22 minutes I think.

I understand their desire to reduce spam, but it gets quite annoying.

Scalability / Reliability

[omkhar]

I'm actually surprised, I was expecting much bigger Iron, esp on the DB side. Ah well.

Very curious that /. chose to use "free" distros. I would have thought SLES or RHEL would have been a consideration.

I think we all want to know...

[The Real Toad King]

How many FPS in TuxRacer you can get on them. Sure, individually, probably not so good, but working together, you could probaly break the 1000 FPS mark if you're lucky.

Active-Active Gigabit uplinks

[Anonymous Coward]

Active-Active Gigabit uplinks;

Anybody know what this is and how to purchase it?

Re:Active-Active Gigabit uplinks

[cecil_turtle]

If you're on an OC-24 or above loop your providers may be able to sell you bandwidth in gigabit increments. If you're on an OC-3 or OC-12 loop you can normally buy in 100 megabit increments. Otherwise "legacy" OC-3 is something like 155mbps and OC-12 is 622mbps if you're using the whole line (not breaking it off into T-1's, DS-3's, etc).

Once you have multiple uplinks from different providers you would typically use the BGP protocol to announce your IP space on both providers, then when people try to get to your site they will come down the provider which is the fewest hops for them. If one of the lines goes down, the other one is still active and then everybody will go down the running line until the broken one gets fixed.

Artistic solution...

[Anonymous Coward]

"Really, once you've seen one class A data center, you've seen them all. (CT: I've still never seen one. And they won't let us take pictures. Boo savvis.)"

Send in a courtroom artist :-)

25gig HD too much? :)

[jshriverWVU]

I went down memory lane so I fired up archive.org's wayback machine. This was a post on 1998 Booker writes "So IBM announces a 25 gig hard drive... does the world need this yet? Unless this is in a RAID, would you really want to trust 25 gigs on a single drive? What would you use this for? 400+ hours of MP3s comes to mind... "

Re:25gig HD too much? :)

[Joebert]

I wonder if Booker is on the RIAAs' 10 most wanted list, seems possible.

You probably shouldn't tell us too much informatio

[illumin8]

The BigIron 8000s have been in production since we built this data center in 2002 and actually, having just looked at it... haven't been rebooted since.

Gee, you really should update the firmware on your routers and switches more often than once every 5 years (or never). All I really need to do to hack Slashdot now is to look at all of the vulnerabilities on BigIron 8000s for the last 5 years and pick one to exploit. I wouldn't do that, but I'm sure a lot of miscreants could DOS you something fierce, or just plain wreak havoc by shutting off switch ports or routing everything to /dev/null.

Yep, you're dead on the money about Level4 support

[JohnnyComeLately]

I've worked 2nd Tier Technical Support within Sprint PCS, and it even got to the point I was helping their Level 3 with future system designs. Their level 1 was often a joke. One guy wanted me to reinstall packages on a Sun Solaris machine...I said, "This is not Windows...reinstalling will result in EXACTLY the same error" which of course, it did.

Anyway, it did get to a point where I instantly got escalated to their 2 or 3 tier because if I couldn't fix it, or I couldn't find the answer withing a Unix forum on-line, they would have a hard time offering a solution. This was supporting about 300 Sun Netra systems running Solaris 9.

Re:You probably shouldn't tell us too much informa

[Precision]

Lemme know how that works out for you, considering they're doing layer 2 only and don't have an IP address.

No wireless. Lame.

[monktus]

But seriously, I've really enjoyed these recent articles on Slashdot's setup and history, especially as I recently tried, and failed, to install Slash!

Re:No wireless. Lame.

[tetrahedrassface]

Slash is not very hard to install or mess with once you do it a couple of times. You might learn a lot by just not giving up. I did. The biggest issue is getting all of the development packages together and getting DBD::mysql to play nice with libmysqlclient. At least in my opinion. Currently I have 4 instances of slash witht he latest CVS running on this one home box. Thats just four...

Why? I don't know. I like it. I changed the installer script build it into users home directories and then it links it to apache but with one caveat. If you do more than one include directly on httpd.conf it screws up the rendering for some strange reason. So httpd.conf has an Include /usr/local/slash/httpd/slash.conf, and then in that instance inside /usr/local/slash/httpd/slash.conf I link to another one, and so on down the line for all the users.

For my home system it works pretty well. As far as on Redhat 9 or something, that shouldn't be too bad. Although I did try to install it on Fedora and never succeeded, although that was a long time ago. Now that I have seen what it takes I could probably get it running on a toaster. CPAN could take a while on that though.

No Photos? Have you asked?

[anticypher]

Really, once you've seen one class A data center, you've seen them all. (CT: I've still never seen one. And they won't let us take pictures. Boo savvis.)

Have you ever asked if you could take photos of your own installation? Find a manager or someone somewhat in charge of the data center, and let them know you need to get photos for insurance reasons or backup plans. Or the slashdot FAQ.

I've never had a problem taking photos in data centers in Europe and New York, by asking permission each time. It's a great way to document your work for future service calls when you are thousands of kilometers distant and trying to tell the remote hands guys what to do. It would also be a good way to educate slashdotters who don't know how dreadfully boring server installations are.

the AC
personally, I'm curious for how neatly the cabling is dressed

Re:No Photos? Have you asked?

[jimi1283]

This is standard policy for data centers, they never let you take pictures inside cages.

Re:No Photos? Have you asked?

[anticypher]

No photos allowed is pretty much blanket policy at every data center.

But if you ask, they generally have no problem of you taking photos of your own installation. Having a reputation and being on a first name basis with the security guards is a big plus.

What they are trying to avoid is documenting their own infrastructure or other people's kit. There are enough competitive forces out there that a few photos of new tech or a clever design could give someone an edge they wouldn't normally have.

There is also the physical security aspect of data centers, some amount of "security by obscurity" is built in to give a random level of risk to anyone hoping to do bad things. Publishing photos of exact locations of security cameras could give bad guys the extra information they think they need to pull off a heist.

Poking through my photo archives, I have shots of my installations at 1 Wilshire, several different companies at 60 Hudson, 111 8th, NOTA, plus many centres around Europe. The only places that flat out denied my requests were phone companies, where the installation was inside an incumbent's facilities. Military and aerospace installations are areas I never even bother asking.

One thing I've learned from the folly of some fool, is to never use a flash inside a data center. Tape over it with black electrical tape just to make sure. Flashes have enough infrared component to set off fire detectors, which lead to evacuation alarms, halon discharge (whatever they use these days), power cuts, and the eventual lawsuit and bannination. When you ask in advance, data center managers will tell you to tape over your flash and make sure it is off or removed.

the AC

Re:No Photos? Have you asked?

[nothermark]

your rack I can understand pictures. I can't see why you would need or get permission for pictures of the center or what the rest of it looks like.

bingo

[Anonymous Coward]

Bad area, twice as expensive as other areas, well overdue for a major quake and disruption.

storage?

[basil montreal]

So those databases store your entire archive of all posted stories and comments? I assumed you would need some sort of high performance shared storage...

Re:storage?

[inKubus]

There's only 21 million comments and far fewer stories. It's really not that many records. They have multiple DB's just for snappy performance, I would bet those servers are sitting idle a lot of the time. I remember after Sept. 11, 2001, they got slammed and they did a story about how they expanded their servers to deal with spikes in traffic..

Considering myISAM can easily do 4G rows and postgreSQL allows unlimited rows you are not going to run into a problem with the DB. And they are basically storing text. The average comment is probably what, 1k, so you're looking at around 25GB total data size, plus indexes, journals, etc. Still easily managable with today's hardware.

I'd like to see what google earth has behind it or shit, google itself. That would be impressive. But it is pretty amazing that a site like slashdot can do so much with so little. It does seem a lot bigger, doesn't it?

Re:You probably shouldn't tell us too much informa

[Anonymous Coward]

Your routers are only doing layer 2?

Why the switch to LVS?

[arnie_apesacrappin]

A long time ago, Slashdot was load balanced by an Arrowpoint CS-800. According to this write up, Slashdot has moved to LVS. What instigated the move? Did you feel that a fancy appliance-type load balancer was no longer needed? I didn't really care for the Arrowpoint switches or the Cisco CSS switches that they became, but I do like F5s and to a lesser extent the Foundry ServerIrons. Was the move away from appliance-based load balancers a money-saving idea, or just a lack of need?

Re:Why the switch to LVS?

[the_tsi]

It was a combination of requirements that LVS met the best at the time, for all the sites. And now it doesn't. We're deploying F5's for the new datacenter.

Re:Yep, you're dead on the money about Level4 supp

[fbjon]

Reinstalling packages could help, if you had screwed up something and had no idea what had gone wrong. Wiping out and starting over is then a (crummy) solution.

On updates

[DragonHawk]

Well, assuming they have a minimal set of software installed (kernel, standard library, Apache, SQL client, SSH), there's not much to maintain. And it's entirely likely that many of those are custom builds, configured and compiled locally, to get exactly the options they need and none they don't. That's not unusual when you're talking about a high profile site with a heavy traffic load. In such cases, optimizations which are not worth it for most cases suddenly become very important.

Of course, I'm just guessing. For all I know, there is a full GNOME install and a world-writable /etc/ on each server. ;-)

Your hardware ain't so great

[Anonymous Coward]

Your hardware still needs work - I just saw this when I went to look at a Slashdot article a minute ago:
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Re:Your hardware ain't so great

[the_tsi]

Pfft, software problem. I didn't get paged.

Yes, MySQL.

[Ayanami Rei]

NT

Sun hardwae

[Anonymous Coward]

For your next hardware update, have you considered some of Sun's Niagra-based systems?

Re:Sun hardwae

[Anonymous Coward]

Pfizer's Viagra based 'system'' is also great for uptime.

Just curious...

[LaminatorX]

Has the Savvis CEO ever taken you guys out for a night on the town?

Re:Just curious...

[inKubus]

CASH: Don't leave home without it.

oh yeah:

There is NO SEX in the champagne room.

root passwd != externally vulnerable

[Steve Hamlin]

I can't wait for "Slashdot's Setup, Part 8 - Root Passwords"

And what would you do with them? Knowing the root password shouldn't get you into a properly configured and patched system.

I even remember one cracking contest where the owner of the machine gave out the root password to the target machine. (quick google: nope)

You could attack the bandwidth, or try to get physical access. But if Cmdr. Taco can't get in....

Re:root passwd != externally vulnerable

[JuliaNZ]

I even remember one cracking contest where the owner of the machine gave out the root password to the target machine. (quick google: nope) The one I remember was a Mac running LinuxPPC. I was running it too, at the time, so it was an interesting competition.

Anniversary ... day??

[spazoid12]

Can't an anniversary be a single day? This Slashdot anniversary is dragging out longer than a Twinkie can sit on a shelf. Reminds me of my buddy Rick who insisted his "birthday" be "birthmonth" instead. Blah blah blah. I'm important.

Re:You probably shouldn't tell us too much informa

[illumin8]

Actually BigIrons are intelligent layer 2 and 3 (and some can even do 4 and above) switches, so they damn well better have an IP address. The management IP might not be accessible from the internet, but that doesn't mean you can't take it over. Some switches can be crashed (buffer overrun) just by routing bad data through them.

I'm not saying I would do anything like that, I'm just cautioning system administrators (I am one) to think twice before bragging about the uptime and exact architecture of your boxen. It lets shady characters know exactly what to target.

Re:You probably shouldn't tell us too much informa

[Anonymous Coward]

Actually BigIrons are intelligent layer 2 and 3 (and some can even do 4 and above) switches, so they damn well better have an IP address. The management IP might not be accessible from the internet, but that doesn't mean you can't take it over. Some switches can be crashed (buffer overrun) just by routing bad data through them.

No, BigIrons 'can be used as' intelligent layer2 and 3 switches. 'Can be used as' is different than 'are'. And there is NO reason why they 'need' to have an IP address. Ever heard of terminal/console servers? How about out-of-band Ethernet management ? And for $DEITY's sake, stop saying 'rout[ing|er]' ... they're using them as L2 switches. They're 'switching'. Perhaps ask about the difference between the two at your next entry-level Cisco training course (which you appear to be long overdue for).

I'm not saying I would do anything like that, I'm just cautioning system administrators (I am one) to think twice before bragging about the uptime and exact architecture of your boxen. It lets shady characters know exactly what to target.

Yes....caution 'system administrators' who are looking after the servers and workstations, and perhaps they can pass the word along to the 'network administrators' who actual take care of the 'network' devices like rout...I mean switches. And for the love of all things holy, please don't use the term 'boxen' again. It's the year 2007.

Yeah, but how many page views?

[shmlco]

Yeah, but how many page views are we currently talking about? The FAQ says 80 million pages per month, but that info was last updated in '04.

Without knowing how heavy the load is, I don't know if I'm supposed to be impressed or not... (grin)

Re:Yeah, but how many page views?

[jdjbuffalo]

I'm probably responsible for about 1% of those. Sorry, I've been busy lately. I'll see if I can't double that 80 million :P

Re:Hey CmdrTaco! Quit Flattering Yourself, Douche!

[Anonymous Coward]

Jealous?

RAID? Mail-Server?

[BrianWCarver]

Is the reason for the dual hard drives that you're using a RAID 1 mirror?

Also, when I forget my password Slashdot emails it to me. Which server acts as that mail server?

I feel like there are other small details that got left out... Tell us MORE! Very interesting.

Re:RAID? Mail-Server?

[Precision]

Yes, all our servers are at least RAID1, as for email, this article was Slashdot specific machines only. There are quite a few shared systems, including the outgoing mail relay.

Re:You probably shouldn't tell us too much informa

[Precision]

Ding, we /used/ to use them as layer 3 routers, but they couldn't keep up after the years and alas, they've been relegated to dumb layer 2 switches now. The poor cpu's can't keep up with anything else. We do have OOB serial management on them like you mentioned however.

Re:You probably shouldn't tell us too much informa

[jhfry]

I am not familiar with this switch, as you appear to be, but do these switches us multiple (redundant) processor modules. I know on some Cisco and other vendor's higher end switches, you can do a flash to one module and force a fail over so you can flash the other... or remove the processor board altogether and update it in spare chassis and return it to service.

access log

[jeffstar]

How far does the access log go back?

I may have dreamed^H^H dreamt this but I thought there used to be a slashbox that said what OS or browser was being used by what % of users/page views. I think it disappeared when IE or windows started to become the majority?

Or maybe I just have reeeeaaalllly lame dreams.

Wow with all that hardware,...

[AbRASiON]

You guys might want to check out Vista,.......

Re:You probably shouldn't tell us too much informa

[illumin8]

I normally don't reply to Anonymous Cowards, but you're being such a pedantic ass that I felt it necessary.

And for $DEITY's sake, stop saying 'rout[ing|er]' ... they're using them as L2 switches. They're 'switching'. Perhaps ask about the difference between the two at your next entry-level Cisco training course (which you appear to be long overdue for).

Perhaps you need a Cisco refresher course. "Switching" layer 3 is really just routing at the IP level. WTF is a Cisco Router other than a layer 3 switch? And, if you want to be really pedantic, a layer 2 switch actually routes packets between physical ports on a switch based on the MAC address (layer 2 address) of each host attached. So yes, it is both a switch and it also routes packets. Quite being a pedantic asshole.

And for the love of all things holy, please don't use the term 'boxen' again. It's the year 2007.

I'll use the term boxes or boxen as I see fit thank you very much. Did I offend your sensibilities? Oh noes!

Re:You probably shouldn't tell us too much informa

[illumin8]

Ding, we /used/ to use them as layer 3 routers, but they couldn't keep up after the years and alas, they've been relegated to dumb layer 2 switches now. The poor cpu's can't keep up with anything else. We do have OOB serial management on them like you mentioned however.

Sorry I didn't realize you are actually the sysadmin or network admin at Slashdot. You don't have a /. symbol by your name so it's hard to tell if you're just a low uid pretending to know everything or if you actually do. :-)

And I hope you compile your own Apache because whatever shipped with Redhat 9 surely has vulnerabilities. :-)

You really shouldn't use RedHat or CentOS

[Hayden Panettiere]

Please switch all of those machines to Debian GNU/Linux or FreeBSD. RedHat's software is undesirable from both a technical perspective, and also morally, because RedHat is a business. Using CentOS is a little better, but it's still not very good software.

Funnycomment, but root means nothing :)

[Anti-Trend]

If the /. syadmins are smart, and you know they are, direct root login is disabled anyway. You'd have to get perimeter access, then get a shell with wheel access in order to even try. Hell, I could give you all of my root passwords and my IPs right now and you still couldn't login with it.

slashdot stats

[koutkeu]

I m curious what kind of load does this hardware handle? pages/sec? peek/average bandwith?