Slashdot Mirror

Facebook also has (or had, as of May 2012) a large presence in a datacenter owned by BAIS, located at the border of Sunnyvale/Santa Clara, called "SC2". It's a fantastic datacenter, to be honest: I used to rent 24U of rack of space (secured cage, not shared rack) + bandwidth (3.0mbit 95th-percentile) + power for about US$750/month, plus insurance costs (around US$500/year). Most of the expense was the physical rack space.

Let me be clear: the datacenter is amazing, is engineered Mostly Right(tm), and is very large (83,000 square feet) when compared to their previous DC (called "SC1", which consisted of the classic chicken-wire-esque fencing model, crappy AC (floor blowers were brought in but couldn't solve the situation) and tons upon tons of customers who did crap like this. Those idiot customers were "siphoned out" as a result of the SC2 migration (given new requirements/etc.) and I was honestly glad to see those customers go. There were a few in SC2 who continued to operate like this (devices/wiring literally being crammed into a secure cage, to the point where it blocked 90% of airflow), but no where near the number as in SC1).

At a networking level the services offered were done "mostly" right (there were a couple router or switch failures during the years I was there, where redundancy kicking in required manual intervention -- Tom Wye (CEO) would always respond personally in Email about such incidents, which was positive), though I imagine Facebook's network was separate from the BAIS network. My point is that the DC as a whole was done really, really well.

So when I say "large presence", what do I mean? Quite literally: half the datacenter (41,500 sqft) was just for Facebook, and that half was cordoned off (extra set of fencing and separate badge readers) too.

How did I know it's Facebook? Because there were multiple (not a couple, but 5 or 6) gigantic Facebook banners/stickers/labels all over the equipment, visible from behind the fencing, and because I ran into other customers who seemed to know that fact (conversations implied they knew people who worked at Facebook, who recommended BAIS, and that's why they themselves were getting rack space there).

The downside to Facebook owning half the DC is that BAIS stopped caring as much about smaller customers, since half their revenue was coming from Facebook. This manifested itself negatively in a couple of different ways, which I can itemize if people are interested in knowing. The short version is that very selective rules were applied to only the "smaller" customers, while the big boy a thousand feet away wasn't given the same degree of scrutiny, including being allowed to violate "hazardous material" requirements for several months (possibly indefinitely).

I have no idea what Facebook does at SC2, but given its size, I'm surprised it wasn't mentioned. No, it's not a full-fledged ground-up facility like the one at Prineville or Google's The Dalles datacenter, but it's worth pointing out that it's a good size and still located in Silicon Valley.

Facebook also has (or had, as of May 2012) a large presence in a datacenter owned by BAIS, located at the border of Sunnyvale/Santa Clara, called "SC2". It's a fantastic datacenter, to be honest: I used to rent 24U of rack of space (secured cage, not shared rack) + bandwidth (3.0mbit 95th-percentile) + power for about US$750/month, plus insurance costs (around US$500/year). Most of the expense was the physical rack space.

Let me be clear: the datacenter is amazing, is engineered Mostly Right(tm), and is very large (83,000 square feet) when compared to their previous DC (called "SC1", which consisted of the classic chicken-wire-esque fencing model, crappy AC (floor blowers were brought in but couldn't solve the situation) and tons upon tons of customers who did crap like this. Those idiot customers were "siphoned out" as a result of the SC2 migration (given new requirements/etc.) and I was honestly glad to see those customers go. There were a few in SC2 who continued to operate like this (devices/wiring literally being crammed into a secure cage, to the point where it blocked 90% of airflow), but no where near the number as in SC1).

At a networking level the services offered were done "mostly" right (there were a couple router or switch failures during the years I was there, where redundancy kicking in required manual intervention -- Tom Wye (CEO) would always respond personally in Email about such incidents, which was positive), though I imagine Facebook's network was separate from the BAIS network. My point is that the DC as a whole was done really, really well.

So when I say "large presence", what do I mean? Quite literally: half the datacenter (41,500 sqft) was just for Facebook, and that half was cordoned off (extra set of fencing and separate badge readers) too.

How did I know it's Facebook? Because there were multiple (not a couple, but 5 or 6) gigantic Facebook banners/stickers/labels all over the equipment, visible from behind the fencing, and because I ran into other customers who seemed to know that fact (conversations implied they knew people who worked at Facebook, who recommended BAIS, and that's why they themselves were getting rack space there).

The downside to Facebook owning half the DC is that BAIS stopped caring as much about smaller customers, since half their revenue was coming from Facebook. This manifested itself negatively in a couple of different ways, which I can itemize if people are interested in knowing. The short version is that very selective rules were applied to only the "smaller" customers, while the big boy a thousand feet away wasn't given the same degree of scrutiny, including being allowed to violate "hazardous material" requirements for several months (possibly indefinitely).

I have no idea what Facebook does at SC2, but given its size, I'm surprised it wasn't mentioned. No, it's not a full-fledged ground-up facility like the one at Prineville or Google's The Dalles datacenter, but it's worth pointing out that it's a good size and still located in Silicon Valley.

Has been done, though not updated with the new list. Take a look here.

No, not in the top 5 any longer. But those Mac clusters that are in the top 500 give by far the most bang for the the buck. See for yourself on this list. You want to look at the 'R-max MFlop / dollar' column. Because of this fact I personally don't understand why there aren't much more Mac clusters being built.

Interestingly Apple's G5 based supercomputers are a lot cheaper than the rest. But there is also one very cheap Xeon-based system.

We're talking HUNDREDS of machines for a RENDERING farm, this is apples to oranges buddy, not desktop mac vs desktop amd machine.

Once the build is setup for the opteron machine and rolled out to each new one, there IS no fucking maintenence (unless they have things setup far differently from what one would expect for such a large operation)

Ok, even in this environment - essentially a large, uniform cluster - how, then, do you explain something like this?

The IBM PowerPC 970 family (and the forthcoming dual-core PowerPC 970MP) isn't a bunch of dogs - they stand up EXTREMELY well against, e.g., Opteron. The real issue is still Mac OS X Server and Xserve's relative immaturity when compared against, say, Intel/AMD solutions running Linux and a lack of solutions designed for Mac OS X Server and the PowerPC architecture as opposed to, say, Linux on x86-64. But there's no reason, given what we've seen with clusters so far, that an Apple solution can't be a real - or even better - contender.

The really funny thing is, they'll continue to deny it in the face of things like this and this.

That's why I took a couple week hiatus from even posting here. That, and the fact that after hundreds of points of positive moderation, all it took was a couple of factually correct posts in a story about John Gilmore modded down to "-1, Troll" to get me literally banned[1] from posting to slashdot for a week.

Real nice.

[1] And yes, that really is the full story. These posts - 1, 2, 3 - got me banned from posting, even as three other posts in the same story got moderated to +5: 1, 2, 3. Sure, I could have worked my way around the "ban", but what's the fucking point? To post to a place where you're not wanted and opposing ideas are shouted down?

Regarding the visibility, if VT managed to get lower prices in exchange of publicity, I say it is a smart move on their part. Everybody is entitled to do this and can do it, so it's a level field. MareNostrum's builders should do it as well. Component maintainers and suppliers have to value what they can win by giving discounts to important clients.

Hey, they do it at my local gym, by working at a big corporation I am given a good discount, that in exchange of the good publicity they expect me to give about them at work.

By efficient I meant in price/returns terms. VT is much cheaper and quite powerful even if the $20M figure were taken into account. Yeah, I know about the law of diminishing returns, but the VT-MN disparity is too big IMHO.

It's hard to know the real price on anything. Take for example potable water, quite cheap at malls but it can be argued that it's real economic and enviromental costs are quite higher. For the sake of comparison we can only stick to published costs. I am sure that at the UPC they'll leech work time from undergraduates and PhD students as well, just like everybody else, again level field.

Check out the 2nd chart in the cost comparison. IBM plans to have a Blue Gene/L system 4 times as fast as their current number 1 system by May 2005. Not only have they sniped the lead, they're planning to completely out-class the competition.

The comparison loses all credibility when you see the author's home page and his photos. Prepare to gag.

The summary called it "an excellent cost comparison." I'm sorry, I just can't trust a comparison from an insane, cat-loving Mac fanatic.

The comparison loses all credibility when you see the author's home page and his photos. Prepare to gag.

The summary called it "an excellent cost comparison." I'm sorry, I just can't trust a comparison from an insane, cat-loving Mac fanatic.

The comparison loses all credibility when you see the author's home page and his photos. Prepare to gag.

The summary called it "an excellent cost comparison." I'm sorry, I just can't trust a comparison from an insane, cat-loving Mac fanatic.

I don't know how much Dell's Tungsten cluster cost but those guys went online last year and got ranked #4 (just behind this Mac cluster) and they're #5 or something now. These bozos have spent a year fscking around with upgrades and from the theoretical #3 (as they were taken out since the cluster couldn't enter production) will have dropped to #7 or more in the next ranking....

Tungsten cost $12 million. Just for the hardware.

System X cost a total of $6 million, and it's still faster.

Not to mention that Virginia Tech was able to pull of a publicity coup and become #3 in the world, #2 in the US, and #1 academic for a paltry $5.2M. And they were "taken out" of the list voluntarily, because they dismantled the entire thing to replace it with Xserve G5s. With the renewed US focus on supercomputing, no one will likely EVER be able to hit #3 on this list for something anywhere close to $5.2M again.

Here's the current list:

http://www.netlib.org/benchmark/performance.pdf

Here's just the current top 20, as of 10/26/04:

http://das.doit.wisc.edu/misc/top500.jpg

Confusingly, you seem to have forgotten that since VT dropped on the list, since VT is still much faster than Tungsten, that means Tungsten also dropped. Tungsten is currently #16. For $12 million. VT's 2.5 Tflops faster - a respectable standalone clusters' worth faster - for half the price. Plus VT got all the huge publicity and news articles, and attracted millions of dollars in funding and grants for their new supercomputer center. Not to mention bringing a whole new OS, platform, interconnect, and processor onto the scene, which will benefit everyone (competition and choice is good, right?).

Also, here's a really great cost/performance comparison of all the top clusters.

Nice try at trolling, but next time don't be so obvious and pathetic about it, especially when Tungsten looks like it clearly is the raw end of the deal, when you have to spend over twice as much money to get a cluster that performs significantly worse, and has worse power requirements.

Yes, it was up for a while, but mostly for testing and tuning.

The one critical problem with the initial cluster was that the Power Mac G5 didn't have ECC memory, meaning that any long calculation would really have to be run twice - or at least until the result was the same - to essentially insure a soft error did not go unnoticed (and no, VT's special "error detecting" software didn't account for this).

The Xserve G5, however, does have ECC memory, making the current cluster just as capable as anything else in the top 10.

I'm not denigrating the original cluster, however: VT played by the rules, and made it to #3 in the world, #2 in the US, and #1 academic for a mere US$5.2M. They also broke the burgeoning Dell/Linux hegemony for cheap clusters, proving that you could use Apple, PowerPC, Mac OS X, and Infiniband to make clusters just as cheap, if not cheaper (note how much better the Apple clusters perform per processor than the closest Dell P4 Xeon 3.06GHz Linux cluster several spots below...additionally, check out this fantastic cost comparison of many of the top machines). Not to mention bringing a new 64-bit player to the HPC table. And one would hope that competition, even in supercomputing, is a good thing.

Because the next list in June 2004, and the one after it, and the one after that, will include many new very, very expensive clusters (some $100M+) with performance far, far beyond 10Tflops.

So, yes, someone can build a 1101-G5 cluster right now, and be faster than VT's cluster. But they won't be on any list, and they definitely won't be anywhere near #3 on the next Top 500 list. And neither will VT.

That's why the whole VT #3 thing is the coup that it is: the timing was *perfect* for them to take the #3 spot for a mere $5.2M. The PR and grants they'll get *because of* that are more than worth it. That will never happen again for anywhere near that small an amount of money anytime soon.

See some of the new clusters that will be in the Top 10 on the next list:

http://www.bayarea.net/~kins/AboutMe/GIFs/TOP500_l ist_for_CPU.gif

Because of Saddam's aggressiveness, the Gulf War came about, during which a lot of American technology (Patriot missles, A10 Warthog, etc) was tried in battle for the first time and found to work really well.

Huh? "Patriot missile" and "work really well" are two phrases I never thought I'd see in a sentence together.
Some more information on the patriot.
IIRC, the A10 was around long before the Gulf War, and in fact was in the middle of being phased out before the war started (and the Pentagon realized that they didn't have anything that could replace it.)