Slashdot Mirror

I like this: http://www.rackable.com/sgi/sgi_logo_guidelines6.pdf ...extremely specific usage and typesetting guidelines for the new logo which are then comprehensively broken by the last page of the same document :)

It doesn't sound like Rackable is paying much for SGI's assets; but, they are picking up SGI's considerable debt, several hundred million dollars, in the deal.

No, Rackable isn't picking up all of SGI's debt. TFA says Rackable is assuming certain liabilities relating to the assets.

Falcon

It's really not so bad. Most drives are rated to about 55deg C (131F), 104F is only 40C.

The key is to design the server with sufficient airflow to try and keep the temperature of the components close to the room's temperature.

Looking at the Datasheet, it looks like they are running the servers on DC power. That way, each server doesn't have it's own power supply, they just hook up to a separate power unit elsewhere in the rack.

The servers don't seem to have fans either. The fans are in the cabinet door.

This setup reminds me of the description of Google's search cluster racks I saw somewhere.

This could result in huge savings. I remember some Sun data center guy talking about one of their new data centers and how they were able to run it at 74F. He said each deg F the could keep the temperature up resulted in 4% power savings.

Rackable Systems has their version as well - http://rackable.com/products/icecube.aspx?nid=datacenter_5 - and I think the idea was in the works long before Sun, although Sun has a lot more public spotlight than Rackable Systems, so everyone assumes it was their idea.

http://rackable.com/products/icecube.aspx?nid=datacenter_5

I believe Rackable Systems went to market with this concept first, didn't they?

You mean like these http://www.rackable.com/solutions/dcpower.htm?

Those Rackable guys http://www.rackable.com/solutions/greencompute.htm already did a DC powered system and they claim up to 30% of decrease in power consumption. Also Sun Niagara chipped servers http://www.sun.com/servers/coolthreads/se_t1000/fe atures.xml lower the energy consumption considerably compared to other x86 systems. They're naturally suitable for virtualization at the same time.

dcpower

Feb 15th: Rackable Systems granted patent for DC power to server racks.

"...The patented designs, released in 2003, leverage step-down power converters and alternating current (AC) to direct current (DC) power converters--commonly known as rectifiers--to distribute DC power to systems inside a server cabinet. Failover protection may be achieved by replacing a standard AC power supply with a highly reliable DC power card. Additionally, a secondary voltage step-down may be used within each system. This novel method of power distribution may occur inside or outside of the server cabinet, allowing for the flexibility to provide DC power to either a single cabinet or entire row of cabinets populated with systems..."

Rackable systems is doing this. 48VDC racks.

http://www.rackable.com/products/dcpower.htm

It's a great idea... I mean think of the number of times we go from AC to DC to AC to DC... Espeically when you throw a UPS in there that stores everything on 12V or 6V batteries. You loose so much in the transfers.

They can deliver a DC-powered rack that will do the A/C conversion in the rack using a rectifier, so you save power by making the conversion only once. They also can take DC to the rack, and put pretty much whatever you want into the systems. Not to mention the high density you can get.

Blade enclosures also use a similar trick, the blades all get DC. And many data centers also have DC available already, you just have to ask.

http://www.rackable.com/

I build Linux and Apple clusters for biotech, pharma and academic clients. I needed to announce this because clusters designed for lifesci work tend to have different architecture priorities than say clusters used for CFD or weather prediction:) Suffice it to say that bioclusters are rate limited by file I/O issues and are tuned for compute farm style batch computing rather than full on beowulf style parallel processing.

I've used *many* different platforms to address different requirements, scale out plans and physical/environmental constraints.

The best whitebox vendor that I have used is Rackable Systems (http://www.rackable.com/ [rackable.com] . They truly understand cooling and airflow issues, have great 1U half-depth chassis that let you get near blade density with inexpensive mass market server mainboards and they have great DC power distribution kit for larger deployments.

For general purpose 1U "pizza box" style rackmounts I tend to use the Sun V20z's when Opterons are called for but IBM and HP both have great dual-Xeon and dual-AMD 1U platforms. For me the Sun Opterons have tended to have the best price/performance numbers from a "big name" vendor.

Two years ago I was building tons of clusters out of Dell hardware. Now nobody I know is even considering Dell. For me they are no longer on my radar -- their endless pretend games with "considering" AMD based solutions is getting tired and until they start shipping some Opteron based products they not going to be a player of any significant merit.

The best blade systems I have seen are no longer made -- they were the systems from RLX.

What you need to understand about blade servers is that the biggest real savings you get with the added price comes from the reduction in administrative burden and ease of operation. The physical form factor and environmental savings are nice but often not as important as the operational/admin/IT savings.

Because of this, people evaluating blade systems should place a huge priority on the quality of the management, monitoring and provisioning software provided by the blade vendor. This is why RLX blades were better than any other vendor even big players like HP, IBM and Dell.

That said though, the quality of whitebox blade systems is usually pretty bad -- especially concerning how they handle cooling and airflow. I've seen one bad deployment where the blade rack needed 12 inch ducting brought into the base just to force enough cool air into the rack to keep the mainboards from tripping their emergency temp shutdown probes. If forced to choose a blade solution I'd first grade on the quality of the management software and then on the quality of the vendor. I am very comfortable purchasing 1U rackmounts from whitebox vendors but I'd probably not purchase a blade system from one. Interestingly enough I just got a Penguin blade chasssis installed and will be playing with it next week to see how it does.

If you don't have a datacenter, special air conditioning or a dedicated IT staff then I highly recommend checking out OrionMultisystems. They sell 12-node desktop and 96-node deskside clusters that ship from the factory fully integrated and best of all they run off a single 110v electrical. They may not win on pure performance when going head to head against dedicated 1U servers but Orion by far wins the prize for "most amount of compute power you can squeeze out of a single electrical outlet..."

Not affiliated with them, just like their half-depth 1U servers quite a bit.

Their DC power stuff is quite cool:

http://rackable.com/products/dcpower.htm

-Chris

My .02 cents worth ...

I build Linux and Apple clusters for biotech, pharma and academic clients. I needed to announce this because clusters designed for lifesci work tend to have different architecture priorities than say clusters used for CFD or weather prediction :) Suffice it to say that bioclusters are rate limited by file I/O issues and are tuned for compute farm style batch computing rather than full on beowulf style parallel processing.

I've used *many* different platforms to address different requirements, scale out plans and physical/environmental constraints.

The best whitebox vendor that I have used is Rackable Systems (http://www.rackable.com/ . They truly understand cooling and airflow issues, have great 1U half-depth chassis that let you get near blade density with inexpensive mass market server mainboards and they have great DC power distribution kit for larger deployments.

For general purpose 1U "pizza box" style rackmounts I tend to use the Sun V20z's when Opterons are called for but IBM and HP both have great dual-Xeon and dual-AMD 1U platforms. For me the Sun Opterons have tended to have the best price/performance numbers from a "big name" vendor.

Two years ago I was building tons of clusters out of Dell hardware. Now nobody I know is even considering Dell. For me they are no longer on my radar -- their endless pretend games with "considering" AMD based solutions is getting tired and until they start shipping some Opteron based products they not going to be a player of any significant merit.

The best blade systems I have seen are no longer made -- they were the systems from RLX.

What you need to understand about blade servers is that the biggest real savings you get with the added price comes from the reduction in administrative burden and ease of operation. The physical form factor and environmental savings are nice but often not as important as the operational/admin/IT savings.

Because of this, people evaluating blade systems should place a huge priority on the quality of the management, monitoring and provisioning software provided by the blade vendor. This is why RLX blades were better than any other vendor even big players like HP, IBM and Dell.

That said though, the quality of whitebox blade systems is usually pretty bad -- especially concerning how they handle cooling and airflow. I've seen one bad deployment where the blade rack needed 12 inch ducting brought into the base just to force enough cool air into the rack to keep the mainboards from tripping their emergency temp shutdown probes. If forced to choose a blade solution I'd first grade on the quality of the management software and then on the quality of the vendor. I am very comfortable purchasing 1U rackmounts from whitebox vendors but I'd probably not purchase a blade system from one. Interestingly enough I just got a Penguin blade chasssis installed and will be playing with it next week to see how it does.

If you don't have a datacenter, special air conditioning or a dedicated IT staff then I highly recommend checking out OrionMultisystems. They sell 12-node desktop and 96-node deskside clusters that ship from the factory fully integrated and best of all they run off a single 110v electrical. They may not win on pure performance when going head to head against dedicated 1U servers but Orion by far wins the prize for "most amount of compute power you can squeeze out of a single electrical outlet..."

I've written a lot about clustering for bioinformatics and life science. All of my work can be seen online here: http://bioteam.net/dag/ -- apologies for the plug but I figure this is pretty darn on-topic.

-chris

It's designed for large scale server rooms, and as such won't particularly adapt to what you want, but this does show that others have had the same thoughts and are applying them to various niches. They do also explain what they see as the benefits of this arrangement.

http://www.rackable.com/products/dcpower.htm

Server? You mean, like the tens of thousands of machines Google runs? The tens of thousands of commodity-PC-based machines? That they don't even bother swapping out when they fail, until the next regular maintenance cycle?

I'm sorry, but where did you hear of this? My friend works for google and has toured two of their major datacenters.

They use Rackable Systems almost exclusively. Where did you get your information?

We considered the Xserve , but eventually went with this box instead:

http://www.rackable.com/products/storage.htm

Incidently, I believe the Xserve RAID box is just a SAN unit, so you'd still need a front-end server (like a G5) to actually "serve" the disk.

Looks like Rackable is selling a system that does 88 machines per rack: the C1000 series. They list Google as a customer so I'm assuming that that may be it...

Disclaimer: I'm the author of the original analysis.

Cool... Real info... I've learned from another thread that they seem to be using machines from rackable. Based on the info on the rackable site, it looks like they are using C2000 series. If you get a chance, could you check out if that assumption is correct (disclaimer, I'm the author of the estimates listed at the top of the post :) )

Have a look at Rackable. Notice that Google is one of their customers. They offer up to 88 half-depth 1U servers in a rack and can even do DC power to cut down on heat generation from such a dense configuration. I'm sure Google is taking advantage of their custom configuration services, too.

Their servers are 1U but half depth, so they put them from the both sides of the rack. See here

google doesn't buy pre-built machines

Yes, they do.

rackable sell 1U high, half depth servers so 88 in a rack is easily do-able.

Well, racks of PCs are heavy, and noisy, and it's going to cost you a packet to ship them to the states (make sure you take the HDs out first, and package them up properly).
If you can get away with it, you should consider using something like these:
http://linitx.com/index.php?cPath=14
If not, then the only rack-mount machines that are worth purchasing IMHO are these:
http://www.rackable.com
I've deployed about 35 of these, and they do nearly everything right - price, performance, remote management, footprint, service. I've used Dell, HP, IBM, Gigabyte and others, but only Rackable seem to really know what they are doing at a decent price. Highly recomended.

588 servers, each with 2 CPUs, in seven racks.

That's 84 servers and 168 CPUs to a rack. Now blades are supposed to be more space efficient than regular rackmount servers.

Rackable sells short 1U dual-proc systems - so short you can fit 2 back-to-back in a standard 4-post cabinet, so they can fit 176 2.8 GHz Xeons in a standard cabinet.

Doesn't make the blade servers sound too impresssive...

and lots of those 54K servers were the cheap, 4-systems-on a-fiberboard-shelf systems. They told us they had a 25% failure rate with those. They were Pentium and Celeron based. And they dump A LOT of heat into our cage.

Then Google moved to a newer, more elegant system from These guys. Better heat dissipation as well (heat pumped up and out, instead of in all directions). And don't get me started on the wiring mess that was once Google - spaghetti everywhere, and HP switches strapped to the cabinets.

As far as I know Rackspace is a managed hosting company. Rackable Systems makes servers - Yahoo and Google both use them. Anyone know if the article has it wrong, and Pixar is actually using Rackable machines?

How do I know? Because they dump all of their heat (at least from their older systems) into our cage. Bless their little hearts. I believe their newer systems are from Rackable Systems in the South Bay. Rackable Systems

I just checked out the rackable systems site and the 1RU 1000 server looks IDENTICAL to sun's LX50 intel server. Same specs and identical physical boxes. I got my LX50 free when we bought a V480, now I know where to get another one cheaper :)

From there, they figured out a functional failover system and set up four geographically distributed data centers.

Oh, and they coded up a search engine thing at the same time.

Ok, I'll try and be a little more accurate. this article states that google buys servers from rackable systems and king star computer. It looks like a 250W power supply is pretty standard on these small rack servers (king star ones anyway). But from what I understand, a 250W power supply doesn't draw 250 watts continuously all the time, that more of a max rating. So lets grab another number out of the air. Say 150 watts.

150 WATTS * 10,000 = 1,500,000 * 3600 / 1000 = 5.4 million killowatts * 24 * .00346 =

$448,416

They don't directly inccur power costs though, their respective colo facilities do. I know google probably has at least one cage in Equinix from the (rather old article) above. And if you're a colo facility, I'd be willing to bet you might be able to get a better price on power than your average home user.

here's another google article for those interested.

I hear they use massive Linux clusters, too, which I'd love to get my mitts on.

yes, they do... I used to do a lot of work in a datacenter for a customer whose cage was right next to google.com's. they had about 20 racks of 80 each (40 on each side of the rack) of these - and that was just in that one datacenter. not sure how they used them, but they looked cool.

I've seen their cage out at Exodus in Virginia. Pretty cool.. They have like 6 racks of servers there - each rack is 80 servers I believe. They use systems from Rackable. Generally in a hosting facility you pay per rackspace and bandwidth -- more servers/rack means less cost/month in space.

Rackable systems at www.rackable.com has half width 1 U machines. They can get 80 machines in a standard rack plus a 4U network switch on both sides. Very impressive. They are used by google and others.
Stuart Eichert

Rackable (http://www.rackable.com has some really nice high density rackmount systems with interesting cooling and wire management features. If you want you can get the systems without top or bottom covers and they can be installed tray-like into the rack.

They have 1/2 width 1U units that can go dual CPU which means you can get 88 dual-processor systems in a single rack :)