Slashdot Mirror

So did significant improvements to HBase, PyTorch, Haxl, PHP compilers, and much more.

And Facebook is inventing sophisticated AI hardware (https://code.fb.com/ml-applications/the-next-step-in-facebook-s-ai-hardware-infrastructure/) including semiconductor design (https://www.networkworld.com/article/3268974/hardware/is-facebook-looking-to-build-its-own-data-center-chips.html). and is the primary contributor to the Open Compute Project's work on more efficient data center hardware (https://www.opencompute.org//

TL/DR: The only reason they are able to invade our privacy that effectively is that they really are an impressive technology company.

What you are asking for is essentially the open compute design.
http://perspectives.mvdirona.com/2011/05/open-compute-ups-power-supply/
http://www.opencompute.org/wiki/Open_Rack/SpecsAndDesigns

The standard setup is three power supplies in each rack (they are shifting to two with v2) supplying DC power to each server. With one UPS rack supplying DC power to six server racks. No inverter in sight.

Also, Google used to run 12V batteries in every server but more recent designs haven't been publicly released.

In Open Compute I believe the power supplies wait random amount of time before applying power to the rails to fire up the load. What an idea. You auto-magically spread out the time of the start up load over a short time.

I doubt it.

Google, Microsoft and Facebook also built their own servers, they aren't selling them.

These are servers built for specific (set of) tasks.

I would rather see them join the open compute project (where you already have some of the designs from companies like Facebook and Microsoft):
http://www.opencompute.org/

Obviously, they could do both. But selling open source hardware that would be weird for Apple, I think ?

Anyway, other companies do sell hardware from opencompute designs.

It's not the first one
It's not even the first time it is used for something related to computers.
OCP OCP OCP OCP
How many more uses of OCP do you want?

When anyone picks a 3 letter abbreviation for something there will be collisions. Even within the same field so context won't help much.

Correct.

'Open compute' seems to be about publishing Facebook's techniques to run a large server-farm.

Perhaps it's meant to remind us of OpenStack, but it's certainly nothing to do with FOSS hardware.

'thestack.com' is clearly garbage.

http://cumulusnetworks.com/blo...
http://www.datacenterknowledge...
http://opennetlinux.org/
http://www.opencompute.org/
http://www.wired.com/2013/03/b...

Get with the times, the Big Iron Networking gear (like usead at Google and Facebook) are switches running Linux.

I thought they were already using DC power:

"The battery cabinet is a standalone independent cabinet that provides backup power at 48 volt DC nominal to a pair of triplet racks"

http://www.opencompute.org/pro...

http://www.opencompute.org/pro...

I thought they were already using DC power:

"The battery cabinet is a standalone independent cabinet that provides backup power at 48 volt DC nominal to a pair of triplet racks"

http://www.opencompute.org/pro...

http://www.opencompute.org/pro...

Yes, I guess Abrash really dropped the ball when he didn't do his due diligence on what language the parent company of the VR company he just accepted a job at build their website in!!!@@@

What an odd piece of techno-elitist whining. For what it's worth, Facebook's engineering acumen can be seen in some of the software they've released, including the Hiphop VM for PHP as well as the recently announced Facebook Hack. They have contributed to a huge variety of OSS ( https://code.facebook.com/proj... ) - and things like the OpenComputer project ( http://www.opencompute.org/).

The scale alone of Facebook is an impressive engineering feat. That is entirely the point of Abrash's post, really.

This is one of those RFA to get to the RA type stories.

The link next to the quote is the one you want :-
http://www.opencompute.org/2011/11/17/learning-lessons-at-the-prineville-data-center/

There are a few differences in how the figures are compared here. With a commercial building like Trump World Tower, the figure is for semi-finished space. That's the cost to erect the main structure, build the lobby and other common areas, shell each office (4 walls and a dropped ceiling), and sell the space. It does not include the cost to fit out an individual office space. If you bought a floor and wanted it done in Marble, that would be on-top of the $300 million cost, paid by the tenant. It's not cost to the investors in Trump World Tower, so isn't in the $300M figure. If somewhere someone tallied all the construction and build out costs for all the tenants of that building, it would substantially higher.

In a single user data center, the costs to build include the shell, power and the fitment of the space. To use some official numbers from a builder in the data center marketplace, CBRE suggests "Data center construction costs average $295 per square foot ($150 to $200 per SF shell, $12M to $18M per MW thereafter depending on the required design resiliency) ".

1.4 million square feet and $300 per, that's a $420M for a shell. I would hope a project of that size could get some economies of scale and come in at least 20% cheaper than that figure, but it really depends on some of the features a tenant might want.

I suspect you could run shell costs from near half that for a "bare bones" setup, to near double for some of the fancier features possible to add (biometrics on every door type bells and whistles).

The big question, is how much power (and cooling, they go together). Low power equipment might require 75 watts/square foot (105 MW), giving a power cost (using the low figure of $12M/MW) of 1.26 Billion; and high power equipment at 300W/square foot (420 MW) would be 5 Billion! Facebook has actually been a pioneer in reducing these costs with it's Open Compute project to make for more efficient setups. This should reduce their power cost well below the average, perhaps shaving 20-30% off that figure as well.

There's one last thing, what about servers? If it's a single tenant data center some folks might include the servers for such a data center. Conservatively 40 servers a rack, 30 square feet per rack, the building could house 1.86 million servers. At $5000/server, that would be another 9.3 billion!

Facebook claims just over a billion active users, or about 537 users/server, if this was their only data center. I'll let the rest of the crowd here debate if that's a reasonable amount of infrastructure per user, or too low, or too high.

While I agree with you on many points.

It is possible serverrooms are going to look very different in the coming years:

http://www.datacenterknowledge.com/archives/2013/01/22/silicon-photonics-the-data-center-at-light-speed/
http://www.wired.com/gadgetlab/2010/07/silicon-photonics-50-gbps/
http://www.opencompute.org/ocp-summit-iv-videos/

Basically, they are sawing the motherboard in 2, where the CPU and memory are on the daugterboard, and the rest of the components (SATA,USB3, PCIe slots, sound, video outputs) are all that remain on the motherboard

It is actually a micro-server architecture. Think small form-factor blade servers with an optional PCIe interconnect, optional remote SATA devices, and one mandatory ethernet interface, all running through what looks like an ordinary PCIe slot, but isn't.
http://www.opencompute.org/wp/wp-content/uploads/2013/01/Open_Compute_Project_Micro-Server_Card_Specification_v0.5.pdf

No. These are micro-servers we are talking about. Two RAM slots typical. Low power energy efficient CPUs too. ARM to start with.

Take a look:
http://www.opencompute.org/wp/wp-content/uploads/2013/01/Open_Compute_Project_Micro-Server_Card_Specification_v0.5.pdf

More important, it means that elements of memory and processing that now must be fixed closely together can be separated within a rack, and used as needed for different kinds of tasks.

This statement is in reference to Intel's proposal, which is still vaporware. I seriously doubt they are talking about locating main memory away from the processors. That would more or less be suicidal.

Facebook's design certainly does no such thing.
http://www.opencompute.org/wp/wp-content/uploads/2013/01/Open_Compute_Project_Micro-Server_Card_Specification_v0.5.pdf

Here is a link to an actual specification. If you read it, you will see that about half of what has been written about this announcement is wildly off base. We are talking micro-servers here - complete with on board cpu, ram, boot eeprom, flash storage, and ethernet. PCIe and SATA connections to the backplane are optional. Think small form factor blade server.
http://www.opencompute.org/wp/wp-content/uploads/2013/01/Open_Compute_Project_Micro-Server_Card_Specification_v0.5.pdf

http://www.opencompute.org/wp/wp-content/uploads/2013/01/Open_Compute_Project_Micro-Server_Card_Specification_v0.5.pdf

It uses that connector but the signals are ethernet, SATA, etc. RAM and optionally mSATA are on board.

Unless you're the big boys: Then you're just specing it from the ODM and bypassing Dell/HP/etc: http://opencompute.org/

Take the heat you produce, and dump it somewhere else.

Sure but there are different ways of doing it.

Google says they have the cold air come up from their raised floor.

Facebook does it differently- the cold air drops down:
http://opencompute.org/2012/08/09/water-efficiency-at-facebooks-prineville-data-center/

I'm no data center engineer but the Facebook way makes more sense to me.

If Google sold servers, HP and Dell would die overnight.

No. You're wrong on so many levels, it's hard to believable.

Google's solution is cheap, UNRELIABLE servers. I liked the idea of a built-in battery for about 5 seconds, until I realized that the PSU isn't going to have any way to do a weekly self-test of the battery, or allow hot-swapping it... the features that separate decent UPSes from low-end consumer crap. I liked the idea of motherboards stripped of unnecessary components, until I saw it only had a single Ethernet port (can't even do bonded/trunked NICs). I liked the all-14V power supply, until I noticed there's only a single PSU, no hot-swap.

If any of the above sounds like a good idea to you, you're not a Dell/HP/IBM customer. Even though my company does extensive clustering and has plenty of redundancy, we're not Google-scale, where it would be perfectly okay if a bunch of servers just crap out one night... If you've got your own compute-farms, fair enough, but my experience says there are fleetingly few companies who can make economic use of cheap junk servers, and many who want to try it are instead making a decision of false-economy that will come back to bite them.

Just the "12volt-only" power supplies with built-in batteries with "12volt-only" motherboards makes them more reliable than anything out there.

Have you ever touched a rack of servers before? Two big UPSes at the bottom of the rack, connected to two PDUs, connected, alternately, to each of the two hot-swap power supplies in each server. It's a pretty standard configuration, which is EXTREMELY reliable (unless the UPSes are complete crap, and they both run their self-tests at the same time--I'm looking at you APC!).

This configuration is infinitely more reliable than Google servers. Google's form of redundancy is switching to an entirely different DATACENTER at a moment's notice... The batteries are only there to smooth out minor power fluxuations, and soften the fall, so ALL the servers in a datacenter don't drop at the exact same moment in time.

HP and Dell either can't or won't license this from Google.

Probably BOTH. Like I said, the reliability just isn't there. If you don't care, you're probably a SuperMicro customer, rather than a Dell/HP/IBM customer.

But if your workload does fit with the Google server model, you can actually go one step-up, and get Facebook servers solutions, thanks to the Open Compute project they put together.

    http://opencompute.org/

The big difference being that, instead of an internal battery, Facebook has a rack of batteries for every two racks of servers, which connect to the DC-input of the PSUs. This is clearly more reliable than the Google model, because the centralized batteries allow you to routinely test, before they're needed, and swap dead batteries without taking any servers down. And since the smarts are in the (inexpensive) PSUs, you don't need special 12v-only motherboard designs at all.

You can see it all in-action in a number of Youtube videos, such as this one:

http://www.youtube.com/watch?v=JZUX3n2yAzY

"they [facebook] contribute little back"

http://opencompute.org/

https://github.com/facebook/

Facebook is giving back some rather large projects to the global open source community.

I think this is going to happen, but it will start at the big companies first and reach low end consumer hardware later. From the OpenCompute project About page, http://opencompute.org/about/ "The result is that our Prineville data center uses 38 percent less energy to do the same work as Facebook’s existing facilities, while costing 24 percent less."

Energy costs are a big concern at the major hosting, social networking, and search companies. Facebook, Google, Microsoft, IBM, Amazon, Ebay, etc... have millions of servers, so they can save at a minimum tens of millions of dollars in energy costs per year by switching their servers to more efficient designs. Eventually so many big companies will be buying this kind of hardware that it will show up on Amazon and Newegg and become the small business and home server norm too.

On the other hand, while I'm pleased about the entire development I think "open source hardware" is a misleading description. The external dimensions are open specifications, the component layouts are open, the power supply, the cooling system, etc... but I haven't seen anything from OpenCompute to indicate that the network cards, the processors, the graphics chips, the machine BIOS, etc... etc... is open source and free for anyone to reproduce at will. That's true open-source hardware, and at the rate things are going I would be surprised if I see an open source processor design that matches a 2010 Core i3 processor in my lifetime.

Reading the parent post feels like wiping your ass with silk.

Re OP: If you really want a FOSS personal cloud, you might be interested in knowing that Facebook documents its whole platform.

Also, unless cloud got dumbed down to the point where hosting anything on a single server is "being in the cloud", you might find it easier to build a fucking toaster.

The problem with 1" shims is that the copper power bars are still going to be in the way of the long server chassis. The pictures I saw had the copper bars running up the middle of the rack limiting your usable depth by quite a bit. (page 5)

The spec can be downloaded from here

At first, I thought this sounded like a stupid idea too. Then, I read the spec. They're not just changing the width of the equipment area, and it's not just an extra 2".

External width is unchanged from the 19" rack standard, it's still nominally 23.6". No replacement of floor tiles or room redesign necessary.

Equipment width is increased from ~17" (on a 19" rack) to 21", it's allows 4" wider equip.

Power is handled in 3 "zones" per tower. Each power zone provides 12.5V DC power on each of 3 independent pairs of power rails, No AC power supply is required for each piece of equipment, but they will need DC-DC converters and VRs to supply the voltages needed for their specific components. That saves some space on each device, and provides a slight improvement in efficiency. Because this is standardized worldwide, there is no need for each device to have different power circuitry for different countries.

Because a zone can have triple power rails, devices can use 1, 2, or 3 power rails to provide whatever level of redundancy is appropriate.

Space for switches is included in each rack, along with power monitoring/reporting circuitry per rack.

Battery backup power can either be built into the power supply for each zone, or supplied from a separate battery rack.

The specification allows for many AC or DC power sources, this is the only significant part of the spec that will vary by country as the power units will need to support the available AC and/or DC supply.

All devices are to be hot-plug compatible.

So, it does have a lot of advantages.

Here are the concerns I have with it:

All power rails appear to be exposed. While they are on the back, this could be a significant safety (personnel and/or fire) issue. Considering that you can up to 500A @ 12.5V DC running through the zone power rails, and potentially more for the main cabinet DC power rails, exposed seems like a bad idea.

The standard allows depths from 36" to 48". With the way devices connect to power rails on the back, it looks like this means you will need to use devices designed for the specific depth of your rack, or use a shim to extend the device to match the depth of the rack. I believe they should standardize on one, or at most 3 depths, and have a standard set of shims to connect the devices designed for the shorter depths to fit the deeper racks.

Close, Each bus bar pair can support up to 500A, which can be increased by adding more copper. Each bus bar pair can support up to 6KW, while power can be increased by using more copper. The bus bars can be adjusted for higher current, depending on the desired power per column, as they are interchangeable. Normally the Open Rack has three bus bar pairs installed, but it is also configurable with two or one bus bar pairs in the power zone: link

Do some basic research before you post. The Facebook Prineville datacenter is the most electrically efficient datacenter every built. The entire design, from the power & cooling to the servers themselves has been open sourced. Have a look at them yourself, and if you're such a damn efficiency expert IMPROVE upon them and Facebook will likely adopt your suggestions. Otherwise STFU.

They can also copy some ideas from Facebook (or others):
http://www.wired.com/wiredenterprise/2011/12/facebook-data-center/all/1
http://opencompute.org/
http://opencompute.org/2011/11/17/learning-lessons-at-the-prineville-data-center/

They'd probably have to modify the air filter/intake sections since in many places the air there is rather dirty. And sometimes it's not just "conventional" pollution but dust/sand blown in from the desert areas

They can also copy some ideas from Facebook (or others):
http://www.wired.com/wiredenterprise/2011/12/facebook-data-center/all/1
http://opencompute.org/
http://opencompute.org/2011/11/17/learning-lessons-at-the-prineville-data-center/

They'd probably have to modify the air filter/intake sections since in many places the air there is rather dirty. And sometimes it's not just "conventional" pollution but dust/sand blown in from the desert areas

The excess heat will be used to heat the office space, according to http://opencompute.org/.

The problem is that the heat generated by DCs is pretty low energy and pressure. You can't heat much up with it or travel it very far before you lose it all in transmission inefficiencies.

XMPP is now specified by RFC 6120 and RFC6121, but don't mind that. I think GP was referring to http://opencompute.org/

Here.