ARM Based Server Cluster Benchmarked

An anonymous reader writes "Anandtech compares the Boston Viridis, a server with Calxeda's ARM server technology, with the typical Intel Xeon technology in a server environment. Turns out that the Quad ARM A9 chip has it weaknesses, but it can offer an amazing performance per Watt ratio in some applications. Anandtech tests bandwidth, compression, decompression, building/compiling and a hosted web environment on top of Ubuntu 12.10." At least in their tests (highly parallel, lightweight file serving), the ARM nodes offered slightly better throughput at lower power use, although from the looks of it you'd just be giving money to the server manufacturer instead of the power company.

Comments

More to datacenter costs than power

[avandesande]

What if you are nearing the limits of the datacenter, cooling, power delivery etc. I don't have exact numbers but the cost for watt is greater than what you pay the power company.

Re:More to datacenter costs than power

[lucm]

What if you are nearing the limits of the datacenter, cooling, power delivery etc. I don't have exact numbers but the cost for watt is greater than what you pay the power company.

That's more complicated than it looks. On a first look it may seem like this does not change cooling requirements; most datacenter simply use a formula such as watts/3 to get a rough ideas of the needed BTUs. However the more you space out heat sources, the more natural cooling (convection) can do a magnificent job as the air flow is more optimally utilized. Or maybe having more heat sources can mitigate the benefits; it's hard to tell, that's why God created CFD applications.

The only hard limits would be physical space and power distribution inside the datacenter.

This being said, for years datacenters have been built without actual knowledge of how technology would evolve over their lifetime and so far they managed to work with neverending change. This just is one more.

Re:More to datacenter costs than power

[poetmatt]

actually the power cost is a savings, but you are correct that physical space is also it's own cost. They showed (in contrast to the trollmitter) that the servers were relatively close in performance in real world scenarios, but there are absolutely things that are more built for x86 at the moment.

Re:More to datacenter costs than power

[hawguy]

That's more complicated than it looks. On a first look it may seem like this does not change cooling requirements; most datacenter simply use a formula such as watts/3 to get a rough ideas of the needed BTUs. However the more you space out heat sources, the more natural cooling (convection) can do a magnificent job as the air flow is more optimally utilized.

Every densely packed datacenter I've seen uses forced air cooling to suck in cool air from the cool aisles and blow warm air into the hot isles. Natural convection seems less important in such a scenario.

Re:More to datacenter costs than power

[swb]

Many (most?) data centers I've been in have been buildings converted from some other use -- office buildings, warehouses, etc. But regardless of how they are built, they always seem to have relatively low ceilings, even in converted spaces where they rip out the ceiling grid.

I get the density argument, but I often wonder if someone built a data center with a 50 foot ceiling (a large, flat building) if you wouldn't gain some cooling benefit from convection that would be worth the sacrifice in vertical density versus the cost in intensive forced air cooling.

Re:More to datacenter costs than power

[hawguy]

Many (most?) data centers I've been in have been buildings converted from some other use -- office buildings, warehouses, etc. But regardless of how they are built, they always seem to have relatively low ceilings, even in converted spaces where they rip out the ceiling grid.

I get the density argument, but I often wonder if someone built a data center with a 50 foot ceiling (a large, flat building) if you wouldn't gain some cooling benefit from convection that would be worth the sacrifice in vertical density versus the cost in intensive forced air cooling.

I don't see why high ceilings would make a difference - you have X BTU/hr of heat to remove - letting it accumulate at the ceiling doesn't seem to make much difference (unless you have a lot of conductive losses through the walls/ceiling).

I can't beleive there's any economic argument for giving up 66% - 75% of your potential floor space (12 or 16 foot ceilings versus 48 foot ceilings) just to let heat rise to the ceiling.

Hot aisles make the heat exchangers more efficient.

Re:More to datacenter costs than power

[lucm]

Many (most?) data centers I've been in have been buildings converted from some other use -- office buildings, warehouses, etc. But regardless of how they are built, they always seem to have relatively low ceilings, even in converted spaces where they rip out the ceiling grid.

I get the density argument, but I often wonder if someone built a data center with a 50 foot ceiling (a large, flat building) if you wouldn't gain some cooling benefit from convection that would be worth the sacrifice in vertical density versus the cost in intensive forced air cooling.

I don't see why high ceilings would make a difference - you have X BTU/hr of heat to remove - letting it accumulate at the ceiling doesn't seem to make much difference (unless you have a lot of conductive losses through the walls/ceiling).

I can't beleive there's any economic argument for giving up 66% - 75% of your potential floor space (12 or 16 foot ceilings versus 48 foot ceilings) just to let heat rise to the ceiling.

Hot aisles make the heat exchangers more efficient.

That's not the point of improved convection. The idea is not to let heat rise, it is to give more room to the air so the flow can establish wider patterns, bringing cooler air in contact with the heat source without requiring additional power for blowers. Contrary to popular misconception, there is no need to blow artic air on a server to cool it down.

Re:More to datacenter costs than power

[hawguy]

Many (most?) data centers I've been in have been buildings converted from some other use -- office buildings, warehouses, etc. But regardless of how they are built, they always seem to have relatively low ceilings, even in converted spaces where they rip out the ceiling grid.

I get the density argument, but I often wonder if someone built a data center with a 50 foot ceiling (a large, flat building) if you wouldn't gain some cooling benefit from convection that would be worth the sacrifice in vertical density versus the cost in intensive forced air cooling.

I don't see why high ceilings would make a difference - you have X BTU/hr of heat to remove - letting it accumulate at the ceiling doesn't seem to make much difference (unless you have a lot of conductive losses through the walls/ceiling).

I can't beleive there's any economic argument for giving up 66% - 75% of your potential floor space (12 or 16 foot ceilings versus 48 foot ceilings) just to let heat rise to the ceiling.

Hot aisles make the heat exchangers more efficient.

That's not the point of improved convection. The idea is not to let heat rise, it is to give more room to the air so the flow can establish wider patterns, bringing cooler air in contact with the heat source without requiring additional power for blowers. Contrary to popular misconception, there is no need to blow artic air on a server to cool it down.

But you get the same benefit from hot aisles/cold isles for the price of some baffles to separate the hot/cold air - much cheaper than 50 foot ceilings.

Re:More to datacenter costs than power

[Anonymous Coward]

Precisely. We pay for our datacentre by the rack, each rack comes with a certain amount of power - 14A I believe. You can get up to 18A, but the cost for 18A is about double that of 14A. The DC has a set maximum cooling it can achieve, divide that by the number of racks you can fit in, and that is how much you can have per rack.

Therefore, our racks can have at most 14A worth of kit. This means that many of our racks have 4U-8U completely vacant. We can't put anything in there, because it would blow our power allowance. If we can fill our racks with low power ARM servers, we can potentially serve more from each rack, and then perhaps we won't need so many racks, giving us an actual cost saving.

You'll be giving money to someone

[Anonymous Coward]

" although from the looks of it you'd just be giving money to the server manufacturer instead of the power company."

Isn't that the truth. This is the new market paradigm for just about everything. You no longer pay for a product or service. You pay for what you get out of it.

For example, the way fuels have been priced for the last decade or so (since the first runup after 9/11), you pay for the energy you get out of the fuel, not the fuel itself.

Case in point, diesel cars are 20% more efficient than gasoline cars, so diesel fuel costs 30% more. Natural gas furnaces are 25% more efficient than fuel oil furnaces, so natural gas costs 30% more per BTU input than fuel oil.

Corporate America is going to stick it to you no matter what you do to get ahead. If you find a clever way to save money, our greedy corporate masters will STEAL it from you one way or another, because at the end of the day, they are pulling all the strings and turning all the knobs.

HP makes two inkjet printers that are identical in every way, except one has an adjustable ink density in software, allowing you to reduce ink usage by 30%. Ink cartridges for that printer cost 40% more.

You can get "free" energy by installing a solar array, but the moment you grid-tie the power company only gives you credit for generation, so you end up paying THEM for "transmission" and "transition" on your own goddamn electricity that you paid to produce for your own home.

Shit like that...

Re:You'll be giving money to someone

[rufty_tufty]

Another way of saying it is that capitalism sort of works. Or at least the laws of supply and demand do.
Product A is cheaper than product B. Demand for Product A increases. Price of Product A increases as price of product B decreases. Per unit of usefulness they end up costing the same.
To get back on topic though in this case energy costs are fixed by supply and demand. At the moment ARM cores are in server terms a niche product so you don't get the benefits of bulk supply. Those efficiencies can be improved on though, cost of energy less so.

Re:You'll be giving money to someone

[gl4ss]

it's corporates swindling corporates though with these.

they're paying premium for cheaper hw so they can claim to be green.

Re:You'll be giving money to someone

[poetmatt]

actually, no. This is the definition of capitalism failing. Not due to capitalism or overregulation or underregulation, but a lack of clear regulation.

Re:You'll be giving money to someone

[rufty_tufty]

Can you explain this please, I don't get your argument.
I was referring to:

For example, the way fuels have been priced for the last decade or so (since the first runup after 9/11), you pay for the energy you get out of the fuel, not the fuel itself.

As far as I am aware finding the price of something by it's value to society is a good thing. How would you rather it worked?
The alternative I can see is that some things are socially or politically favoured and so are forced down our throats whether it's a good idea or not. As long as all costs* are taken into account then what's the problem?
*As a counter example I know not all costs of coal are taken into account, and they should be and this is a failing of capitalism but that's not related to the paying for the energy content of something which was the point of the OP.

Re:You'll be giving money to someone

[afidel]

so natural gas costs 30% more per BTU input than fuel oil.

What planet do you live on?

heating oil $4.058/gallon, 138,700 BTU/gallon = $29.26 per million BTU
natural gas $ 0.55143 per hundred cubic feet (ccf), 102,000 BTU's per ccf = $5.41 per million BTU

Re:You'll be giving money to someone

[Anonymous Coward]

diesel cars are 20% more efficient than gasoline cars, so diesel fuel costs 30% more.

Not to derail your point, but diesel engines reach peak efficiency MUCH sooner than gasoline. Your average efficiency with diesel is closer to 40% over gasoline.

Re:You'll be giving money to someone

[afidel]

Oh, and the natural gas price includes delivery but the fuel oil price does not so the discrepancy is even larger.

Re:You'll be giving money to someone

[Phreakiture]

Natural gas furnaces are 25% more efficient than fuel oil furnaces, so natural gas costs 30% more per BTU input than fuel oil.

I have to disagree. I did the math on this years ago when deciding whether to replace my gas furnace with another gas furnace or get an oil furnace. The data point very clearly in the opposite direction of what you are saying.

This may vary geographically, but the most recent data I could find for where I live (upstate New York) is this: Gas costs $11.49 for 1000 cu.ft. as of last November ; #2 home heating oil costs $3.934 per gallon as of the same point in time. Natural gas has an energy density of 950-1150 BTU per cu.ft.; heating oil is 139,600 BTU per gallon. That works out to a price of $9.991-$12.095 for 1,000,000 BTU worth of natural gas, or $28.181 for 1,000,000 worth of heating oil. In sort, because of the lower efficiency of oil heaters (25% is exaggerated, BTW), you get shafted twice for using oil.

The only real difference is with oil, you get to choose who does the shafting.

Re:You'll be giving money to someone

[Anonymous Coward]

In the case of grid-tie surcharges, use that extra electricity for something if you don't want to sell. Split water into hydrogen and oxygen for the purest possible water. Add salt and you can extract chlorine gas in quantity. Cool it down and you have rocket fuel. Feed it to storage tanks and you can use a fuel cell as both a heat source and a generator in the winter.

Re:You'll be giving money to someone

[aaarrrgggh]

He is talking about the furnace cost, not the fuel. The fuel cost nets things out to around zero.

Re:You'll be giving money to someone

[Anonymous Coward]

It's close to the tragedy of the commons where -- provided that the energy consumption is viewed at the level of the whole society -- the incremental efficiency gains turn into economic deficiency to the society at large. The cycle can be broken by revolutionary changes to the efficiency but the benefits might ultimately go to the new set of domestic players or exported out of the society to another society.

Re:You'll be giving money to someone

[afidel]

Bryant preferred series 110,000 Btu 2 stage 95% efficient: About $1,725

Bryant Preferred 80 115,000 Btu 374RAN oil furnace, 83.5% efficient: $1,779

Same quality unit from the same manufacturer, same input BTU, MORE expensive for the oil and you have to add a fuel tank to the cost of the oil unit. He's simply wrong.

Re:You'll be giving money to someone

[Archangel Michael]

Passing up Moderation to ask the poster a question, but first, recap of the part I'm asking about:

Corporate America is going to stick it to you no matter what you do to get ahead. If you find a clever way to save money, our greedy corporate masters will STEAL it from you one way or another, because at the end of the day, they are pulling all the strings and turning all the knobs.

Question: Are our government masters ever "Greedy" and do they ever "STEAL".

The reason I ask, is because people who make statements like you did, tend to believe that Government can do no wrong, as long as they "stick it to the rich". I find this mentality to be childish and self serving.

To the point about fuel prices (and other things), have you ever considered that the government restrictions cause some of the price differentials? I mean, after all, we can't drill, or build pipelines or refineries or ... due to government restrictions. Perhaps it is government that is "Greedy" (power hungry) Stealing (via unsustainable social programs) from our children. But when we talk about it, we are labeled "tossing grandma off the cliff" and having to make choices between "fire, police and education" while our Government Masters are busy planning their next golfing trip, birthday bash with Beyonce and Adel and closing White House Tours down.

In other words, your view is simple, narrow minded and out of touch with the real Greed.

Re:You'll be giving money to someone

[Anonymous Coward]

Most consumers don't want to buy gasoline or natural gas for the sake of just having it sit around. Their actual demand is for energy it uses, so this is the expected and desired result.

Say tomorrow someone invents a process that doubles the efficiency of natural gas, so demand will skyrocket as the price is currently low. Are you suggesting that regulation should force that price to stay the same? In other words, forcing an inelastic supply, despite oil and gas products tending to be pretty elastic.

Instead, the price will balance out as so that what people are looking for, energy, will be price balanced taking into account difficulty of different sources (both getting the source, and the efficiency of using it). Even though this price change happens, the total efficiency will go up as any individual product efficiency goes up, even if the prices increase for said product (in effect, the total supply of energy increases when the efficiency of any individual source improves, so the average price will drop for a given demand). And there still is room for regulation when dealing with externalities that are otherwise not accounted for, but regulations are not needed just because competing products will rebalance prices.

Re:You'll be giving money to someone

[farble1670]

To the point about fuel prices (and other things), have you ever considered that the government restrictions cause some of the price differentials? I mean, after all, we can't drill, or build pipelines or refineries or ... due to government restrictions.

so .... what does a govt get out of restrictions? under the table payments from those deep-pocketed environmental non-profits. or maybe it's Big Solar lobbyists? wait ...

you can make the point that the govt is misguided in placing restrictions drilling, pipelines, etc ... but it isn't greed. there's clearly more $ to be had all around by sucking every drop out of domestic oil deposits.

Re:You'll be giving money to someone

[Anonymous Coward]

> Split water into hydrogen and oxygen for the purest possible water.

Which is excellent for cleaning your chemistry set, but not really suitable for drinking. 'Purity' is vastly overrated.

already in as a file server....

[johnjones]

ARM based servers already has file server cluster design win's

http://www.eetimes.com/electronics-news/4407353/Baidu-taps-Marvell-for-32-bit-ARM-storage-server

what will be intresting is ability to leverage designs for phones as clusters because then you can use the volume e.g. SOC for phones costs $20 roughly so imagine filling a DC with those...

have fun

John Jones

Re:already in as a file server....

[Anonymous Coward]

Hell, I wanna built my PC from those! 1024-core (@1.4GHz). 512 GB of RAM. 16TB of 512-associative SSD storage. That's the equivalent of e.g. 512 Samsung Galaxy S III phones. Put them all in in copper slots with water channels inside the copper separators, and a nice big passive (optionally active) radiator outside.

Unfortunately, such a phone is much more expensive because of all the other stuff inside. But if you’d remove all that, including batteries, displays, wireless tech, etc...

Re:already in as a file server....

[Austerity Empowers]

Except, of course, the Marvell chip sucks.

Re:already in as a file server....

[Anonymous Coward]

so get 250 exynos 2 chips on daughterboards, a mainboard to handle internode coms and power, a storage array, and some cooling that is capable of handling the kilowatt of heat that thing will put out on load. Exynos 2 chips are 40 bucks when bought in bulk and they are a quad core SOC. Bonus points if you have openCL parts that can run on their GPUs.

Conclusion

[Anonymous Coward]

This setup is slower than a beowulf cluster of beowulf clusters

Power cost

[Anonymous Coward]

Depending on where you are, even a small percentage of power savings could pay for the hardware fairly quickly. Here in the Silicon Valley at least, PG&E charges upwards of $0.30/kWh for the average home power consumer, and their rates go higher based on usage tiers. Running a data center of supercomputer cluster wouldn't be cheap when it costs me ~$300/month to power my desktop PC and toaster oven.

Re:Power cost

[gl4ss]

Depending on where you are, even a small percentage of power savings could pay for the hardware fairly quickly. Here in the Silicon Valley at least, PG&E charges upwards of $0.30/kWh for the average home power consumer, and their rates go higher based on usage tiers. Running a data center of supercomputer cluster wouldn't be cheap when it costs me ~$300/month to power my desktop PC and toaster oven.

but the joke is the hw is inferior and has cheaper parts...

article:
" $20,000 is the official price for one Boston Viridis with 24 nodes at 1.4GHz and 96GB of RAM. That is simply very expensive. A Dell R720 with dual 10 gigabit, 96GB of RAM and two Xeons E5-L2650L is in the $8000 range; you could easily buy two Dell R720 and double your performance. The higher power bill of the Xeon E5 servers is that case hardly an issue, unless you are very power constrained. However, these systems are targeted at larger deployments."'

12k gets you a lot of electricity. you could run 4 kw for a year for the price difference even with your ridiculous domestic use electricity price(datacenter would buy it cheaper) and it's not like the arm cluster wouldn't use any power at all.

Re:Power cost

[Guspaz]

I think you missed this part:

However, these systems are targeted at larger deployments.

And this part:

Buy a whole rack of them and the price comes down to $352 per server node, or about $8500 per server. We have some experience with medium quantity sales, and our best guess is that you get typically a 10 to 20% discount when you buy 20 of them. So that would mean that the Xeon E5 server would be around $6500-$7200 and the Boston Viridis around $8500.

It's still more expensive, but the gap narrows substantially.

Re:Power cost

[hawguy]

Depending on where you are, even a small percentage of power savings could pay for the hardware fairly quickly. Here in the Silicon Valley at least, PG&E charges upwards of $0.30/kWh for the average home power consumer, and their rates go higher based on usage tiers. Running a data center of supercomputer cluster wouldn't be cheap when it costs me ~$300/month to power my desktop PC and toaster oven.

PG&E uses a tiered rate structure so while the highest rate may be in the 34 cent/KWh range, the average rate for most homes is lower.

Here are the Tiers:

Baseline
Tier 2 101%-130%
Tier 3 131%-200%
Tier 4 201%-300%
Tier 5 >300%

Here are the rates (Residential E1, no time of day):

$0.13230
$0.15040
$0.30025
$0.34025
$0.34025

Baseline quantities depends on region - my single family townhome has a baseline of 273KWh. Baseline is supposed to be 50 - 60% of an average home's power usage. We tend to stay under 300KWh/month, so pay close to the baseline rates. Though if you had a 400W server at home running 24x7, its 288KWh/month of usage would likely all be charged in a higher tier.

I think commercial rates are around 19 cents/KWh (no tiers), industrial rates are around 13 cents/KWh.

Re:Power cost

[farble1670]

Running a data center of supercomputer cluster wouldn't be cheap when it costs me ~$300/month to power my desktop PC and toaster oven.

and of course you are exaggerating. i live in san jose, and during the summer when i don't have heat running (and i don't have AC), my gas+electric bill is around $70. that's for a small house. maybe it is those grow lights in your basement?

Energy Comparison

[turb]

While the page with benchmark data includes an intel v ARM comparison, when it came to the power consumption charts there was no intel data to be found. None.

If one of the major themes of the product is power consumption, wouldn't it stand to reason that Intel numbers to compare would be critical as part of the review?

Re:Energy Comparison

[afidel]

Did you miss this page or something?

Re:Energy Comparison

[fatphil]

Not quite. When comparing computational performance, they compared against Atoms, and when comparing power consumption they compared against Xeons.

That's deliberately misleading, and even I as an ARM fan (who uses no Intel CPUs at home at all), I think this is bogus. (But I don't blame ARM, I blame AT and the server prociders who were spamvertising on AT.)

Re:Energy Comparison

[kangsterizer]

that.
puts them in my "trash press" list.

Re:Energy Comparison

[fatphil]

Alas, you'd probably have to put /. itself on that list too :-(

You have to be selective at all levels.

Re:Energy Comparison

[Anonymous Coward]

Why are many people here so quick to dismiss an article without even properly reading it?
There is a reason for this... There are very few Atom servers around, and we did not have one in the lab. So there is no power measurements possible as it would not be fair to compare a full blown server to a motherboard. And we did compare computational performance on both Xeon and Atom.

GPU?

[Steve Baker]

I guess these A9's are not SoC integrated with a GPU, like say an Exynos. 24 GPUs + 96 ARM cores in a box could make them attractive for some compute applications. High end GPUs would probably smoke them good though.

A9's are nice, but more compelling as a desktop replacement for the spreadsheet and wordprocessor set, or low-power home servers / appliances. They're just seriously bandwidth challenged, but the average corporate desktop doesn't need it. Replacing hundreds of x86 desktops with Exynos's and you'd see a pretty quick ROI with the power savings.

A15's and the upcoming ARMv8's will be more interesting here, but as they ramp up bandwidth and performance, they'll likely meet Intel in the middle.

Re:GPU?

[Austerity Empowers]

They could integrate, but high end GPUs burn 120W/GPU and are good only at crunching numbers.These are good at serving web pages, files and routine DB work. And then go to sleep (and use very little power) when people aren't asking for web pages. They're not intended to replace Xeon's or GPUs for number crunching, the realization is that number crunching isn't that important for a lot of applications

It's not entirely unlike the cell phone/tablet vs. Desktop debate. For most people, cell phones offer all the computing power they will ever need. A few people need desktops. The belief is that most datacenters dont' crunch so much and are bound by hard disks, network and how much power their power company can possibly provide.

Re:GPU?

[Shinobi]

" the realization is that number crunching isn't that important for a lot of applications"

The reality is, it's important for a LOT of applications, but it's in the background. SSL is just one example.

As a VPN gateway for example, I think the Xeon would just smash any figures the ARM cluster could put up, incl watt/connection etc

Re:GPU?

[UnknowingFool]

What he means is that the kind of number crunching that would favor a GPU over a CPU is not for many applications.

Surprised no one has made this connection yet

[Burz]

the ARM nodes offered slightly better throughput at lower power use, although from the looks of it you'd just be giving money to the server manufacturer instead of the power company.

So then... GIVE your money to the server manufacturer instead for crissesake. There seems to be an obvious environmental benefit to be had.

wrong chips: we need numbers on A-53/A-57

[Anonymous Coward]

Lets not pretend that 32bit cpus make any sense on the server nowadays. We need the pure address space, no workarounds.
The designs we are waiting for are the A-53, the A-57 64bit ARM chipsets, and the big-little combinations.

I want to know how the silicon based on those designs coming out from Applied MicroSystems, AMD, HiSilicon etc perform, in absolute and in a per-watt basis. For now there are no numbers I know of.

So please rerun this whole experiment as soon as the first A-57s (especially) and the A-53s come out!

a9's are 32bit

[mjwalshe]

So of limited use for HPC clusters

Re:a9's are 32bit

[fatphil]

So why were they better than Xeons on some of the data-processing rate benchmarks?

Re:a9's are 32bit

[mjwalshe]

careful selection of benchmarks :-)