30,000-Core Cluster On Amazon EC2

← Back to Stories (view on slashdot.org)

30,000-Core Cluster On Amazon EC2

Posted by ryuzaki0 on Tuesday September 20, 2011 @06:36AM from the still-needs-a-few-more-cores dept.

Joining the ranks of accepted submitters, hooligun writes with an article in Ars Technica about a rather large cluster built on EC2. From the article: "The details are impressive: 3,809 compute instances, each with eight cores and 7GB of RAM, for a total of 30,472 cores, 26.7TB of RAM and 2PB (petabytes) of disk space. Security was ensured with HTTPS, SSH and 256-bit AES encryption, and the cluster ran across data centers in three Amazon regions in the United States and Europe."

23 of 59 comments (clear)

Min score:

Reason:

Sort:

Re:use? by bigjocker · 2011-09-20 06:44 · Score: 2, Funny

They are using it to pump the economy. The heating produced by this cluster must be cooled with extra air conditioning systems, increasing the demand for power and for air conditioning unis, thus creating new jobs and incentivizing the research for new energy sources.

--
Life isn't like a box of chocolates. It's more like a jar of jalapenos. What you do today, might burn your ass tomorrow.
First time accepted submitter Beowulf by clyde_cadiddlehopper · 2011-09-20 06:51 · Score: 1

Imagine the possiblilities. /. on steroids.

--
Obi-Wan: "I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror and were sudden
$1279 per hour by certron · 2011-09-20 06:54 · Score: 2

Before anyone else asks what I was about to, the full title of the article is: $1,279-per-hour, 30,000-core cluster built on Amazon EC2 cloud
How does that compare to the cost-per-core-hour for other Amazon EC2 offerings? Is this a value meal deal or just a lot of burgers?

--

fair.org counterpunch.com truthout.com indymedia.org salon.com
eff.org guerrilla.net debian.org gentoo.org
1. Re:$1279 per hour by mikeytag · 2011-09-20 07:03 · Score: 3, Informative
  
  The article said each instance had 7GB of memory and 8 cores. That would translate to the High-CPU Extra Large Instance Type:
  High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform
  Source: http://aws.amazon.com/ec2/
  That instance type will run you $0.68/hour standard or $0.24/hour spot. (US-East Pricing) (Spot pricing allows you to take advantage of unused EC2 instances at a discount. Also worth noting is that spot pricing changes over time.)
  30,000 cores equates to 3,750 instances across different regions. Here is the breakdown on hourly pricing for standard and spot. (Reality is it was probably a mixture of both and the pricing for different regions varies).
  Standard US-East: $2,550/hour
  Spot US-East: $900/hour
  The exact mix of machines in each region wasn't specified but $1,279/hour sounds about right if there is a mix of standard vs spot across different regions.
2. Re:$1279 per hour by geekthecat · 2011-09-20 15:00 · Score: 1
  
  It More like a SH!t burger! Wow! The performance is not impressive at all, yet the money there making is. Because the servers they are using are probably IBM POWER7. The article states that last jun 7,000 cores on EC2 was capable of ranking at 232 on the Top 500 list of super computers with a performance of 41.82 Teraflops. So looking over the list and comparing how a 7,000 cores of POWER7 system will do. We see at rank 50 a 6,912 cores of POWER7 with a performance of 212.12 Teraflops. Now lets up the cores to 30,000, so (30/7=4.28571) now 4.28571*41.82 = 179.22857 Teraflops for an INTEL i7 solution, and (30/6.9 = 4.34783) now 4.34783 * 212.12 = 922.2617 Teraflops to an IBM POWER7 solution. Wow! IBM's POWER7 has over 5 times the performance of INTEL's i7. So the cost of 30,000 POWER7 cores would be (30,000/16 = 1,875) which equates to 1,875 PS702 Blade severs each at a cost of $16,544.00 so the total cost would be (1,875 * $16,544.00 = $30,020,000.00) roughly 30 million and the cost of 30,000 INTEL i7 would be (30,000/12 = 2,500) so 2,500 HP Proliant BL2x220 blade server would cost (2,500 * $15,059 = $37,647,50) roughly $37 million. Now if AMAZON charges $1,279 per hour for 179.22857 Teraflops of 30,000 cores of i7 prcessors, then 30,000 cores of POWER7 would be 922.2617 Teraflops an increase in performance by a factor of (922.2617/179.22857 = 5.14573) so 5 time the performance 5 time the profit. Profit would be (5 * $1,279.00 = $6,395.00) so $6,395.00 * 24 hours * 365 days = $56,020,200.00 million using IBM's POWER7 or ($56,020,200.00 / 5.14573 = $10,886,735.2154) about $11 million using INTEL's i7 processors. So in conclusion you'll pay more for INTEL's i7 products and they're 5 times slower then IBM's POWER7. Just goes to show how many idiots there are in IT.
Re:use? by 0123456 · 2011-09-20 06:54 · Score: 1

to run Crysis
Ray-traced :).
Isn't EC2 really a cluster? by pz · 2011-09-20 07:01 · Score: 1

Help me understand something here ... isn't EC2 really one gargantuan cluster far bigger than 30,000 cores? So why is it news that it ran a big job? Was there some significant step forward in software that allowed features that were not previously available on EC2?

--

Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
1. Re:Isn't EC2 really a cluster? by Happy+Finish · 2011-09-20 07:11 · Score: 1
  
  Help me understand something here ... isn't EC2 really one gargantuan cluster far bigger than 30,000 cores? So why is it news that it ran a big job? Was there some significant step forward in software that allowed features that were not previously available on EC2?
  TFA is angled more at the fact that anyone can go out and rent something like this for their own ends.
Re:use? by hedwards · 2011-09-20 07:05 · Score: 1

Hacking the Gibson?
But, what was their password? by G3ckoG33k · 2011-09-20 07:15 · Score: 1

But, what was their password? So many details about that computer, but no password...
Link a percentage of the top 500 together! by Commontwist · 2011-09-20 07:31 · Score: 1

How powerful would one estimate linking multiple cloud and, of ten percent of the top 500 supercomputers would be? That would be one massive number cruncher.
1. Re:Link a percentage of the top 500 together! by mscman · 2011-09-20 08:41 · Score: 1
  
  Cloud computing and the Top500 computers are comparing different things. Generally, "Clouds" cannot efficiently run codes you would run on a Top500 machine, and vice-versa. They are large machines serving different purposes.
2. Re:Link a percentage of the top 500 together! by F.Ultra · 2011-09-20 09:22 · Score: 1
  
  They are actually set up quite similarly, the key difference is that cloud usually uses virtualization while the super computers doesn't so there is about 5-10% slowdown which you have to compensate by using more nodes.
3. Re:Link a percentage of the top 500 together! by mscman · 2011-09-21 03:28 · Score: 1
  
  No, they really aren't. I work on a top 20 machine, and can tell you that attaching this via a high-latency interconnect (read: the web) would completely kill the purpose of using this machine. And no, you cannot just "compensate by using more nodes." Amdahl's law kills that idea right out. I've worked in both "cloud computing" (back when it was known as "grid computing") and HPC or High-Performance Computing. While they are similar in some ways, they are designed to fulfill different purposes and are best suited to different job types. The codes being run by Cycle for this project are EP codes, ones that would not necessarily benefit from the top 50 machines in the world. These machines are better suited for MPP work, which depends more on low-latency, high-speed interconnects.
4. Re:Link a percentage of the top 500 together! by F.Ultra · 2011-09-21 10:16 · Score: 1
  
  Ok so the nodes in the cloud is not connected via Infiniband but by Gigabit Ethernet but what made you think that they where connected via the web? And still I don't think that it invalidates that they are constructed quite similarly. Oh and of course you can offer inifiband clustered nodes as the cloud.
5. Re:Link a percentage of the top 500 together! by F.Ultra · 2011-09-21 10:19 · Score: 1
  
  The machines that make up the cloud is set up quite like supers. It's just that they might get a little lousier connection (GB Ethernet instead of Infiniband) but that is by choice of Amazon, there is nothing stopping them from selling IB connected nodes in their cloud. Of course in this specific example they connected more than one cloud with each other and then they had to communicate over the Internet. But that was also by choice and could be compared to when you connect different super centers which happens from time to time.
Bandwidth? by oliverk · 2011-09-20 07:33 · Score: 1

Didn't we just read that the US has fallen to #25 on the international speed list? So, is this like serving up Skynet over a 28.8 modem?

--
---- Please be nice in case my Slashdot karma ~= my real life karma.
Charity? by damuhatori · 2011-09-20 07:39 · Score: 1

They should donate a couple of hours a month to curing a disease.
Re:Windows 8 by Fortunato_NC · 2011-09-20 08:51 · Score: 1

I know I'm feeding the troll but...
I'm running the Windows 8 developer preview (64-bit) on a five and a half year old laptop. Granted, I kicked the RAM up to 4GB ($44 shipped from NewEgg) and replaced the Core Duo with a Core 2 Duo (a T5600, $25 used on fleabay buy it now), but it runs well at 1900x1200 on hardware I basically rescued from the dumpster. You need to update your stock lines and stop mindlessly bashing.

--
Blogging Weight Loss, Distance Education, and more at verlin.com
Re:security for sure by GameboyRMH · 2011-09-20 10:03 · Score: 1

Of course if Amazon had to, they could rip the storage encryption key from the VM's RAM...

--
"When information is power, privacy is freedom" - Jah-Wren Ryel
communication latency by Orp · 2011-09-20 11:18 · Score: 1

Neat, but for any job that isn't embarrassingly parallel, communication latency and speed will kill you when your nodes are spread across continents. If you're not doing any communication, well then groovy. Usually these large core servers are only 'earning their keep' when you're taking advantage of very fast interconnect hardware and doing things that can't be done by just a bunch of CPUs.

--
A squid eating dough in a polyethylene bag is fast and bulbous, got me?
Re:use? by gregrah · 2011-09-20 15:03 · Score: 1

I love the fact that there are at least 5 answers above mine, and no one has actually RTFA, so no one actually knows.
Re:Why explain Petabytes by tehcyder · 2011-09-21 01:56 · Score: 1

If you don't know the scale from yocto to yotta, then you need hand in your geek card.
How many digits do yu know pi to? I always say that if you don't know at least the first thousand, you're no geek, and should have your geek card forcibly removed at gunpoint.

--
To have a right to do a thing is not at all the same as to be right in doing it