Installation of Blue Waters Petaflop Supercomputer Begins

← Back to Stories (view on slashdot.org)

Installation of Blue Waters Petaflop Supercomputer Begins

Posted by Unknown on Monday January 30, 2012 @12:15PM from the too-many-flops-to-fail dept.

An anonymous reader writes "The National Center for Supercomputing Applications at the University of Illinois is finally getting the troubled Blue Waters supercomputer installed. After IBM walked away from the project after 3 years of planning, Cray stepped in to pick up the $188 million contract. Now, in around 9 months time, Blue Waters should be fully operational and achieve performance of 1 petaflop or more. As for the hardware... who wouldn't want access to 235 Cray XE6 cabinets using AMD 16 core Opteron 2600 processors with access to 1.5 petabytes of memory (4GB per chip) and 500 petabytes of local storage."

45 of 86 comments (clear)

Min score:

Reason:

Sort:

When can I get one on my desktop? by sehlat · 2012-01-30 12:16 · Score: 2

That's the real question.
1. Re:When can I get one on my desktop? by gentryx · 2012-01-30 12:50 · Score: 4, Informative
  
  Let's look this up. 7 years ago #1 on the Top500 was an IBM BlueGene/L at 70 TFLOPS. I can't see that performance anywhere close on the desktop or even on the notebook market.
  Assuming you're running a good SLI systems and that your GPUs actually deliver the performance the manufacturer is claiming them to have, you'd get in the best case something around 1.5 TFLOPS which corresponds roughly to a 1998 ASCI Red.
  
  --
  Computer simulation made easy -- LibGeoDecomp
2. Re:When can I get one on my desktop? by dak664 · 2012-01-30 13:00 · Score: 4, Informative
  
  At a gigaflop per watt that's 24 MWh a day, $1.3 million a year in power bills at $0.15/kWh.
3. Re:When can I get one on my desktop? by itamblyn · 2012-01-30 13:01 · Score: 1
  
  I've found it's actually closer to about 15 years.
4. Re:When can I get one on my desktop? by Arakageeta · 2012-01-30 13:36 · Score: 1
  
  SLI is absolutely useless for CUDA-based (Cray's uses NVIDIA GPUS) GPGPU.
5. Re:When can I get one on my desktop? by gentryx · 2012-01-30 13:46 · Score: 1
  
  Sure, I was using SLI as an abbreviation for a multi GPU system. And since I was refering to a hypothetical desktop, it might even run AMD GPUs, not just Nvidia chips). But yeah, I know: AMD GPUs generally suck at scientific computing. Sadly.
  
  --
  Computer simulation made easy -- LibGeoDecomp
6. Re:When can I get one on my desktop? by SrLnclt · 2012-01-30 13:57 · Score: 5, Informative
  
  The University of Illinois generates much of their own power, and has relatively low electric rates because of this. This year the rates are posted as $0.0754/KWh. Its also doubtful they will be operating continuously at peak capacity.
  
  And on a related note, the building housing Blue Waters has been certified LEED Gold by the USGBC in an effort to minimize the energy and cost impact of operating the new facility.
7. Re:When can I get one on my desktop? by ajlitt · 2012-01-30 14:05 · Score: 1
  
  When you spend that much for today's fastest computer, you're foolish not to run it at peak capacity all the time. If you can't schedule enough jobs to keep it busy, why have one?
8. Re:When can I get one on my desktop? by Anonymous Coward · 2012-01-30 16:47 · Score: 1
  
  Seems worth it, considering how quickly I can accumulate Bitcoins once it's in my possession.
9. Re:When can I get one on my desktop? by afabbro · 2012-01-30 17:22 · Score: 1
  
  Let's look this up. 7 years ago #1 on the Top500 was an IBM BlueGene/L at 70 TFLOPS. I can't see that performance anywhere close on the desktop or even on the notebook market.
  Whoosh...
  
  --
  Advice: on VPS providers
10. Re:When can I get one on my desktop? by gentryx · 2012-01-30 20:47 · Score: 1
  
  That's true. Just have a look at the queues of some of the petaflop machines. They're usually at least 5x oversubscribed, meaning that more projects apply for compute time than is actually available.
  
  --
  Computer simulation made easy -- LibGeoDecomp
11. Re:When can I get one on my desktop? by Anonymous Coward · 2012-01-30 20:53 · Score: 1
  
  If you can't schedule enough jobs to keep it busy, why have one?
  This one's easy: to get the next one. Let me explain.
  Supercomputers are an easy way to get government grants. Like any computer, they can do all sorts of useful things that are (or can at least be portrayed as) in the public interest. Combine this with the competence of a typical government, and it is rather simple to convince a government that since bigger is better, a bigger computer can obviously do more for the public. This is a blatant lie, but adding a dash of fear-mongering ("but the Chinese have a faster one than we do!"),
  should be enough to shut up any naysayers.
  OK, so now you know how to get a grant for $$$ to go buy a huge supercomputer. Why don't you rush out and do this? $200M to spend on hardware for a few years of non-stop letter writing is a pretty good deal if you can pull it off, right?
  The answer is simple: you can't do it because you don't already have one. Why on earth should the government give you $200M to install a big Cray when they could get people like NCSA who have installed supercomputer after supercomputer for years to do it instead?
  Now you can see the answer to your question. The kiss of death, the single worst thing that can happen to a supercomputer centre is for its machines to become obsolete, with no hope of future funding. When this happens (and it does, from time to time), there are layoffs, transfers, no more junkets, free business class travel anywhere, etc. etc. Without an existing supercomputer, an institute becomes no better placed than you to request a huge cash injection from the government to install and run a supercomputer in the public interest. Odds of that happening? About zero.
  The question of whether the supercomputer is busy (and even if it is busy, whether it is doing anything useful) is totally and utterly irrelevant to the people who install and run supercomputers. All that matters is that the facility is perceived to be helping society, and that those taxpayer dollars keep coming in...
12. Re:When can I get one on my desktop? by rubycodez · 2012-01-31 04:06 · Score: 1
  
  That would depend type of problem? If I had a lproblem suitable for a vector supercomputer, maybe I would only need one or four processors. An NEC SX-6 processor had 8 GFlop in 2001, By 2005, the SX-8 had 16 vector GFLOPS per CPU. Then in 2008 the SX-9 was up to 102 GFLOP, about where our core i7 desktops are. so maybe there is four or five year lag
13. Re:When can I get one on my desktop? by WhiteSpade · 2012-01-31 04:14 · Score: 3, Informative
  
  UIUC runs their own power plant, and I used to live in an apartment on campus not too far from it. That thing put off so much steam that every morning fog was rolling across the street in front of where I lived. If I remember correctly, they also use the steam to heat a lot of the buildings on campus as well, via steam tunnels under the streets. They leak a lot, so there were always a few places you could stop on the sidewalk to warm yourself up before walking the rest of the way to class. Most of this was on the older side of campus. I'm sure most UIUC Slashdot readers spent more of their time north of Green street in the engineering quad where everything is a lot newer ;-).
  ---Alex
14. Re:When can I get one on my desktop? by Born2bwire · 2012-01-31 05:51 · Score: 1
  
  Well, they can finally get to the south side of campus, the new petascale building is by Lot E14.
15. Re:When can I get one on my desktop? by Helios1182 · 2012-01-31 10:41 · Score: 1
  
  I'm sorry, but to claim that getting the money for a project like this is both easy and simply a way to keep people living the high life shows an absolute lack of understanding of how research happens.
That number ... 2600 by ackthpt · 2012-01-30 12:36 · Score: 1

It rings a bell for two things: Atari and the hacker magazine.
I wonder if there's a connection somewhere ;)

--

A feeling of having made the same mistake before: Deja Foobar
1. Re:That number ... 2600 by Zakabog · 2012-01-30 12:46 · Score: 2
  
  Very likely there is a subconscious connection as it's really an Opteron 6200, the 2600 is a typo.
Um, me by jimhill · 2012-01-30 13:00 · Score: 4, Interesting

If the Cray architecture selected for Blue Waters is akin to that of Cielo then UIUC is going to rue -- RUE! -- the day they got in bed with these Cray con-men. The uptime and filesystem stability of Cielo is an absolute dog (as in, at least 2 FS rebuilds per week with data loss accompanying 2 in 5).

--
Learn to spell: nickel, missile, lose, solely, amendment, speech, kernel, probably, ridiculous, deity, hierarchy, versus
1. Re:Um, me by CajunArson · 2012-01-30 13:08 · Score: 1
  
  You raise an interesting point. The usual level of Slashdot "commentary" on Supercomputers usually isn't much above the level of jokes about Crysis and pissing matches between AMD ARM and Intel fanboys. Slashdot generally misses those little trivial details like... does it actually work doing something other than a meaningless Top500 benchmark.
  
  --
  AntiFA: An abbreviation for Anti First Amendment.
2. Re:Um, me by Anonymous Coward · 2012-01-30 13:09 · Score: 2, Informative
  
  Different file systems - Cielo is running Panasas (pfs) and Blue Waters will be running Lustre...
  Ironic CAPTA: painless
3. Re:Um, me by kwiqsilver · 2012-01-30 14:14 · Score: 1
  
  According to Google, Cray and Intel are working together on future supercomputers.
4. Re:Um, me by TheSunborn · 2012-01-30 14:43 · Score: 4, Insightful
  
  No they have not. Take a look at multicore spec test at http://electronicsnexus.com/articles/Opteron-Xeon-Benchmarks-2012-01.aspx where the 4x6282SE Opteron is the fastes 4 processor system testet. Or to quote
  "For example, note that the top-end 16-core 6282SE Opteron is a match for the top-end 10-core Xeon on floating point, and is not far behind it on integer either"
  Oh and the opteron cost less then half the price of the 10-core Xeon chip. So I think that slightly better floatingpoint performance, for less then half the price, make opteron the obvious choice, assuming you can split the workload so you can really use all the cores. Something I assume they master, since they are going to run their code on more then 1000 cores at a time.
5. Re:Um, me by rubycodez · 2012-01-30 14:46 · Score: 1
  
  some problems still need vector supercomputers, funny Cray has even slapped their label on NEC SX vector supercomputers. Seymore must be doing 10,000 rpm.
6. Re:Um, me by rbmyers · 2012-01-30 14:48 · Score: 1
  
  No kidding - Seymour may be rolling in his grave over having his name attached to anything massively parallel. His entire design philosophy was to have just a few uber processors cranked up as fast as possible, although I wonder if by now he'd have changed his mind. Multiple processor servers were expensive when he passed away and the multiple core race we have going on now wasn't even fantasy.
  The number of processors isn't the issue. The degree of connectivity is the issue, and IBM, Cray, and Seymour would all get it, even if the current "Cray" and UIUC aren't going to admit it. This version of Blue Waters is just another in a long line of massively parallel jokes. The version of Blue Waters proposed and abandoned by IBM would have been worth talking about.
  Flops are nearly free. Connectivity is expensive. That's why flops, irrelevant though they may be, are advertised.
7. Re:Um, me by Junta · 2012-01-30 15:22 · Score: 1
  
  In all truthfulness, the 10-core Xeon's (Westmere architecture still) aren't Intel's shining star of FP performance. Intel's strength is in their 8-core Xeons (Sandy Bridge) that are only recently coming into the market (not lagging Interlagos much at all). HPC has rarely been about the expensive high-end Xeons (massively expensive and generally 'last-gen' compared to the middle-tier Xeons with the main historical benefit of getting you to 4 sockets in one 'system', which is largely a moot point in HPC which generally is fine if split into whatever socket count you want, and *mostly* optimizes for lowest cost per socket, though IO per socket and larger failure domains can play a significant role against high socket systems as well in these environments.
  
  --
  XML is like violence. If it doesn't solve the problem, use more.
8. Re:Um, me by gmhowell · 2012-01-30 21:02 · Score: 3, Funny
  
  You raise an interesting point. The usual level of Slashdot "commentary" on Supercomputers usually isn't much above the level of jokes about Crysis and pissing matches between AMD ARM and Intel fanboys.
  Back in my day, the Slashdot "commentary" on Supercomputers was about Beowulf clusters and Natalie Portman and hot grits.
  Damned kids running around on my lawn again...
  
  --
  Jesus was all right but his disciples were thick and ordinary. -John Lennon
9. Re:Um, me by afidel · 2012-01-31 05:10 · Score: 1
  
  You STILL can't buy an E5 Xeon from anyone unless you were one of the few shops to order a datacenter full of them and are working with your OEM on the errata fixes. To say that the E5 Xeon isn't trailing Interlagos by much is a huge stretch since they're still essentially vaporware.
  
  On an unrelated note WTF are they using 4GB DIMM's? 8GB DIMM's have been the sweet spot for servers for the last ~18 months. The only thing I can think of is that they don't have the internode bandwidth to effectively use a global memory space that is twice as large so even the fairly minimal system cost increase to go to 8GB DIMM's isn't justified.
  
  --
  There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
10. Re:Um, me by jimhill · 2012-01-31 15:50 · Score: 1
  
  We were notified last week that Those Who Run The Machine are throwing in the towel on Panasas and are securing a Lustre-based farm for Cielo.
  
  --
  Learn to spell: nickel, missile, lose, solely, amendment, speech, kernel, probably, ridiculous, deity, hierarchy, versus
Depends... by Junta · 2012-01-30 13:01 · Score: 4, Funny

How big is your desk?

--
XML is like violence. If it doesn't solve the problem, use more.
Typo by reking2 · 2012-01-30 13:08 · Score: 5, Informative

Please correct "Opteron 2600" to "Opteron 6200". There are no 2600 series chips from AMD.
Re:The Real Question by kwiqsilver · 2012-01-30 14:09 · Score: 1

Most Crayons despise Windows and would never insult such a beautiful machine by subjecting it to that monstrosity. But there are a few lower end models available with Windows.
Cray? by SwampChicken · 2012-01-30 14:35 · Score: 1

Woah. Now thats a name I haven't heard of for a while.... I'm glad to hear that they're still in the game!
Nope... by Junta · 2012-01-30 14:48 · Score: 1

You can get an 64-bit Rpeak of about one teraflop out of about 4 of nVidia's top-end (C/M2070) GPGPU cards and 4 beefy Intel processors.
You quoted the 32-bit Rpeak, which is not particularly relevant to the discussion. GTX 580 64 bit Rpeak is about 168 Gigaflops.

--
XML is like violence. If it doesn't solve the problem, use more.
Re:AMD.. by Junta · 2012-01-30 14:58 · Score: 1

While I agree that Opteron is a suboptimal processor for this nowadays (now lagging Intel equivalent flops and memory), the feat of efficiently putting that scale of processors to productive work is still non-trivial. That's pretty much why Cray has been stuck with Opteron so long, they pinned all their efforts on hypertransport based technology while most competitors pinned it to more processor agnostic infiniband via pci express. They have reaped some benefits (a theoretically better IO architecture initially, now it's dubious; and it just sounds more impressive in some ways), but now they are firmly on the wrong side of the fence. It will be interesting to see what happens next, if they do a QPI effort or start hedging their bets on Infiniband like everyone else. That has historically been for most people marrying yourself to Mellanox instead of the processor vendor, but maybe Intel will inject some vitality into QLogic's lackluster IB implementation (QLogic though probably thought the same thing as they picked up the IB pieces of PathScale and Silverstorm, but maybe third time's a charm?)
Of course, if they are falling down on the job on the software side (filesystem wise) like a few people in this thread have suggested, that's far more dire than Intel v. AMD.

--
XML is like violence. If it doesn't solve the problem, use more.
Obligatory by Whiteox · 2012-01-30 17:16 · Score: 1

Yes, but does it run .... oh forget it

--
Don't be apathetic. Procrastinate!
Ahem, NVIDIA? by Mike_K · 2012-01-30 17:24 · Score: 3, Informative

It is very nice that AMD Opterons are mentioned and petaflops are celebrated, but aren't those petaflops mostly delivered by NVIDIA's Kepler Tesa cards?
From the TFA:
Cray XK6 blades with NVIDIA(R) Tesla(TM) GPUs, based on NVIDIA
(NASDAQ: NVDA) next-generation 'Kepler' architecture, which is
expected to more than double the performance of the Fermi GPU on
double-precision arithmetic.
1. Re:Ahem, NVIDIA? by gentryx · 2012-01-30 21:42 · Score: 1
  
  Actually most of the cabinets will be XE6, not XK6. Most codes at U of I aren't GPU ready.
  
  --
  Computer simulation made easy -- LibGeoDecomp
Hardware Moves Ahead, Software...not so much... by afabbro · 2012-01-30 17:30 · Score: 1

Hardware compared to, say, 1970? Mammoth progress. Room-sized state of the art then is dwarfed by a low-end laptopnow.
Software compared to, say, 1970? We've moved a little, but really it isn't all that much different. Things are more GUI, some fads have come and gone, but as Robert Martin puts it, it's still just sequence, selection, and iteration.

--
Advice: on VPS providers
1. Re:Hardware Moves Ahead, Software...not so much... by LeDopore · 2012-01-31 02:02 · Score: 3, Informative
  
  Dear afabbro,
  You are largely correct. Most software has not sped up much since the 1970s, and it could even be argued that developers write such sloppy code these days that even our improved compilers can't compensate, especially in applications where performance is no longer critical.
  On the other hand, since about 2006 there have been some tremendous advances in algorithms. One optimization problem I work on, Basis Pursuit Denoising http://en.wikipedia.org/wiki/Basis_pursuit_denoising, has had on the order of a 10-fold increase in real-world speed on constant hardware every year for the past 5 years (see http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5940245 for my contribution).
  These advances are not just academic games; they are actually worth doing. They could eventually lead to computers with sensory processing routines that have a mote of common sense to them, able to perform some real-world tasks we currently need humans for.
  While I agree that by and large, most software is getting fat and lazy, there are a few problems where today's algorithms on 2002 hardware mop the floor with 2002 algorithms on today's hardware.
  Best,
  LeDopore
  
  --
  Expected time to finish is 1 hour and 60 minutes.
2. Re:Hardware Moves Ahead, Software...not so much... by LeadSongDog · 2012-01-31 04:04 · Score: 1
  
  One optimization problem I work on, Basis Pursuit Denoising , has had on the order of a 10-fold increase in real-world speed on constant hardware every year for the past 5 years
  
  Great, so how about making OCR on noisy scans work next? The archive.org desperately needs something that works....
  
  --
  Oh, I'm sorry sir, I thought you were referring to me, Mr. Wensleydale.
3. Re:Hardware Moves Ahead, Software...not so much... by LeDopore · 2012-01-31 04:24 · Score: 1
  
  There's been some really promising work in the direction of OCR-like problems lately. Here's an algorithm that can efficiently learn a small dictionary of symbols (like letters) and decompose a signal into elements that fit within this "low-rank" dictionary plus sparse noise (bugs squashed on the text?) plus Gaussian noise: https://sites.google.com/site/godecomposition/.
  It's not literally magical, but it's super-duper awesome (an no, I'm not an author of this one) and it should contribute to the minor revolution in signal processing (compressed sensing & low-rank matrix completion) that's been gaining momentum since about 2005. If our machines can learn features efficiently and robustly from natural images, many industries are in for a wild shake-up. More on this minor revolution is available at http://nuit-blanche.blogspot.com/.
  These algorithms are part of the reason why self-driving cars are starting to work, and I have the excited feeling like we're on the cusp (read, next ten years or so) of a sea change in our ability to have machines able to understand and interact with the physical world with a dash of common sense.
  
  --
  Expected time to finish is 1 hour and 60 minutes.
Re:who wouldn't want access? by fritsd · 2012-01-31 00:16 · Score: 1

Depends on how many ganglia those prawns have, AND on the verisimilitude(sp?) of their simulation. AND on how many prawns we are speaking about, of course..

--
To be, or not to be: isn't that quite logical, Slashdot Beta?
Re:AMD.. by Wesley+Felter · 2012-01-31 02:54 · Score: 1

The next generation Cray (XE7?) will attach to the processor via PCIe, so they can use Intel or AMD. They're definitely not going to use IB when their Gemini interconnect is better.
Re:there is no such thing as "1 petaflop" by rubycodez · 2012-01-31 04:09 · Score: 1

there could be petaflop per second. my desktop can do a petaflop per 30,000 seconds.