Slashdot Mirror

← Back to Stories (view on slashdot.org)

DOE Asks For 30-Petaflop Supercomputer

Posted by Soulskill on from the go-big-or-go-home dept.

Nerval's Lobster writes "The U.S. Department of Science has presented a difficult challenge to vendors: deliver a supercomputer with roughly 10 to 30 petaflops of performance, yet filled with energy-efficient multi-core architecture. The draft copy (.DOC) of the DOE's requirements provide for two systems: 'Trinity,' which will offer computing resources to the Los Alamos National Laboratory (LANL), Sandia National Laboratories (SNL), and Lawrence Livermore National Laboratory (LLNL), during the 2016-2020 timeframe; and NERSC-8, the replacement for the current NERSC-6 'Hopper' supercomputer first deployed in 2010 for the DOE facilities. Hopper debuted at number five in the list of Top500 supercomputers, and can crunch numbers at the petaflop level. The DOE wants a machine with performance at between 10 to 30 times Hopper's capabilities, with the ability to support one compute job that could take up over half of the available compute resources at any one time."

44 of 66 comments (clear)

  1. So . . . by Anonymous Coward · · Score: 1

    What are they going to use this machine for? Hopefully it's not Skyrim.

    1. Re:So . . . by godrik · · Score: 4, Informative

      This machines are most likely goign to be the replacement of the ones we already have. NERSC is presenting the projects that are run on its computing infrastructure on its web site [1]. You can see on the first page the project that are currently running jobs and what they are doing. For instance this project [2] is about designing artifical photosynthetic cells. If you are interested just check the project they are funding.

      [1] https://www.nersc.gov/
      [2] https://www.nersc.gov/science/energy-science/artificial-photosynthesis-i-design-principles-for-light-harvesting/

    2. Re:So . . . by Raul654 · · Score: 5, Informative

      Back when I worked for Supercomputing group at Los Alamos, the supercomputers were categorized into 'capacity' machines (the workhorses where they did most of the work, which typically run at near full utilization) and capability machines (the really big / cutting-edge / highly unstable machines that exist in order to push the edge of what is possible in software and hardware. One example of such an application would be high energy physics simulation) . It sounds like these machines fall into the latter category.

      --


      To make laws that man cannot, and will not obey, serves to bring all law into contempt.
      --E.C. Stanton
    3. Re:So . . . by davester666 · · Score: 1

      C&C for SkyNet!

      --
      Sleep your way to a whiter smile...date a dentist!
    4. Re:So . . . by religious+freak · · Score: 1

      I don't know. Personally whenever I see machines with specs like these I get the idea that the only practical application would be advanced AI.

      Yes, I know the NNSA and others use this type of hardware to simulate physical environments and nuclear events but I just can't help but think there's a pretty good possibility our government is racing toward advanced AI systems. These computer folks are some of the best in the world and they know as well as anyone what an advancement in weapons tech an AI would be. At least that's what I'd be telling my superiors if I were working in these high computing departments.

      --
      If you can read this... 01110101 01110010 00100000 01100001 00100000 01100111 01100101 01100101 01101011
    5. Re:So . . . by IAmR007 · · Score: 1

      Even if the names don't get changed, they still get upgraded a lot. The power costs are so significant (several million dollars a year) that running a system that's more than a couple years old is completely unfeasible. For example, I have an account on HECToR, which has gone through three/four upgrades since it was first built in 2008: 11,328 2.8GHz cores to 22,646 2.3GHz cores to 44544 cores to 90,112 2.3GHz cores (and ram upgrades along the way for a total of 90TB now). One of those was a two-part upgrade.

    6. Re:So . . . by Orp · · Score: 2

      I am an early user on the Blue Waters petaflop machine (http://www.ncsa.illinois.edu/BlueWaters/). Mean time to failure for such a huge machine becomes a real issue when you have about 700,000 cores and who knows how many spinning hard drives and all that network infrastructure. However I and my research collaborators have managed to get jobs through that take on the order of 12 hours of wallclock time without a hardware fault, which is amazing IMO. I do wonder whether we can simply continue to expand the same basic computing infrastructure to a 10-20 PFLOP machine. There will have to be redundancy built into the hardware of any such machine such that if a compute node goes offline it seamlessly offloads it to another. Writing fault tolerant massively parallel code is possible but very challenging and most scientists won't or can't do it.

      --
      A squid eating dough in a polyethylene bag is fast and bulbous, got me?
    7. Re:So . . . by alexandre_ganso · · Score: 1

      There are a lot of different research that benefits from these kinds of machines. Mind you, the machine will hardly be running a single program at 30 pflops scale, but instead running dozens of smaller jobs at the same time, and economy comes with the scale. It's simpler to scale your job from 10,000 processors to 1 million on the same machine than running the smaller job in one site than porting to the big one. Besides, give 30 pflops and people on the physics, math and biology department will ask for 50 :)

  2. They just discovered by Janek+Kozicki · · Score: 4, Funny

    bitcoins? :)

    --
    #
    #\ @ ? Colonize Mars
    #
  3. Department of Science? by Shag · · Score: 4, Informative

    Oh, if only science were elevated to Department status, with a cabinet-level secretary!

    I think you mean Department of Energy, Office of Science.

    --
    Village idiot in some extremely smart villages.
    1. Re:Department of Science? by ILongForDarkness · · Score: 1

      Because clearly energy which is a part of science is more important than all science :)

    2. Re:Department of Science? by SpaceCracker · · Score: 1

      Congressmen have already decided on creating the Department of Science. They're just looking for intelligent designers to do the job...

      --
      sigo ergo sum
  4. How about a by saphena · · Score: 1

    Large cluster of Raspberry Pis

    1. Re:How about a by VorpalRodent · · Score: 4, Informative

      This is Slashdot - I believe the meme you're looking for is "Beowulf cluster", and such a cluster of these things would probably even meet the recommended specs for Crysis.

      --
      Take it to the limit, everybody to the limit, come on, everybody fhqwhgads.
    2. Re:How about a by mug+funky · · Score: 1

      but does it run linux?

    3. Re:How about a by alexandre_ganso · · Score: 1

      Mostly, because of the network. Although the cpus (or the whole system, in the case of the Pi) are cheap, the inter-communication is SLOW. And this gets worse with scale. So what starts bad (with the PI) network, gets much worse in bigger scale.

      Although this is an excellent test bed for teaching parallel computing - EXACTLY because it scales so badly, so the bad effects are exaggerated.

  5. Re:GPU for peta flops by GiganticLyingMouth · · Score: 1
    According to NVIDIA, the peak single precision floating point throughput is 3.95 TFLOPS for a Tesla K20X. If, for simplicity, we round up to 4TFLOPS and assume that it operates at peak efficiency (which is unlikely), you'd need 250 of these. Of course that also assumes linear scaling with the number of processing units (which is also unlikely). This would be for 1 PFLOPS mind you; the DOE wants 10 ~ 30.

    So how many GPU for a Peta Flop ?

    Not to be overly pedantic, but PFLOPS --> 10^15 FLOP/sec, so saying "Peta Flop" doesn't make any sense - that would just be 10^15 floating point operations, which without a time-scale (seconds) is meaningless.

  6. mmh... by BarfooTheSecond · · Score: 1

    ... I've heard rumors that Microsoft will participate in the tender and propose their "HPC" solution based on Windows 8 (well, it's for all computer platforms, they say)

    1. Re:mmh... by unixisc · · Score: 1

      If they are based on the x64, they can. But the best way to do it would be to build them from Itanium III, and then have either Debian or FreeBSD (preferably Dragonfly BSD) running on it.

    2. Re:mmh... by BarfooTheSecond · · Score: 1

      Forgot to mention I was kidding :-)

      Sure Windows 8 will never run on such big machines. It will never run on my PC too, btw :-)

      btw: is Windows kernel still limited to 64 cpus? (I'm not talking clusters and the like, but "single image")

    3. Re:mmh... by alexandre_ganso · · Score: 1

      Hum, why BSD? He mentioned linux not because is the best solution ever (which might or might not be), but because a lot of petaflop-capable code was written specifically to run on it.. and because the big names (IBM, Crazy) fully support it. In fact, I don't remember ever using a BSD-based supercomputer. The top500 only shows one machine, at 0.1 petaflop, running a bsd-based OS. Search for os here: http://top500.org/statistics/sublist/

  7. DOE already has 2 of them... by WhitePanther5000 · · Score: 2

    27 Petaflops at Oak Ridge
    20 Petaflops at Lawrence Livermore

    http://top500.org/lists/2012/11/

  8. Make that 3 of them by WhitePanther5000 · · Score: 1

    If their bottom line is 10 Petaflops, they have a 10 Petaflop one at Argonne too.

  9. Isn't New Mexico selling a supercomputer? by dr_leviathan · · Score: 1

    Maybe the DOE should bid on that supercomputer being liquidated by the US state of New Mexico.

    --
    Religion is poison to rationality, and we lose sight of that at our own peril. -- Lurker2288
    1. Re:Isn't New Mexico selling a supercomputer? by l0ungeb0y · · Score: 2

      Yeah -- but that is only spec'd at 172 TFlops, a long way away from 30 PFlops.

  10. Nota problem, buy one today by Henriok · · Score: 1

    What's the problem? They can buy one that fits their need today. There are a variety of designs that will deliver this kind of performance available today from the likes of Cray and IBM.

    --

    - Henrik

    - when the Shadows descend -
    1. Re:Nota problem, buy one today by dutchwhizzman · · Score: 1

      It's not the fact they already exist, but that they have to spend government money. Since they are spending that money, they have to get the taxpayers their moneys worth and have to put out a "tender" so suppliers can compete offering the best deal. In order to prevent personal preferences of people in power, bribes and such, tenders are usually rather strict in their requirements and procedures. This is about a lot of tax money, so it gets a lot of attention. Your local community probably puts out tenders for a contract to fix the holes in the road each year, or for putting a new roof on the public school three streets down as well. Those don't make it to SlashDot, but they work the same way.

      --
      I was promised a flying car. Where is my flying car?
  11. this means the NSA already has one by decora · · Score: 1

    and we won't learn about it until James Bamford writes another book. . .

    1. Re:this means the NSA already has one by NothingMore · · Score: 1

      There are already 3 at national labs that are > 10 PFLOPS... ORNL's Titan, LLNL's BG/Q, and ANL's BG/Q. The story behind this is they want it cheaper than the crazy price tag those machines have.

    2. Re:this means the NSA already has one by AHuxley · · Score: 1

      At 200MW? 80-65-megawatt 'power' upgrade seem to the quoted in some press too. Take off some power for the protester zapping electric fence and see how many chips you can power and cool with that mw range :)
      http://www.datacenterdynamics.com/focus/archive/2012/03/light-shed-nsa’s-massive-supercomputer-project-spying

      --
      Domestic spying is now "Benign Information Gathering"
  12. Re:GPU for peta flops by GiganticLyingMouth · · Score: 1

    From wikipedia: FLOPS stands for "FLoating-point Operations Per Second, also flop/s". The /sec is part of the acronym. Hence "FLOPS/sec" would be floating point operations per second per second, which probably isn't what you meant. Likewise, saying "FLOP" denotes the execution of one floating pointer operation. There's no time metric, and is equally meaningless.

  13. Puts the core in politically core-rect. by Impy+the+Impiuos+Imp · · Score: 1

    Fastest on earth, "yet filled with energy-efficient multi-core architecture." :rolleyes

    These are at cross-purposes. Do they want fastest on Earth, or pretty fast, but efficient, which is already driven by market mechanisms?

    "Hey! Multi-core and multi-cultural both have 'multi' in it! Can we have multi-cultural architecture, too? How much extra is that?"

    --
    (-1: Post disagrees with my already-settled worldview) is not a valid mod option.
    1. Re:Puts the core in politically core-rect. by bazmonkey · · Score: 2

      Fastest on earth, "yet filled with energy-efficient multi-core architecture." :rolleyes

      These are at cross-purposes. Do they want fastest on Earth, or pretty fast, but efficient, which is already driven by market mechanisms?

      No, it's not. Today's supercomputers are thousands upon thousands times faster than those of decades past but are NOT taking up thousands of times more space or electricity.

      Hopper is 16,000 nodes and two Pflops. Cray can't just make 10 of them, put 'em together, and consider the order filled. Efficiency is a LOT of the challenge in making the world's fastest computers.

  14. Re: artificial photosynthetic cells by neonsignal · · Score: 3, Insightful

    Are you really claiming that a computer being run by Los Alamos and called Trinity is primarily going to be used for alternative energy?

  15. System Archetecture by Required+Snark · · Score: 2
    They have a comparatively large number of existing codes (around 600) that run with no GPU acceleration. They want to continue this code base and not have to modify it very much, so they are not going to use any non CPU integrated coprocessors. According to the article:

    They could have built such a machine, but it would have required either discrete accelerators (a programming model they would rather skip) or something more proprietary like the Blue Gene platform (an architecture they have avoided). The hope is that by 2015, they will be able to get something on the exascale roadmap, but with a programming model that is reasonably friendly to CPU-based codes.

    That most likely means integrated heterogeneous processors like NVIDIA's "Project Denver" ARM-GPUs, AMD's x86-GPU APUs, or whatever Intel brings to the table with integrated Xeon Phi coprocessing. Although more complex than a pure CPU solution from a software point of view, the integrated designs at least avoid the messy PCIe communication and the completely separate memory space of the accelerator device.

    Note that one of the possible contenders is ARM with an integrated GPU. Slashdot readers are generally hostile to the idea of ARM for servers or HPC, but it is going to happen. Making the Top 100 list in the future will require more and more attention to FLOP/Watt, and ARM has a basic advantage over legacy oriented x86 architectures. Being dismissive of ARM is just as much of a fanboy attitude as being rabidly for any other architecture.

    --
    Why is Snark Required?
    1. Re:System Archetecture by alexandre_ganso · · Score: 1

      Don't forget that x86 comprises five of the top 10, being the rest Powerpc-based (BG/Q and Power7). Other contenders have much more chance on this market than, say, the workstation market.

  16. Titan by cmdr_tofu · · Score: 1

    Am I wrong thinking that this is not dramatically faster than Titan (27 TF peak)
    http://www.top500.org/system/177975

    The specifications in the doc are interesting nonetheless!

  17. Re: artificial photosynthetic cells by AHuxley · · Score: 3, Funny

    Little Red Blogger from the Hood asks:
    "What a deep budget you have," ("The better to educate you with"),
    "Goodness, what big networks you have," ("The better to save you taxes by networking with)
    "And what big transformers you have!" ("The faster to compute for you with"),
    "What a big results you have," ("The better to nuke you with!")

    --
    Domestic spying is now "Benign Information Gathering"
  18. How do you keep hardware from killing your work? by PeterM+from+Berkeley · · Score: 1

    A question I wonder about---I guess "10-30 petaflop" of a standard multi-core architecture would require > 1M computational cores. Suppose you're running a code on 500,000 cores.

    What is the mean time to failure of a core or some other piece of hardware required for that core to work? With 500k cores, I'd expect one to die every few hours or even minutes. Either that, or a random bit-flip from a cosmic ray.

    Given that, how do you finish a computation that takes more than an hour or so? And how do you guarantee the integrity of that computation?

    D'you have three cores working every piece of the problem, and have them "vote" on the result, taking the majority opinion in the INEVITABLE case of disagreement? That sort of implies that to have 10 petaflop of effective computing power, you need 30 petaflop of actual computing power, + all the overhead/hardware required for voting.

    Integrity of the results for such large computations just seems like a very difficult problem!

    --PM

  19. I hope DoE will -model- LFTRs, to speed approvals by ivi · · Score: 1

    Liquid Fluoride THORIUM Reactors (LFTRs) could get a leg up
    for -earlier- construction approvals, ie, if DoE puts some super-
    computers to the task of modeling them mathematically, eg, to
    help bring them on-line sooner.

    Or... we can let India and/or China do all that... and buy the com-
    pleted technology from them, after they've done that.

  20. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion

  21. Double standards by Cinnaman · · Score: 1

    It's pretty cynical that western governments want to tax harmless carbon dioxide (eg. in Australia) and limit our energy consumption through constantly jacking up the rates yet build extremely power-hungry installations in order to crunch all the data needed to surveil the citizens and build a profile of them.

  22. How much of the universe can it simulate? by elucido · · Score: 1

    If it's less than that of a human hair, they will need more processing power.

  23. Re:I hope DoE will -model- LFTRs, to speed approva by alexandre_ganso · · Score: 1

    Another good thing is that by having these more "friendly" reactors, you can power more supercomputers! It's a win-win situation