Slashdot Mirror


Indiana University Dedicates Biggest College-Owned Supercomputer

Indiana University has replaced their supercomputer, Big Red, with a new system predictably named Big Red II. At the dedication HPC scientist Paul Messina said: "It's important that this is a university-owned resource. ... Here you have the opportunity to have your own faculty, staff and students get access with very little difficulty to this wonderful resource." From the article: "Big Red II is a Cray-built machine, which uses both GPU-enabled and standard CPU compute nodes to deliver a petaflop -- or 1 quadrillion floating-point operations per second -- of max performance. Each of the 344 CPU nodes uses two 16-core AMD Abu Dhabi processors, while the 676 GPU nodes use one 16-core AMD Interlagos and one NVIDIA Kepler K20."

83 comments

  1. Biggest? Really? by wonkey_monkey · · Score: 3, Interesting

    Computers used to be a lot bigger.

    --
    systemd is Roko's Basilisk.
    1. Re:Biggest? Really? by Anonymous Coward · · Score: 2, Interesting

      How can you tell? In TFA, you just have a photo of one corner of the building.

    2. Re:Biggest? Really? by reboot246 · · Score: 2

      That's a building? It looked more like a stack of 4x8 Styrofoam sheets. I guess that's what passes for architectural design nowadays.

    3. Re:Biggest? Really? by Chrisq · · Score: 1

      Computers used to be a lot bigger.

      But the largest computer ever built (physically) executed only 75,000 instructions per second, and had 70k of memory (though that was 32 bit words")!

    4. Re:Biggest? Really? by Anonymous Coward · · Score: 1

      Yea, hard to tell scale, but you can see a security camera and the outside lighting, so it seems to be one-story....

    5. Re:Biggest? Really? by cdrudge · · Score: 2

      IU is a public university. Would you prefer tax dollars get spent on a masterpiece of architectural design for a data center?

    6. Re:Biggest? Really? by gtall · · Score: 1

      Depends upon what you mean by public. Most of their money these days come not from the state but from tuition, grants, etc. Many "public" unis are in the same position because lawmakers have increasingly found education to be not worth their while.

    7. Re:Biggest? Really? by Anonymous Coward · · Score: 0

      It probably is.

      Those "masterpiece" buildings that some universities have that look pretty usually are nightmares to actually use.

    8. Re:Biggest? Really? by Z_A_Commando · · Score: 3, Insightful

      It's a Tier 3 data-center built to withstand F5 tornadoes and earthquakes. All the pretty glass stuff doesn't really survive in 300MPH winds. Also, the main receiving area in the back looks like something out of Jurassic Park. And in Bloomington, they think limestone is very pretty.

  2. Re:Dedication ceremony? by Cenan · · Score: 3, Insightful

    Or it could be something completely innocent like cutting of tape and speeches and shit. You know the kind of stuff you do when you show your stakeholders what their money went to. Stop your idiotic religious babbling.

    --
    ... whatever ...
  3. Imagine a Beowulf.. by blackicye · · Score: 2

    Cray is still kicking around??

    1. Re:Imagine a Beowulf.. by fuzzyfuzzyfungus · · Score: 4, Informative

      Cray is still kicking around??

      I don't think they've kicked out an original processor design in ages; but they are still(among) those you talk to if you want something a little more tightly coupled, and/or a bit more 'turnkey' than "10,000 of whatever dell is selling, and some 10GbE switches".

    2. Re:Imagine a Beowulf.. by RicktheBrick · · Score: 3, Informative

      Maybe it is because Seymour Cray died in 1996 of complications from a automobile accident. http://en.wikipedia.org/wiki/Seymour_Cray I would think that most people who have a passing interest in supercomputer would have known that fact. Seymour Cray did a lot of work in establishing the supercomputers. I can remember seeing his supercomputers on the cover of Popular Science back in the 70's. I think he deserves a little more respect than shown here by that remark.

    3. Re:Imagine a Beowulf.. by Anonymous Coward · · Score: 1

      Google "cray blackwidow". Last custom cray-designed vector processor

    4. Re:Imagine a Beowulf.. by Anonymous Coward · · Score: 0

      Cray is still kicking around??

      The name is still alive

    5. Re:Imagine a Beowulf.. by OhSoLaMeow · · Score: 1

      Still too soon.

      --
      They can take my LifeAlert pendant when they pry it from my cold dead fingers.
    6. Re:Imagine a Beowulf.. by blackicye · · Score: 1

      Maybe it is because Seymour Cray died in 1996 of complications from a automobile accident. http://en.wikipedia.org/wiki/Seymour_Cray I would think that most people who have a passing interest in supercomputer would have known that fact. Seymour Cray did a lot of work in establishing the supercomputers. I can remember seeing his supercomputers on the cover of Popular Science back in the 70's. I think he deserves a little more respect than shown here by that remark.

      I meant no disrespect to Seymour Cray, I just didn't think the company was still all that relevant any more.

  4. AMD ?? by Anonymous Coward · · Score: 0, Funny

    You want PERF you must go INTEL !!

    Intel RUELZ !!

    1. Re:AMD ?? by Anonymous Coward · · Score: 0

      Warz teh intel with more than four COREZ? Itz multi-processing Biznitches!

  5. Won't be long now... by Anonymous Coward · · Score: 1

    until some wannabe comedian makes a "Does it run Linux?" post. Despite the fact that it's one of the earliest /. memes and has been used over a million times, it will get moderated "+5 Funny" because originality and creativity are lost on this crowd.

    1. Re:Won't be long now... by JustOK · · Score: 4, Insightful

      you must be new here.

      --
      rewriting history since 2109
    2. Re:Won't be long now... by Anonymous Coward · · Score: 0

      Well that is a valid question. So, does it run Linux?

    3. Re:Won't be long now... by Entropius · · Score: 2

      Yes. What else would it run?

    4. Re:Won't be long now... by Anonymous Coward · · Score: 0

      100 meters in 9 seconds flat?

    5. Re:Won't be long now... by Anonymous Coward · · Score: 1

      Do you like apples? TFA says it runs Cray's version of SUSE.

    6. Re:Won't be long now... by Anonymous Coward · · Score: 0

      I was kind of expecting OpenIndiana, to be honest.

    7. Re:Won't be long now... by Entropius · · Score: 1

      It tried to run, but as soon as they fired the starting pistol it just fell over. Someone called it a "flop".

  6. Biggest and probably dull too look at... by mendax · · Score: 2

    It's great to see a university have a monster like this for research use. And old universities you would think are well suited for these kinds of monsters. Their computer centers were built at a time when the computers really were filled with monster machines that your iPad would run circles around today performance-wise. They were replaced in the 1990s by servers that would fit into a closet. But they still have all this space that can be filled with racks upon racks of supercomputer nodes. However, I suspect that IU may have built a new building for these new and improved monsters. But anyway, these new monsters are nice to contemplate but they're not a pretty to look at. Computers in the old days were designed to be both functional and attractive to members of the unwashed masses who could gaze at them through the glass windows and drool, and be hypnotized at the blinking lights and the spinning tape drive reels spinning. And the glass windows were there to allow the institution to show the machine off as a kind of status symbol. There was no picture in the article of this new beast but I will bet $0.02 that it's pretty dull looking.

    --
    It's really quite a simple choice: Life, Death, or Los Angeles.
    1. Re:Biggest and probably dull too look at... by mendax · · Score: 1

      Ack.... it's 3:30 in the morning and I can't spell. It's "probably dull TO look at". That ought to be modded down just for that little slip.

      --
      It's really quite a simple choice: Life, Death, or Los Angeles.
    2. Re:Biggest and probably dull too look at... by fuzzyfuzzyfungus · · Score: 2

      Check this one out...

      (More broadly, though, the point is largely valid. Reel-to-reel deserved to die, technologically; but damn did it look 'high tech' churning away in the background, now that everything fits in standard 72u racks, it's mostly just a 'Should we go for 'basic black, or spring for custom powdercoat and a cool cutout design for the doors?' game.

      Of course, seeing as the CM-2 won 'coolest-looking computer of all time', with the CM-5 playing 'solid; but ever-so-slightly-disappointing-sequel', perhaps it's only fair for everyone else to just stop trying.

    3. Re:Biggest and probably dull too look at... by mwvdlee · · Score: 1

      http://newsinfo.iu.edu/pub/libs/images/usr/15356_h.jpg
      Not too shabby looking for a line of racks. Please donate your $0.02 to a charity of your own choice.
      Though from the looks of it, I expect it to be mostly calculating ballistic trajectories originating in eastern Europe.

      --
      Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
    4. Re:Biggest and probably dull too look at... by dbIII · · Score: 2

      Not ten feet away from me is a box of nine track reels - it's still not dead yet. It should be, and it would be if people had transcribed their media within a sane timeframe, but it's still in use on occasion. I don't know if those reels in that box will ever be read again since they were a third copy in 1982, but I had to get a few transcribed up to a couple of years ago after the original owners threw out their copies and then found out a decade later that they wanted the data.

    5. Re:Biggest and probably dull too look at... by mendax · · Score: 2

      Looks rather boring to me. But I'm old school... and after just made my first visit to the Computer History Museum in Mountain View in five years and seen bits of the beauty of old room-filling computers from the last sixty years on display, I can say with some certainty that the IU machine is dull to look at.

      --
      It's really quite a simple choice: Life, Death, or Los Angeles.
    6. Re:Biggest and probably dull too look at... by Charliemopps · · Score: 1

      About 50 feet from me there's about 100 boxes of microfiche that represent billing records from the 1960s on back. I have no idea why we still keep them, but they're there sure enough.

    7. Re:Biggest and probably dull too look at... by fahrbot-bot · · Score: 1

      Not ten feet away from me is a box of nine track reels...

      About 50 feet from me there's about 100 boxes of microfiche...

      About a mile away from me is something called a Library, filled with things called "books" and "magazines" - I hear they're like papery blogs. I have no idea why we still keep them, but they're there sure enough.

      --
      It must have been something you assimilated. . . .
  7. Research Hell by kurt555gs · · Score: 4, Funny

    Can you imagine how many Bitcoins this thing could mine per hour?

    --
    * Carthago Delenda Est *
    1. Re:Research Hell by Anonymous Coward · · Score: 0

      How do you think they plan on paying for it?

  8. and with all that computing power.. by Anonymous Coward · · Score: 0

    they're still unable to figure out why the big ten sucks at bowl games.

    1. Re:and with all that computing power.. by anjrober · · Score: 1

      IU has never been a football power house
      its all basketball
      and all in, they had a very good year. would loved to see them do better in the tournament but a great regular season.

  9. Re:Dedication ceremony? by Anonymous Coward · · Score: 5, Funny

    I realize that you don't "believe", but SkyNet will be real one day and we never know which super computer will be the first node in humanity's Beowulf Cluster of Death.

    Maybe, just maybe, if we're nice to them and show them some respect they'll let us service their modules until we die of natural causes.

  10. Not many with only 676 GPU nodes by Anonymous Coward · · Score: 1

    The bitcoin pools are mostly PCs with GPU rigs, and they often number into the tens of thousands, so basically this supercomputer is puny compared to even the GPU bitcoin pools.

    1. Re:Not many with only 676 GPU nodes by flowerp · · Score: 2

      Bitcoin is now dominated by FPGA and ASIC miners (dedicated hardware), most GPU farms have moved on to litecoin.

      --
      --- Eat my sig.
    2. Re:Not many with only 676 GPU nodes by Anonymous Coward · · Score: 1

      RTFS... in what world are 676 Kepler K20s called puny?

    3. Re:Not many with only 676 GPU nodes by gman003 · · Score: 1

      The Bitcoin world.

      Kepler is good for stuff involving lots of double-precision floating-point, like scientific computing. Physics, chemistry, stuff like that.

      AMD has a lead on Bitcoin mining because a) it's integer, not float32 or float64, b) AMD has a shitload of slower cores rather than a smaller number of more efficient, powerful cores, and c) due to some weird coincidences of architecture, AMD designs (both VLIW5 and GCN, IIRC) can run the "main loop" of Bitcoin mining in only one instruction, while Nvidia's Fermi and Kepler designs require three instructions.

      The most badass Bitcoin miner would consist of Radeon 7990s. Fortunately, Indiana University seems to be planning to use this for SCIENCE and not Bitcoins. Science is probably a better investment, in the long run.

  11. Re:Dedication ceremony? by Anonymous Coward · · Score: 1

    I dunno, Indiana does have those awful "In God We Trust" license plates.

  12. So they'll be mining a shitload of Bitcoins. by davesag · · Score: 1

    They'll just use it to mine Bitcoins I'm sure.

    --
    I used to have a better sig than this, but I got tired of it
  13. Quality name? by Anonymous Coward · · Score: 0

    Now they have Big Red Two and Big Red too. Twice the fun!

    At least we (Purdue University) name our supercomputers something interesting.

    1. Re:Quality name? by slugstone · · Score: 0

      like what? Carter, that interesting?

    2. Re:Quality name? by jtillots · · Score: 1

      I wanted to name it Trident. It's 1.) IU related (our symbol is called the trident), 2.) it's powerful since it's a weapon, and 3.) it's gum-themed (expanding on BigRed as a chewing gum). Who's with me!

  14. some big ten schools are not sports colleges by Joe_Dragon · · Score: 2

    some big ten schools are not sports colleges.

    No at some you have real classes to take and pass.

    1. Re:some big ten schools are not sports colleges by Anonymous Coward · · Score: 0

      I went to the University of Minnesota and I can tell you that we're neither.

  15. Another Bitcoin Joke by puddingebola · · Score: 1

    It's already found 2 verified bitcoins and paid for the first month's electric bill.

  16. Re:Dedication ceremony? by Anonymous Coward · · Score: 1

    That's just an alternative to the Free Market sky fairy used elsewhere in the US.

  17. First task: by Anonymous Coward · · Score: 0

    Come up with something better than "Big Red III" for the next machine.

    1. Re:First task: by tqk · · Score: 1

      "Hellboy!" (apologies to Ron Perlman & Co.).

      --
      "Tongue tied and twisted, just an Earth bound misfit ..." -- Pink Floyd.
  18. I have to question the wisdom of this by Virtucon · · Score: 2

    While it has been in vogue for years for universities to have this capability in-house, I have to question the wisdom of this kind of investment in a few areas. First, recently there was an article on Slashdot posted about the Federal Government retiring Roadrunner because in less than 5 years because it was too much of a power hog. I haven't seen anything in the press releases about Big Red that would indicate that IU has solved the power obsolescence issue; in five years, we'll probably see Big Red II retired because it wasn't power efficient given newer technology. IMO in five years, IU will be looking to fund Big Red III so I hope they get their value out of this investment, total operating costs (TOC) because it has to be very, very expensive to keep the lights on for this thing. Second, with Utility Computing models available in the Cloud with AWS, Google Apps etc. for large scale experiments, more and more companies are choosing the utility model to run their research rather than buying it. I don't need to cite them all here but there's stories day in and day out of companies and universities leveraging utility based, cloud models for HPC. You have one resource here at IU when you could lease multiple Cloud based resources with hundreds of thousands of nodes simultaneously, not just rely on one large machine in your data center. I can imagine there are quite a few experiments that IU can do with it, but when I read their press, it's available to IU students and Faculty, does that mean they won't let other academic institutions use it? If that's true it's a very expensive resource that only one institution can use and I doubt that they can keep it busy 24x7x365 for its useful life with experiments. Maybe I'm wrong but I just can't see this kind of large scale investment being feasible over the coming years because it will just be too inexpensive and disposable to run it in a Cloud based model.

    --
    Harrison's Postulate - "For every action there is an equal and opposite criticism"
    1. Re:I have to question the wisdom of this by Anonymous Coward · · Score: 0

      The cloud idea is always wonderful, but in the end, there's just too many risks involved... (I.e. - data loss, unexpected and or continual downtime, IP, etc.)

      Investing in cloud solutions only to suddenly experience 1 single instant of these sorts of dangers can sometimes be all that's required to completely derail an entire company, product, research, etc.

    2. Re:I have to question the wisdom of this by Thawk9455 · · Score: 1

      It might help to know that many universities can't just use cloud services willy nilly. There are only a few people on campus that are technically allowed to agree to a license agreement (doesn't mean others don't, but for something big it becomes important). Those license agreements have to go through significant negotiations to ensure all requirements for state laws governing the university are met, all NIH or other grant requirements are met, etc. Just the contract negotiations alone can take a year or more, if they can ever come to agreement on terms. This is definitely one area that companies, that have the ability to make their own decisions, can move around and try different services much more quickly. Universities and medical organizations have much more stringent data requirements than most businesses.

    3. Re:I have to question the wisdom of this by gtall · · Score: 3, Interesting

      Indiana University is not simply a university; it is a state school system with several regional campuses. Oddly enough, Purdue is Indiana's second state school system with its own regional campuses. They both share a campus at IUPUI (Indianapolis). I'd be very surprised if there is any free time left for this. And if there is, IU would likely just lease it to Purdue.

      All in all, it is probably cost effective for them to do this. They are unlikely to have made this decision in a vacuum; they are well aware of the alternatives. (I'm an IU alum).

    4. Re:I have to question the wisdom of this by riley · · Score: 4, Insightful

      Cloud computing is not appropriate for all types of research computing. Let's say you want to use Amazon's cloud offering, but you have a genomic and geospatial dataset of 60 TB. While not ubiquitous in research computing, it is not unheard of, especially in the fields of bioinformatics. The cost of storage and the cost of transfer will each away at whatever grant that is funding the research. This is a business decision. Does the cost of the computing resource and operation result in [ more grants / better faculty retention ] than not having it?

      The cost-benefit analysis has been done, and while cloud computing has its place, there are additional costs that make it problematic. The cloud is not a panacea.

      That said, in five years IU could very well be looking for its next big computer. The average lifespan of a supercomputer is 5-8 years. So, five years is on the early side of looking for the next big thing, but not outrageously so.

      Disclaimer -- I run high speed data storage for a university. I've written acceptance test measures for high performance computing resources. I've done the cost-benefit analyses.

    5. Re:I have to question the wisdom of this by Blaskowicz · · Score: 1

      I don't run HPC on 60 TB datasets or do anything remotely like that but I know that for me, storing a video game or something on a local disk is cheaper and higher performance than to signup for an ISP that gives me four aggregated SDSL links (symetrical 20Mbits/s) and rent Amazon storage.

    6. Re:I have to question the wisdom of this by godrik · · Score: 1

      Well, that's a valid questions. And depending on the case, the answer can be different. Many applications (especially the ones that will use GPU clusters) will need a good interconnect. Cray provides that on its machines. Last time I checked cloud platform, they did not have a suitable interconnect (10Gig ethernet has high latency).

    7. Re:I have to question the wisdom of this by iggymanz · · Score: 1

      quit using the cloud word; what are you, a marketing choad?

      multiple "cloud based resources" (i.e. just another goddamn bunch of servers on the internet) don't have the high speed network interconnects of a supercomputing cluster

  19. In a Bitcoin world by Anonymous Coward · · Score: 0

    "RTFS... in what world are 676 Kepler K20s called puny?"
    In a world where 100k GPU's can't compete with the Avalon rigs, so why do you think 20k can?

  20. FWIW by jimbrooking · · Score: 2

    The IU machine at 1 PFLOP would rank around 24th in the world and 11th in the U.S. (http://www.top500.org/list/2012/11/).

  21. Why two different GPUs / graphics cards by angel'o'sphere · · Score: 1

    What is the advantage having two different GPUs in one node? Any idea?

    --
    Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
    1. Re:Why two different GPUs / graphics cards by Anonymous Coward · · Score: 0

      I use molecular dynamics software (Lammps) for my research, and it likes to have one GPU per CPU when possible.

    2. Re:Why two different GPUs / graphics cards by Blaskowicz · · Score: 2

      There aren't two different GPUs in a node, the AMD Interlagos is a Bulldozer Opteron, made of two dies on the chip, same die as the FX-8150 CPU.
      CPU nodes run newer Opteron, about 10% more effcient, made of two Piledriver dies (similar to the FX-8350). It's weird that two different kind of CPU are used but that's probably because the GPU nodes were already made, validated etc.

    3. Re:Why two different GPUs / graphics cards by angel'o'sphere · · Score: 1

      Well, the summary says there are two different GPUs per node, or is this just missleading and they mean a CPU + GPU on a GPU node while the CPU there is different from the one on CPU nodes?

      --
      Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
    4. Re:Why two different GPUs / graphics cards by Blaskowicz · · Score: 1

      It's implied that a GPU node has CPU + GPU, because that GPU won't do anything at all on its own, not even able to talk to the networking or to run an OS (until some future generation of GPU includes CPU cores). The summary said it were "GPU-enabled nodes", too. Then the reader is supposed to understand (know already) that the AMD 16-core chip is a CPU, or infer it somehow.

      Alright, I'm only now seeing the last sentence's ambiguity, the trick is you don't see what's wrong when you understand all the techno-babble words already. The submitter did some lazy editing, adding just three letters would fix it :

      "while the 676 GPU nodes use one 16-core AMD Interlagos CPU and one NVIDIA Kepler K20."

    5. Re:Why two different GPUs / graphics cards by angel'o'sphere · · Score: 1

      Year, after your post, while I was typing, I guessed that.
      Thanx for the clarification, anyway! (I'm a software and process guy, as I lost interest in the most modern car models, or aircraft models etc. I don't realy have a clue right now about CPUs)

      --
      Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
  22. They should make a virtual supercomputer by Anonymous Coward · · Score: 0

    Just offer money (pay even in Bitcoins is fine), and in return you run a node on your computer.

    Open source the node design, with Bitcoin it drove the design of FGPA and now ASICs to run the nodes. Someone is sure to design tiny fast cores capable of just running your nodes. GPU nodes are too general purpose to be the most efficient solution, and once you make the cores simpler they become cheaper and use less power and scale far better.

    As long as you can keep the compute jobs running (so there's income there), the computing power will scale to use it.

  23. It's a good idea. by Anonymous Coward · · Score: 0

    There is a huge benefit to having a computer that you physically control. For instance, you can install whatever software you want, including experimental operating systems that might crash. Sure, you could VM all the machines, etc., but if your *thing* is developing high performance computing, then running on the real metal is important.

    The other aspect is that the money comes out of a different bucket. If you have the big box already installed and sitting there, your little $10k grant can essentially get the computer time for free (assuming the Uni sets it up that way), as opposed to buying by the CPU-Second from a cloud provider. CapEx vs OpEx as it were.

    Finally, the administrative overhead in getting an account and access to an on-campus resource is typically trivial in comparison to negotiating a procurement contract with a cloud provider, particularly if your money is coming from grant funds, since the granting agency usually sticks their fingers into the negotiations as well. (Is the provider compliant with the "Drug Free Workplace Act", for instance)

    If that "personal touch" weren't important, then we'd all be still using timesharing terminals on a mainframe.

  24. Bobby Knight powered? by rimcrazy · · Score: 1

    So do they have Bobby Knight in a closet kicking chairs to power this thing?

    --
    "TV, a medium as it is neither rare nor well done." Ernie Kovacs
  25. Clown owned? by Anonymous Coward · · Score: 0

    My mind read 'clown owned supercomputer'. Where was my subconscious going with that?

  26. It would be nice... by MrLizard · · Score: 3, Insightful

    ...if this got as much attention in the local press as throwing a ball into a basket does.

    1. Re:It would be nice... by Anonymous Coward · · Score: 0

      If it made the school millions of dollars in TV rights and merchandizing, it probably would.

  27. NOT by Anonymous Coward · · Score: 0

    It's only the biggest supercomputer only allowed to be used by one university. Even their own FAQ explains their weasel words: there's at least several (much) larger university supercomputers out there (Blue Waters, Kraken, Stampede). Just because no one else will work with them doesn't make them the biggest.

    http://kb.iu.edu/data/bcqt.html

  28. Who by Richy_T · · Score: 1

    So who was it dedicated to?