Slashdot Mirror

← Back to Stories (view on slashdot.org)

Building The MareNostrum COTS Supercomputer

Posted by timothy on from the available-at-every-pharmacy dept.

karvind writes "IBM Power Architecture Community Newsletter has a story about making a supercomputer (Number 4 on top 500 list) from easily available components (like BladeCenter and TotalStorage servers, 970FX PowerPC processors, and Linux 2.6). A joint venture between IBM and the Spanish government, it is named MareNostrum: the Latin term meaning 'our sea.' Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers."

40 of 187 comments (clear)

  1. specifically by bersl2 · · Score: 4, Informative

    Mare Nostrum refers to the Mediterranean Sea.

  2. Mare Nostrum by nebaz · · Score: 4, Informative

    Mare Nostrum literally means "our sea". It is what the Romans called the Mediterranean Sea during the Empire. As you can see, it was an apt name.

    --
    Rhymes that keep their secrets will unfold behind the clouds.There upon the rainbow is the answer to a neverending story
  3. Humans are so behind the curve by Anonymous Coward · · Score: 5, Funny

    I am an African Grey parrot, and I can tell you that while you humans are celebrating this achievement, I and my fellow Greys are laughing at you. Supercomputers are old news to us; in fact, one of my friends solved the halting problem while taking a crap the other day. Seriously, people, we like you 'cause you feed us, but leave this kind of stuff to us.

    (I tried to register an account but /. thought my user name was too long)

    1. Re:Humans are so behind the curve by EvilTwinSkippy · · Score: 2, Funny
      (I tried to register an account but /. thought my user name was too long)

      What kind of troll would parrot back such a hackneyed old line.

      --
      "Learning is not compulsory... neither is survival."
      --Dr.W.Edwards Deming
    2. Re:Humans are so behind the curve by TeknoHog · · Score: 2, Funny

      Are you an African or European par... oh crap, nevermind.

      --
      Escher was the first MC and Giger invented the HR department.
  4. It's all fine by Anonymous Coward · · Score: 5, Funny

    but does it run Linux? Oh crap, never mind.

  5. Top 500? by Ubi_NL · · Score: 3, Funny

    (Number 4 on top 500 list)
    ...while being Number 6 on top 300 list, and Number 65 on top 2000 list.

    This is like those CDs that have 'best of the Top40' and not contain the top10 list of that

    --

    If an experiment works, something has gone wrong.
  6. and in 32 years by headlessspider · · Score: 3, Insightful

    and after about 16 (or 32) years we'll have that power in our desktops...

    --
    -- and if life has failed you leave the cross you're nailed to
  7. Re:Beowulf cluster? by travisco_nabisco · · Score: 2, Informative

    It probably has to do more with the fact that as you increase the number of nodes, your increase in performance decreases on a per node basis. To get that many nodes working together takes an incredible amount of resource management. It makes you wonder where the limit currently is for if it is worth adding an extra node, or if the resource management requirements negate the extra nodes computational power.

  8. Off who's shelf? by identity0 · · Score: 4, Insightful

    I love the first line in the article, which ends, "is constructed of such totally off-the-shelf parts as IBM BladeCenter JS20 servers, 64-bit 970FX PowerPC processors, TotalStorage DS4100 storage servers, and Linux 2.6. This is its story."

    Right, like I regularly go to Fry's to stock up on some DS4100s and Bladecenters. I'd love to be the geek for whom that stuff is "off-the-shelf". Can you even buy bare PPC CPUs and mobos?

    1. Re:Off who's shelf? by Anonymous Coward · · Score: 3, Insightful

      IBMs shelf. The stuff in itself is nothing particular, just ordinary computer stuff sold by IBM to hundreds or thousands of other customers.

      Just because you "can't" buy the stuff in pieces (IBM will probably gladly sell you the stuff as spare parts, if you prefer to assemble it yourself) doesn't mean it's not off the shelf.

    2. Re:Off who's shelf? by lachlan76 · · Score: 2, Insightful

      You don't need your own fab, and you don't need to rent one. If you don't need custom components, it's off-the-shelf.

    3. Re:Off who's shelf? by PayPaI · · Score: 2, Insightful
      Pegasos.
      $775 for a G4 (1GHz?) on an ATX board with specs comparable or less than a $112 Athlon64 motherboard.
      Terons
      $3,900(!) for a board with a 750FX processor (unknown speed) and technology comparable to what was going out of style 4 years ago. (USB1.1? 10/100 Ethernet? PC133 Memory???)
      I'm not holding my breath for these. Call me when I can get a decent motherboard with a 1GHz processor for less than $200.
    4. Re:Off who's shelf? by blamanj · · Score: 2, Informative

      Well, the info is right here for quantity 1, and there a button that says "Configure and order a JS20," so if you're willing to order a few thousand of them (they're about $4K/ea with 2.5G RAM), you can build your own.

      Actually, that's a reasonable price, considering it's IBM, who aren't usually considered a bargain brand.

    5. Re:Off who's shelf? by clem.dickey · · Score: 2, Informative

      You'll need a chassis for that blade, of course. Luckily, they're half-off through the end of March. Buy the entry model and get change back from your $1000 bill. Oh, and you'll want a rack to put the chassis in. But at $1489 for the rack, check the local surplus store first. And while you there pick up a display, mouse and keyboard.

  9. War in the age of information warfare by flopsy+mopsalon · · Score: 5, Funny

    I happened to look at the Top 500 supercomputers site and I couln't help noticing out of the top 5 supercomputers almost half are in non-US countries like Spain and Japan. This is not to beat some kind of patriot act drum. Instead, it got me to thinking.

    With supercomputing powers now avaible to any country or group with a few readily available components, it is only a matter of time before these supercomputing powers may be used by a rogue state or radical group to cause havoc among electronic communications using methods like denial of service attacks, spyware, and crapflooding message boards.

    I think it is high time the nations of the world put their heads together and addressed this issue. For example, I don't think the US Federal Government even has any cabinet-level position like Secretary of Information Technology or something like that. When are they going to get with the times? It will probably take another terrorist attack or something.

    1. Re:War in the age of information warfare by mikelin.ca · · Score: 2, Insightful

      Uh oh.
      Someone's been watching too much 24.

    2. Re:War in the age of information warfare by grozzie2 · · Score: 5, Informative
      These things are already being used by rogue states. The us military has a bunch of them dedicated to modelling nuclear events.

      With regard to denial of service attacks, there's a cluebox over in the corner, you need to go grab a couple out of the box. DOS attacks dont require a big computer, they require massive bandwidth with massive routing diversity available. The actual computer power required borders on insignificant. A supercomputer like this is useless for that kind of thing, by necessity, it will have an internal networking and communications environment, and likely only a relatively low speed interconnect to external networks.

      But look on the bright side, the knee jerk 'terrorist behind every lamp post' reaction is just what the american government has been trying to instill in the population for the last few years. Your post here shows, it's been an effective campaign, money successfully spent, and the objective achieved. It's become the 'trendy' response to just about everything these days.

    3. Re:War in the age of information warfare by WindBourne · · Score: 2, Interesting

      First and formost, that is NSA's directive; That is to secure our systems and networks.

      Next there have been several CS security czars, but all have quit because they have not had the response from the admin that they thought was needed.

      Will it take a terrorist hit to get us really thinking about all this? Nope. In spite of 9-11, we are still not really any more "secure" than we were on 9-10.

      What it will take is lawsuits against companies. A good one that is going through now, is the one where a guy got ripped off for 90k using BOA. Now it was not BOA fault(they had a secured server). But they allowed that guy to use his system (he seems to forget the use at your own risk think in the EULA that he agreed to). Most likely, BOA will lose money no this. Then they will re-think through there strategy. It will probably be to check the client and browser and see if it is known high risk (hummmm, which OS and browser are very high risk, hummm). At which time, they will warn the user and perhaps suggest another browser (and maybe an OS or 2). At the point that this costs institiions money is when they will take it serious.

      --
      I prefer the "u" in honour as it seems to be missing these days.
    4. Re:War in the age of information warfare by Anonymous Coward · · Score: 4, Insightful


      Keep in mind that these are Top500 KNOWN supercomputers.

      It's quite possible that many more computers that exist but are operating under classified conditions. For instance I would be suprised if the NSA had something for breaking crypto that rivaled some of the machines in the top 50 or so super computers, but it's not something that would appear on this list. This is for boasting rights only, if a place chooses not to publicize their computer, it won't end up on the list.

      American computers pretty much dominate. Some like the Spainish one are built by American companies (like IBM), but just happen to be installed in foreign places.

      Now the NEC Earth Simulator is a BIG exception. It dominated the top 500 for a considurable period of time and is completely japanese built, designed and owned. It is a huge technological acheivement and the pinnacle of technology for it's day. It's hard to relate to how significant it was. It is the top of the line when it comes to old-school massively parrallel supercomputers, blew American supercomputers out of the water.

      But along came Linux clusters (which in many ways is competely unsuitable for some of the things that Earth Simulator can do) and now Blue Gene. Which can possibly do twice the work as Earth Simulator, but consumes a fraction of the power and space needed to house these types of computers.

    5. Re:War in the age of information warfare by EvilTwinSkippy · · Score: 2, Insightful
      Please don't let there be a Secretary of Information Technology. The creation of cabinet level posts is like 1984. We create a Homeland Security Secretary to instill fear and insecurity with his "Terror Alert" system. We have a Secretary of Defense who invades other countries. Our Secretary of Education is wrapped up in testing to the point of interfering with actual education.

      I shudder to think of what a "Secretary of Information Technology" would be. In my tinfoil sheilded skull I imagine a jack boot on the throat of the internet.

      --
      "Learning is not compulsory... neither is survival."
      --Dr.W.Edwards Deming
  10. timely and focused PR by bandix · · Score: 5, Informative

    This is all about timely and focused execution. The speed at which this project was realized is important. Consider: from the initial concept in late December of 2003 to assembling the computer in Madrid took less than a year. Normally, this kind of supercomputer projects take years.

    Lame!

    SGI had NASA AMES' Columbia online in 120 days, and landed #2 on the Top500.

    --
    Brandon D. Valentine
  11. Sounds like by jim_v2000 · · Score: 5, Funny

    Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers.

    Sounds like the specs of Microsoft's Xbox 3...

    --
    Don't take life so seriously. No one makes it out alive.
  12. WHAT'S WITH THE FREE ADVERTISING? by Anonymous Coward · · Score: 2, Insightful

    I'm sorry, but why are we giving IBM free press again? For god's sake, the very first sentence of the freaking article is utter rubbish:

    The MareNostrum supercomputer at the Barcelona Supercomputing Center, ranked number four in the world in speed in November 2004, is constructed of such totally off-the-shelf parts as IBM BladeCenter JS20 servers, 64-bit 970FX PowerPC processors, TotalStorage DS4100 storage servers, and Linux 2.6

    That's it - the first sentence of the article, if you exclude the title, the credits and the date, that is.

    And we can already call advertising bullshit. I'm sorry, but how is the MareNostrum system made of any more "totally off-the-shelf" parts than the number two system on the Top500, NASA's very own Columbia? In fact, the 64-bit 970FX PowerPC processor is NOT an off-the-shelf part: unlike the Itanium 2 CPUs in Columbia, you can NOT buy such CPUs individually (for good reason: unlike the Itanium CPUs in Columbia, MareNostrum's CPUs are not socketed but soldered to the JS20 system board, so there goes upgradeability...)

    Seriously though, why should we give a rat's ass about MareNostrum? Columbia is faster, more efficient, really is made from off-the-shelf parts and also runs Linux.

    What's that? Oh, IBM are a good company and SGI aren't? For fuck's sake, SGI are better friends of Linux than IBM are. What did IBM do for linux? Nothing compared to SGI. IBM ported JFS, a crappy journaling filesystem, oh and they ported it to their own POWER/PowerPC architecture systems. W00p!

    SGI:

    - Gave Linux XFS, one of the fastest filesystems around, with _many_ advanced features (just look in your kernel config sometime)

    - Scaled Linux beyond 64 CPUs for the first time (and indeed, they hold the record at 2,048CPUs): they fixed a _ton_ of scalability problems, and continue to do this on a daily basis (just look at this week's archive of the linux-ia64 mailing list to see what I mean!)

    - Open sourced their Itanium compiler

    - Created OpenGL (notice carefully the Open in OpenGL. You can bet your bottom dollar if IBM created a funky new graphics API it would _only_ work on PowerPC machines with IBM video hardware!)

    Yada yada. All I'm trying to say is there are other companies out there who have really taken Linux to heart and have made open source development in their best interest, not just IBM. SGI is just one example, there are many others.

  13. Re:Beowulf cluster? by commodoresloat · · Score: 3, Insightful

    Yeah but it's made from "easily available components." Hmmm, I think I've seen that before, at the University of Virginia. Don't G5s qualify as "easily available"?

  14. In other news, SHA1 was broken... by Anonymous Coward · · Score: 2, Funny
    Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers.

    It's just the thing to find SHA1 collisions of ISO images in 2^56 operations...

  15. Well yeah,... by Create+an+Account · · Score: 2, Funny

    ...but how fast can it open Photoshop CS?

  16. Re:Who cares about easily available? by jokumuu · · Score: 2, Informative

    Well, if you really need a supercomputer, the first step is not to get the computer, but instead the funding. As super computers go, if your needs are served by a distributed computing environmet, an "entry level" supercomputer does not really cost all that much compared to "traditional" supercomputers. Yes we are still talking about hundreds of thousands of dollars. (there seems to still be 256 processor 2.8ghz xeon based computers with gig ethernet connectivity on the latest top 500 list) If you need to solve problems that do not work in paralell, you are definitely out of luck.

  17. That's a beefy computer by C0d1ngM0nk3y · · Score: 2, Insightful


    but I bet Windows still runs slow on it.

    --
    Coding Monkey.org - Spanging the heavy spade of truth into t
  18. Supercomputer = Pile of computers? by ceeam · · Score: 2, Insightful

    Given enough money how is that impressive anymore? What's the best single-thread-performance machine today?

    1. Re:Supercomputer = Pile of computers? by dsevilla · · Score: 2, Funny
      Yeah, that's the point. Also, they don't even do any advance in programming models. Just MPI/PVM with my old friend fortran or C.

      Haha, but the funny thing is that they'll end doing high performance computing based on Web Services!!

      Best regards,

      diego.

  19. riiight . . . easily available components by Raccroc · · Score: 2, Insightful

    I just love how every time someone writes about another grid or supercomputer or beowulf cluster they always say "easily available components" as if I could find most of them in a standard IT closet or just run down to the local computer shop and pick them up with my corp. AMEX.

    In what world is 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers easily available??? Maybe if you are Big Blue, but then, why would it be more difficult for them to throw together a fully proprietary supercomputer?

    1. Re:riiight . . . easily available components by luvirini · · Score: 2, Insightful

      If you have the money, the components are readily available and tested. It is allmost allways much more difficult to create everyting from scracth than using components that are allready designed and working. Thus the money needed to build it from scracth would be much higher and timeframe longer.

  20. But really... by nicc777 · · Score: 2, Funny

    ...does it have an AGP slot?

    --
    Need an ISP in South Africa?
  21. Mare Nostrum by woah · · Score: 3, Funny
    Mare Nostrum is Latin for "our sea"?

    But I thought it was Ouray Easay!

    What's going on?

    This is all so confusing! I need to take a nap.

  22. Re:Imagine... by Ray+Alloc · · Score: 2, Funny

    You'll notice the "redundant" moderation, which is a perfect fit, given the context :-)

  23. No memory specs? by PornMaster · · Score: 2, Interesting

    Am I the only one finding it curious that nowhere in the specs do they mention how much RAM there is per node or in the aggregate?

    It mentions how many nodes, how many CPUs, how many racks, how much storage, but not how much RAM.

    --
    500GB of disk, 5TB of transfer, $5.95/mo
  24. Re:Beowulf cluster? by tomhudson · · Score: 3, Interesting
    It makes you wonder where the limit currently is for if it is worth adding an extra node,
    If you're the vendor, there is no limit. Every extra node == $$$

    If you're the buyer, there is no limit. Every extra node == _MORE_STATUS_

    If you're the guys writing code for it, there is no limit. Every extra node == job security++

    If you're the people administering this, there is no limit. Every extra node == bigger budget next year

    See, citizen? Size does count.

  25. Re:Beowulf cluster? by antifoidulus · · Score: 3, Informative

    Eh, not really. In a lot of the algorithms, there comes a point when adding nodes will make it SLOWER because the increases in communication time are greater than the decrease in computation time. Now granted this does depend a lot on a) what you are doing with the machine and b) the machines themselves, but just thinking that people who make these things love to just pile on hardware is a bit naive....

    --
    Monstar L
  26. CELL Supercomputer by Omegalomaniac · · Score: 2, Interesting

    The scary thing is, that the CELL with its potential for 25 GFlops of double precision floating point, could rival this system with just 1600 8 SPE units.

    Granted, the CELL isn't exactly off the shelf, and I'm willing to bet 4,564 970FXs will be cheaper than 1600 CELLs for quite some time, so the project still has merit.