Slashdot Mirror

← Back to Stories (view on slashdot.org)

North America's Fastest Linux Cluster Constructed

Posted by CowboyNeal on from the where-was-the-lightning dept.

SeanAhern writes "LinuxWorld reports that 'A Linux cluster deployed at Lawrence Livermore National Laboratory and codenamed 'Thunder' yesterday delivered 19.94 teraflops of sustained performance, making it the most powerful computer in North America - and the second fastest on Earth.'" Thunder sports 4,096 Itanium 2 processors in 1,024 nodes, some big iron by any standard.

84 of 325 comments (clear)

  1. Imagine a ... by Anonymous Coward · · Score: 5, Funny
    pineapple on a monkey.

    And you thought I was going to say something else...

    1. Re:Imagine a ... by JPriest · · Score: 5, Funny

      I see your pineapple monkey and raise you a rabbit with a pancake on its head. (cache)

      --
      Saying Java is nice because it works on all OS's is like saying that anal sex is nice because it works on all genders.
    2. Re:Imagine a ... by nocomment · · Score: 3, Funny

      Imagine SETI running on that!!!

      --
      /* oops I accidentally made a comment, sorry */
      /* http://allyourbasearebelongto.us */
  2. Very great and all... by irokitt · · Score: 5, Interesting

    But why did they use itanium processors? Were they acquiring parts before Opterons were availabel? Did they have a problem with Xeon processors? Or did they have too much cash lying around?

    --
    If my answers frighten you, stop asking scary questions.
    1. Re:Very great and all... by MBCook · · Score: 5, Insightful
      I like the Opteron as much as the next guy and I'm no fan of the Itanic. But the fact is that for some types of calculations the Itanium can smoke Opterons. If you want the fastest, in many cases you want the Itanium. If you want the best value (which still performs quite close to the fastest), you want an Opteron. I don't remember which operations are better on which, so you'll have to look that up (or someone will reply with the answer).

      Depending on budget, price (I wouldn't be suprised if Intel cut them a sweet deal to get this cluster publicized to help our their product's sales), and other factors, the Itanium could have been a good choice.

      Especially if they were using software that had been designed for the Itanium (like they were replacing an older cluster) then they wouldn't have to port the software which would have saved real money.

      I'm not a fan of Intel lately, but the Itanium isn't overpriced garbage no matter what. That smacks of fanboyism. Interesting you didn't add G5s to your list, BTW.

      ALSO: Don't forget that the Itanium 2 was DESIGNED FOR big iron, while the Opteron was designed for servers and small iron. They can be used in other ways (you could run a web site off an Itanium 2), but the Itanium was designed for these kind of applications.

      --
      Comment forecast: Bits of genius surrounded by a sea of mediocrity.
    2. Re:Very great and all... by tap · · Score: 5, Informative

      Do you have any kind of benchmark where the Itanium smokes the Opteron? The Itanium does have a greater memory bandwidth, but not by a lot. If you look at the spec benchmarks, it can be faster on some of them, but not by a lot. However, the Itamium is a lot more expensive!

      Compared to a Xeon or AthlonMP cluster, the Itanium faired poorly in price/performance. The only reason to use Itaniums was if you needed 64 bits for more than 4GB of memory, or needed high single CPU performance for a pooly parallized application. (Of course if your application parallizes poorly, a cluster is probably a bad choice to begin with). Then Opterion came out and changed all that. It's 64 bits, it's fast, and it's a fraction of the price of the Itanium2.

      I just purchased a new Beowulf cluster. The decision was between Xeons vs Opterons. The Opterons had better price/performance, but the Xeons would fit in better with our existing Pentium3 Beowulf, other ia32 servers, and existing software. In the end, we went with Opterons. Itanium2 was never even in contention. Just one look at the price and performce of a Itanium2 system was all it took to cross it of the list.

    3. Re:Very great and all... by evilviper · · Score: 2, Funny
      I don't remember which operations are better on which, so you'll have to look that up (or someone will reply with the answer).

      Wow! What a great argument strategy! Let me try...

      I like slashdot as much as the next guy. But the fact is that CmdrTaco is an evil blood-sucking cyborg who kills a puppy for each and every slashdot subscriber. I don't remember where I found this irrefutable proof, so you'll have to look it up yourself (or someone will reply with it).
      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
    4. Re:Very great and all... by Roydd+McWilson · · Score: 2, Informative

      GCC? On Itanium? Optimized quite well? Whatever. Check out Trimaran for the HP/Illinois/NYU compilers which basically inspired Itanium.

      --
      THE NERD IS THE COMPUTER.
    5. Re:Very great and all... by Yenya · · Score: 2, Informative
      The problems of Opteron against Itanium2 are:
      • You cannot order the bigger L2 cache (Itanium2 can have 6MB).
      • For "randomly branched" code you need as short pipeline as possible. This is the reason Athlon outperformed PentiumIV at the same clock speed. Now Itanium2 has 6-stage pipeline, while Opteron has 20-stage, IIRC.
      OTOH, for full performance you need _much_ finely-tuned compiler for VLIW CPUs such as Itanium2 than for a generic CISC or RISC CPU.
      --
      -Yenya
      --
      While Linux is larger than Emacs, at least Linux has the excuse that it has to be. --Linus
    6. Re:Very great and all... by identity0 · · Score: 2, Informative

      I am not an expert, but in general, Opteron seems to be targeted more for the workstation/server market than the supercomputer market. It's not like they really need x86 backwards-compatibility in the supercomputer field, so Opteron doesn't seem to be optimized for that market. I think Intel may have made IA-64 with supercomputers in mind than AMD did with x86-64.

      Some reps from SGI came to my LUG the other day, and talked about their clusters and supercomputers. The guy doing the Q&A said that he personally liked the Opterons and x86-64, and that the Opterons were fast, but for what SGI does they preferred Itanium. The Opterons have their memory controller embedded in the chip itself, which is great for 1 or 2 or even 8 processors. However when you go up to a 512 processor single-system image supercomputer like SGI's Altix, a lot of the memory controller stuff is done in the switches or otherwise off-chip. Itanium allowed for more flexibility in how they did memory controllers, because they don't have an on-chip one.

      There were some other reasons too, like having more registers, etc. that made SGI choose Itanium over Opteron. I don't know how applicable they are to this situation, as this doesn't seem to be a SSI supercomputer.

    7. Re:Very great and all... by fupeg · · Score: 3, Informative

      Try any from SPEC, for example. Maybe you're thinking about x86 because otherwise, the Itanium2 is way out of the Opteron's league (as well as price range, but that is besides the point.)

    8. Re:Very great and all... by Too+Much+Noise · · Score: 2, Interesting

      Check SpecFP benchmarks - Itanium2 smokes pretty much everything else. Reason? it was meant to be a fp monster from the beginning. Integer math is weak (Opterons kick Itanium on that pretty hard), but FP math, especially vector FP math is Itanium's selling point. Why do you think the vast majority of I2 sales were to scientific research groups? (check the target profile for SGI's I2 clusters - research and defense)

    9. Re:Very great and all... by tap · · Score: 4, Informative

      Ok, checked them again. The best 1.5 GHz Itanium2 SPECfp2000 score is 2148 while the opteron 248 is 1691. That's 27% faster. I'd hardly call that smoked.

      The Opteron 248 is $670 on pricewatch, while the 1.5 GHz It2 is $5200! The motherboards are like $1400 vs $400.

      You have to keep in mind that this isn't a single machine, it's a cluster. You could take the money spent on an Itanium2 cluster, and buy an opteron cluster with five times as many processors. I am well aware that one does not get perfect scaling. But if you are running something on a cluster in the first place, I have a hard time imagining something that is faster with one fifth as many 27% faster processors. Yes, there are codes that would be faster on 1000 Itanium2 vs 5000 Opterons, but you would never runs these on cluster, because they would be faster still shared memory system.

    10. Re:Very great and all... by tap · · Score: 2, Interesting

      I wish those graphs were easier to read! From the looks of it, at ~32 processors the 1.5 GHz It2 has a 12 second run time and the 1.8 GHz Opteron is 24 seconds. The It2 cluster has Quadrics high end Elan4 interconnect, while the interconnect of the Opterion cluster isn't listed. It might have GigE for all we know.

      The It2 probably cost around 5 times as much as the opterons, so a real comparison would be 32 It2 processors vs 160 Opterons. With the scaling shown for that model, the Opterons of equilivent cost would be 2-3 times faster than the Itaniums.

    11. Re:Very great and all... by boots@work · · Score: 3, Insightful

      You make good points throughout your reply, but if you're clustering--the idea of buying the fastest available just doesnt make sense, unless underlying it really is that much faster in even a cluser envrironment?

      That sentence doesn't even parse, but anyhow: single-thread performance still matters to clusters. There is a limit to how much you can effectively parallelize many problems. If that limit is 1, then you need a Cray or something. If the limit is extremely high, you can use distributed.net, or a cluster of recycled C64s.

      In the middle, you might be able to parallelize the task to a limited extent. If you can only split your work into 500 parallel tasks, then you want 500 of the fastest processors you can get. For many applications, that means 500 Itaniums. Even if you could buy 800 Opterons for the money, they might not be as fast.

      only other option would be they thought intel would hold up better/be more stable. /shrug

      Itanium has slightly better manageability; you can find out when a memory module or CPU is likely to fail for example. There is a heap of error detection/correction in the CPU, far beyond Xeon or Opteron afaik. If you have hundreds of machines being able to easily detect failures is worth something.

      (Or you can just take the google route and let it fail and replace the whole box. But that really requires your whole application to be written to accomodate it.)

    12. Re:Very great and all... by tap · · Score: 2, Insightful

      I don't know if I would call NUMALink a true shared memory system. It is NUMA after all! I was thinking of 32-64 way machines with a true shared memory system, or large vector machines based on SX-6 processors for example.

      But look at NUMALink4, its got 6.4 GB/sec per link bandwith and 240ns latency.

      QsNetII is just under 1 GB/sec bandwidth, the limit of PCI-X, with a latency of 3us.

      So, NUMALink4 has 6.4 times the badwidth and 12.5 times less latency than QsNetII. That a much larger performance difference than Opteron vs Itanium!

    13. Re:Very great and all... by joib · · Score: 3, Informative


      There is a limit to how much you can effectively parallelize many problems. If that limit is 1, then you need a Cray or something.

      Well, Crays are also parallel computers, so they won't help you much in this situation. Some Crays do have vector processors, but that is also a sort of parallelism. It's just that you use that parallelism through tuned BLAS libraries or with a vectorizing compiler (e.g. Fortran 95, HPF and such things), instead of doing it manually with MPI or threads or something like that. So if you're problem is totally serial, a vector processor won't help you either.


      (Or you can just take the google route and let it fail and replace the whole box. But that really requires your whole application to be written to accomodate it.)

      Not necessarily. Most supercomputers are not used to run a single job taking months, but rather they run lots of smaller and shorter jobs. On the p690 cluster where I do my stuff, I (and apparently most users) mostly run jobs using about 8-16 cpu:s , with a runtime of a few hours to a day. If one node would fail, the jobs that are executing on that node would also fail. It's no big deal, just resubmit the job to the queue when you get around to it.

      Of course, if you're programming one of the very few and far between applications that has a runtime of months, you certainly want to save intermediate results once in a while. Not only to guard against hardware failure, but also so that the user can check the intermediate result and see if the app is still on the right track. It would be quite a bummer to use months of cpu time only to realize the entire thing is wasted because you specified the initial values wrong.. :-)

    14. Re:Very great and all... by SuperQ · · Score: 4, Insightful

      the problem is not that you couldn't get the processors, the problem is scale.

      A system like this will use a high-speed interconnect, not gige. The popular choice right now is infiniband, and that stuff isn't cheap, and also has limits to the number of ports per IB switch. The system at LLNL has 4 procs per node, which reduces the number of IB switches involved. 5000 dual proc (you suggest 248 proc) machines would require 2500 IB ports, instead of 1024.

      now if you considered the opteron 848 ($1300), in 8proc nodes, that would be something to think about, reduce the number of IB ports in half, and be able to double the processors.

      the other consideration is also processor scale. the 27% per CPU is signifigant, because even with dual proc SMP, you loose some % of the CPU time. There was a posting on an article about how processors scale this way. I forget how the principle works.

  3. I'd hate to be the guy... by krammit · · Score: 3, Funny

    ...who gets the electric bill.

    I cringe when I leave the A/C on for too long..

    --
    "Watch your cornhole, bud."
    1. Re:I'd hate to be the guy... by Anonymous Coward · · Score: 2, Insightful

      Pay taxes? Then you are the guy who gets the bill.

      That said, I think our national labs are pretty great when they aren't designing nukes.

    2. Re:I'd hate to be the guy... by MrRuslan · · Score: 2, Interesting

      You know what would be cool...if it would be posible to somehow recycle part of the heat from this clust into energy.perhaps tunneling all of it into one source and it boil water into steam to make energy...kidding about that last part ;)

  4. "Most" powerful by Alomex · · Score: 4, Interesting

    Look, any way you cut it the 100K computers Google is reputed to have is the most powerful Linux cluster anywhere in the world.

    1. Re:"Most" powerful by 0xC0FFEE · · Score: 5, Insightful

      If google's cluster is interconnected via ethernet, there is a whole range of computational problems it can't tackle. If you want to simulate a spatial phenomenon with lot of things going back and forth in a volume, you're bound to have a _lot_ of communications. The cost of the interconnect system in those simulation systems is often a substantial proportion of the total cost of the installation.

    2. Re:"Most" powerful by irokitt · · Score: 4, Interesting

      4,096 iTanium processors versus ~8,000 boxes sporting Pentium II, III, and 4 processors. But remember that the interest Google has is in disk access and redundacy, not complex mathematical computation. So it isn't configured as a 'supercomputer' per se.

      --
      If my answers frighten you, stop asking scary questions.
    3. Re:"Most" powerful by smitty45 · · Score: 4, Insightful

      Powerful = fastest computation, not biggest. A roomfull of Chevettes do not make a Corvette.

    4. Re:"Most" powerful by Boone^ · · Score: 2, Interesting
      You're right, but this still only uses an off-the-shelf interconnect from Quadrics. Quadrics bills themselves as the "price/performance leader", not the performance leader.

      There are many purpose-built supercomputers coming up (like Sandia's Red Storm) that use custom yet pricy interconnects that end up smoking anything Quadrics can put together. Anytime your interconnect relies on a PCI-type bus, you take a latency penalty on each end. Real supercomputers access memory on other nodes directly, not through a generic shared bus to a fancy network card.

      Read some of this if you're bored, it goes through Sandia's entire thought process. http://www.lanl.gov/orgs/ccn/salishan2003/pdf/camp .pdf

    5. Re:"Most" powerful by tap · · Score: 5, Informative

      I think you've got that backwards, Quadrics is the performance leading, not the price/performance leader. Myrinet, SCI, and Infiniband all beat it in price/performance. Quadrics is faster, and scales to more nodes than the others.

      According to Quadrics latest price list, the cards are $1200 each, $913 per port for a 64 node switch, and $185-$265 for a cable. That's $2300/node.

      Myrinet cards are $595, the switch is $400 per port for 64 nodes, and the cables are ~$50. That's $1050/node.

      Quadric's price for a 1024 node interconnect is $4,176,094. That's hardly chump change. The bandwith is about 10x higher than gigabit ethernet, and the latency about 100x lower.

  5. how fast is it? by chickenrob · · Score: 4, Funny

    Is it fast enough to run all the latest spyware, adware, and viruses and not slow down your solitaire game?

    --
    People say my sig is the best thing about me.
  6. Awesome! by haxeh · · Score: 2, Funny

    That's amazing!

    Now we can... uhh... what are we supposed to do with that much power again?

    1. Re:Awesome! by MrRuslan · · Score: 5, Funny

      It's all for reserved for Doom III on longhorn.

  7. but but but by Anonymous Coward · · Score: 4, Funny

    Can it run Windows?

  8. Did hell freeze over? by SuperBanana · · Score: 4, Funny

    LLNL built a supercomputer, and it's going to do things besides simulate nuclear weapons?

    Quick, someone ring Satan and ask how the sno-cones are.

    --
    Please help metamoderate.
    1. Re:Did hell freeze over? by geek · · Score: 4, Insightful

      I grew up in Livermore, the lab was some 500 yards from my bedroom window. They work on a lot more than nuke simulations, including alternate fuels (my brother in law was driving a hydrogen fuel car from the lab 10 years ago as a test), laser technology and about a million other things. Why is it people like you who hear "Nuke" rant on and on like biased little children and post inflamatory things like this?

      The lab is a GOOD thing damnit. Do you even know what nukes are? What nuclear research has done for us? Grow up man.

    2. Re:Did hell freeze over? by Ralph+Wiggam · · Score: 2, Interesting

      Nuclear weapons are dinosaurs. They did their job from 1945 to 1991. Who are we going to nuke now? The North Koreans, who are proposing a peace treaty? Canada? Nukes are weapons of deterence. Osama isn't sitting in his cave thinking, "We shouldn't mess with the US, they might nuke us."

      -B

    3. Re:Did hell freeze over? by NEOtaku17 · · Score: 2, Interesting

      I don't know if people realize this but one of the major reasons computers exist as they do today is because of military research. Yeah building nukes was one of the driving forces behind more complex and powerful computers. Both nuclear power and computers grew up as a result of WW2. Now stop complaining about military research because you wouldn't even be typing messages on Slashdot if it weren't for the need of more advanced military technology pushing science and application of science further ahead.

      --
      Creative Demolition
  9. Google Cache by nadolph · · Score: 2, Informative

    http://www.google.ca/search?sourceid=navclient&ie= UTF-8&oe=UTF-8&q=cache:http%3A%2F%2Fwww%2Ellnl%2Eg ov%2Flinux%2Fthunder%2F

    --
    With the moo and the cow and the fish. Minesweeper Record: 7 sec
  10. I don't care what anyone says by MrRuslan · · Score: 5, Funny

    this thing should do doom 3 with a software renderer at a very playable 47 FPS...

  11. vs google by docl · · Score: 2, Interesting

    This is probably a stupid question, but would anyone care to explain how this is different than a really large cluster. For example, if people estimate google to approach 100K nodes, how does this compare?

    1. Re:vs google by complete+loony · · Score: 4, Informative
      Google have lots of little (in comparison only) jobs that have to process heaps of data, googles cluster(s) wouldn't perform well in the top 500 list since they don't concentrate on link speed, which is the main factor in performace for supercomputers, but on raw data processing power.

      The GFS article that appeared a while back said they used standard 100MBit ethernet, this is not going to get you a good score in any supercomputer benchmark.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
  12. Re:Whoa. by TravisWatkins · · Score: 4, Informative

    That would be the Earth Simulator in Japan.

    --

    "But I'm still right here, giving blood and keeping faith. And I'm still right here."
  13. Finally... by Fry-kun · · Score: 2, Funny

    ...I can back up my brain

    --
    Did you know that "FTW" ("for the win") is a direct translation of "Sieg Heil"?
  14. The way I see it... by blackula · · Score: 2, Redundant

    ...there are basically three type of clusters: 1) shared nothing: in this, each computer is only connected to each other via simple IP network. no disks are shared. each machine serves part of data. these cluster doesn't work reliably when you have to aggregations. e.g. if one of the machine fails and you try to to "avg()" and if the data is spread across machines, the query would fail, since one of the machine is not available. most enterprise apps cannot work in this config without degradation. e.g. IBM study showed that 2 node cluster is slower and less reliable than 1 node system when running SAP IBM on windows and unix and MS uses this type of clustering (also called federated database approach or shared nothing approach). 2) shared disk between two computers: in this case, there are multiple machines and multiple disks. each disk is atleast connected to two computers. if one of the computer fails, other takes over. no mainstream database uses this mode, but it is used by hp-nonstop. still, each machine serves up part of the data and hence standard enterprise apps like SAP etc cannot take clustering advantage without lot of modification. 3) shared everything: in this, each disk is connected to all the machines in the cluster. any number of machines can fail and yet the system would keep running as long as atleast one machine is up. this is used by Oracle. all the machine sees all the data. standard apps like SAP etc can be run in this kind of configs with minor modification or no modification at all. this method is also used by IBM in their mainframe database (which outsells their windows and unix database by huge margine). most enterprise apps are deployed in this type of cluster configuration. the approach one is simpler from hardware point of view. also, for database kernel writers, this is the easiest to implement. however, the user would need to break up data judiciously and spread acros s machines. also adding a node and removing a node will require re-partitioning of data. mostly only custom apps which are fully aware of your partitioning etc will be able to take advantage. it is also easy to make it scale for simple custom app and so most of TPC-C benchmarks are published in this configuration. approach 3 requires special shared disk system. the database implementation is very complex. the kernel writers have to worry about two computers simultaneously accessing disks or overwriting each others data etc. this is the thing that Oracle is pushing across all platforms and IBM is pushing for its mainframes. approach 2 is similar to approach 1 except that it adds redundancy and hence is more reliable. so what type are we talking about here?

    1. Re:The way I see it... by twoslice · · Score: 3, Funny
      ever

      hear

      of

      Paragraph

      tags?

      --

      From excellent karma to terible karma with a single +5 funny post...
    2. Re:The way I see it... by sapbasisnerd · · Score: 2, Informative
      Had to decide to reply to this or mod it down, decided to reply.

      That's a wildly inaccurate summary of the landscape of RDBMS clustering technology.

      Problem is, that's not what we are talking about here.

      So the answer to your question at this end is almost certainly "none of the above" or probably more correctly "some bits of all of the above". Functionally most of the kind of stuff you do here doesn't need shared concurrent access to the same data files however for simplicity of implementation they probably nevertheless run GPFS so that all nodes can see all files.

    3. Re:The way I see it... by skdffff · · Score: 2, Informative

      There are basically two types of clusters - HA (High Availability) and HPC (High Performance Computing). They both called "clusters" (what confuses some people) but designed for completely different purposes. You're talking about variations of first type while cluster in the article is HPC cluster.

  15. Another Article by Flashbck · · Score: 4, Interesting

    And only 55 people were needed to build it!

  16. Clarification by Anonymous Coward · · Score: 2, Funny

    19.94 teraflops??

    Gimmy something I can grasp; what's this in BogoMips?

  17. 2nd fastest supercomputer by m1kesm1th · · Score: 5, Funny

    Also in completely unrelated news, Bill Gates announced the first fully installed test of Longhorn happened today.

  18. OK, here goes... by rco3 · · Score: 3, Funny

    Hey, with a Beowulf cluster of these, I can run Longhorn!

    OK, I'm done. Sorry. Mod away!

    --

    Ce n'est pas un vrai mouvement de robot!
  19. apple's response will be interesting by Twid · · Score: 4, Insightful

    If I calculate right, they are claiming an Rmax of 19.94 teraflops with 4096 processors.

    The Virginia Tech cluster for Apple had an Rmax of 10.28 teraflops with 2200 processors.

    So, the Itaninum 2 delivered 4.8 gigaflops per processor, the G5 delivered 4.6 gigaflops per processor.

    This seems like a pretty poor showing for Itanium 2, overall. It's a much hotter chip than the Opteron or the G5, so cooling and power costs are likely much higher than a comparable apple cluster. The Xserve G5 is also likely cheaper than a similarly equipped Itanium 2 server, given that the Itanium 2 is $1398 per chip on Pricewatch, and a dual processor Xserve G5 cluster node is $2,999 list. Even with 4 cpus in a single box, I think the Itanium 2 server would easily top $6,000.

    But anyway, good game to Lawrence Livermore. I'll be curious to see if Apple has another volley to fire before the top500 list closes for this round.

    --
    - "When you want something with all your heart, the entire universe conspires to give it to you" -Paulo Coelho
    1. Re:apple's response will be interesting by Jeff+DeMaagd · · Score: 2, Interesting

      I love G5s, but Virginia Tech's cluster IMO can't say much until they get the G5 Xserves, because the PowerMac G5s don't have ECC memory. ECC is very important for such a large scale project that runs simulations where data is stored in RAM for any meaningful duration.

    2. Re:apple's response will be interesting by gerardrj · · Score: 2, Interesting

      There's also the difference in the interconnects, that has a lot to do with the efficiency of the system as a whole.
      Lets see what the VTech system does with ECC RAM installed when some node's aren't double-checking other node's results.

      --
      Article X: The powers not delegated... by the Constitution...are reserved...to the people
    3. Re:apple's response will be interesting by System.out.println() · · Score: 2, Interesting

      And don't forget that the current round of G5's are currently almost a year old... and long due for an upgrade. I hope some other instituion builds a 1,500 G5 2.6 GHz cluster :) (Or something to that effect.)

      --
      I've got more mod points and GMail invi
    4. Re:apple's response will be interesting by fmorgan · · Score: 2, Interesting

      I heard a presentation from VTech on why they selected the G5 over the Itanium (for scientific calculations, with lot of floating poing operations, both are faster than AMD chips; not a big problem for AMD, of course; how many of us need to simulate nuclear explosions in our desktops? well, at least until the next generation of strategy games, of course).

      At the time - this was a study done in July/Aug 2003, remember - the speed of the G5 and the Itanium2 were similar for the same clock speed (for scientific calculations; before someone flames me with something off topic, remember that this is a very particular kind of application); then what happened was that Intel was simply "out-clocked"! Kind of funny when Intel was the big champion of "clock-speed" over AMD, Motorala and IBM.

      This was in a presentation by VTech at an O'Reilly conference; coverage for this with several articles (including /. discussions) can be found here:
      http://www.macdevcenter.com/mac/osx2003/

    5. Re:apple's response will be interesting by prockcore · · Score: 4, Insightful

      This seems like a pretty poor showing for Itanium 2, overall.

      It does? You know that clustered computing doesn't scale linearly. If virginia tech were to double the amount of processors used, they wouldn't double their performance.

    6. Re:apple's response will be interesting by Anonymous Coward · · Score: 4, Insightful

      Actually, there's more to it than that. Virginia Tech's machine only gets ~55% of its peak performance, whereas Thunder gets 87%. Given that Thunder has twice as many processors, that's an EXCELLENT showing. Remember, the actual work that's going to run on Thunder won't scale anywhere near as well as the easily scaled LINPACK benchmark, so the performance gap between "benchmark" and "real world" will only get wider in practice.

      Thunder is an absolutely remarkable machine.

  20. Rejoicing at Intel by Animats · · Score: 4, Funny

    "We sold the Inaniums! We sold the Inaniums!"

  21. Second fastest on earth? by Stevyn · · Score: 2, Interesting

    yeah, that we know about. I remember the article on google a few weeks ago that made everyone think just what they hell they're running over there. I wouldn't be surprised if governments kept other supercomputing clusters secret. I don't mean anything tin-foil-hatish here, I'm just thinking that some governments have test facilities that they don't let the public know about.

  22. Heat by Rick+Zeman · · Score: 3, Funny

    4,096 Itanium 2 processors in 1,024 nodes

    So THAT'S what's causing our heat wave!

  23. Wow by 0xC0FFEE · · Score: 2, Informative

    Here's a picture: http://doc.quadrics.com/quadrics/QuadricsHome.nsf/ DisplayPages/3A912204F260613680256DD9005122C7

  24. Take that Apple! by callipygian-showsyst · · Score: 2, Funny

    Now you can't say you have the fastest "Thupercomputer" any more! You've been beat by Intel and Linux!

    --
    Best Buy can have you arrested
  25. I really hope by Lord+Kano · · Score: 3, Funny

    that they didn't build this just to win 2 grand from distributed.net.

    LK

    --
    "Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
  26. before everyone starts shouting at once... by painehope · · Score: 4, Insightful

    yes, they're hot as hell and eat power the way oprah eats twinkies, and yes Intel has made a poor handling of the Itanium line, but the Itanium architecture is very interesting, and is actually very appropriate for a HPC environment. Not the part of the HPC market that clusters dominate, but the segment that Cray, SGI, HP Alphaservers, etc. have traditionally dominated. The segment that doesn't give a shit about cooling, power consumption, or price-performance, but who just need to get the job done as quickly as possible.

    Some of the coolest features of the Itanium are also some of the reasons why a lot of people don't want to use it. The EPIC ISA, for example. It was designed ( along w/ the physical hardware ) to expose a lot of the internal workings of the processor to the user. But rather than recompile and re-optimize their code, people would rather bitch about migration. That's fine for workstations and servers, but in an HPC environment, you want the nifty features, you want to occasionally hand-tune code segments in assembler, etc.

    Anyways, I'm not a fanboy ( well, maybe an AMD and MIPS fanboy ), just wanted to get in a few honest points before everyone started shooting holes in the Itanic.

    --
    PC moderators can suck my White pierced, tattooed dick. If you think pride == hate, s/dick/Aryan meat mallet/g.
    1. Re:before everyone starts shouting at once... by slamb · · Score: 5, Informative
      Some of the coolest features of the Itanium are also some of the reasons why a lot of people don't want to use it. The EPIC ISA, for example. It was designed ( along w/ the physical hardware ) to expose a lot of the internal workings of the processor to the user. But rather than recompile and re-optimize their code, people would rather bitch about migration. That's fine for workstations and servers, but in an HPC environment, you want the nifty features, you want to occasionally hand-tune code segments in assembler, etc.

      I just coded some IA-64 assembly and from what I've seen, this comment is dead-on. They've got a lot of interesting features:

      • Speculation. The idea is to do memory fetches far in advantage to avoid waiting for the (much slower) memory system. You can do a LD.S operation that tells the machine something like "I might want the value from this memory address in a few instructions." It fetches it from memory, if it's in a good mood. If the address is paged out, it doesn't get it. (Instead, it sets a NaT (not a thing) bit to tell you nothing useful is there.) Later, you do a CHK.S. If it turns out that the speculative load fails, it jumps to some "recovery" code which gets it for real.
      • Lots of registers. 128 general-purpose 64-bit registers. Floating point registers. Some specialized ones, I think.
      • EPIC. (Explicitly Parallel Instruction Computing.) It has different types of instructions, aimed at different execution units. In the current incarnation, there are two sets of these in each processor. You give it bundles of three instructions, more broadly divided into groups. Instructions in a group don't depend on any earlier results calculated by the group, so they can be executed in parallel.
      • Rotating registers. This lets you make different iterations of the same loop work with different registers, to take advantage of EPIC more fully.
      • Predicated instructions. There are a bunch (16? 64? don't remember) of predicate bits, set by the CMP instruction and the like. Every instruction has an associated predicate. (p0 is hardcoded to true, so you normally don't notice.) So you can do conditional execution without jumping. More efficient, especially if it's just a few instructions that differ.

      If you just have a simple sequence of operations, each dependant on the one before, you can't really take advantage of these capabilities. (My code was like this. Even though performance wasn't my reason for writing assembly, it was a little disappointing that I couldn't play with the new toys.) If you're expecting these features to make Word start faster, you'll probably be disappointed.

      But if you're doing intensive computations in a tight loop, you can do amazing things. If you can get all the execution units working simultaneously, it will fly. And the features like rotating registers are designed to make that possible. You need a very good compiler or a very smart person to hand-tune it. You may need to recompile to tune if your memory latency changes (affecting how many iterations to run at once) or they come out with a new chip with more sets of execution units. But in a situation like this, none of that is a problem. They'll have applications designed to run as fast as possible on this machine. They may never be run anywhere else.

    2. Re:before everyone starts shouting at once... by ajp · · Score: 2, Interesting

      Speculation: very cool. Lots of registers, yes, but the Register Stack Engine makes it feel like you never run out: there are automagically more. EPIC: bundles are actually between 3 and 5 instructions. Rotating registers: most important for passing parameters. Predicated instructions? 128 of them.

      However your post, and my post, are wasted on /. These people will always hate Itanium because it's not kewl. But they're not the ones who will be buying the processors.

  27. What about SCO? by watsondk · · Score: 4, Funny


    do they have the nerve to go after this cluster?

    afterall they are trying extortion by lawyer against other large Linux users

  28. Sadly... by System.out.println() · · Score: 5, Funny

    "We sold the Inaniums! We sold the Inaniums!"

    "The Itaniums, however, remain unsold."

    *hopes that was not an actual mistake but rather a poorly conceived pun on "inane"...*

    --
    I've got more mod points and GMail invi
  29. Re:It's all about sticking it to the mac. by damiam · · Score: 2, Informative

    Sorry to burst your bubble, but Itanium isn't x86.

    --
    It's hard to be religious when certain people are never incinerated by bolts of lightning.
  30. Yes but... by goombah99 · · Score: 2, Funny

    Can it run WINE?

    --
    Some drink at the fountain of knowledge. Others just gargle.
  31. Big Iron? by nacturation · · Score: 4, Funny

    Thunder sports 4,096 Itanium 2 processors in 1,024 nodes, some big iron by any standard.

    If the government gets a hold of that, we're going to need some big tinfoil...

    --
    Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
  32. Nah, but who knows what the NSA has cooking. by tukkayoot · · Score: 2, Insightful
    I thought of Google too, but yeah since their network isn't built for sheer computational speed, I doubt it's anywhere near the fastest/most powerful system for many processing tasks.

    The NSA, on the other hand... I would guess that they have the most powerful cluster of machines in the world for breaking encryption. Though perhaps not as powerful as the article's supercomputer for other tasks.

    Plus there are undoubtedly several other highly classified supercomputers designed to chew on other problems.

    So it would seem that you'd have to caveat any claim of regarding the "fastest computer" by saying it's the fastest known, non-secret computer. But then the headline loses some of its appeal.

    1. Re:Nah, but who knows what the NSA has cooking. by DAldredge · · Score: 2, Interesting

      The NSA has their own small scale fab to make their own custom chips. Hell, they do not every try to keep it secret and have let more than one file crew film part of it.

      They can design all the custom chips they want with their rather large budget.

  33. Probably OT, but... by Trogre · · Score: 3, Interesting

    ... if you want a practically guided tour of LLNL, watch TRON sometime. They filmed it there (the science-lab live action stuff anyway).

    --
    "Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
  34. Nope. But they can't do what google does either. by Anonymous Coward · · Score: 3, Informative

    Google's cluster isn't a computational cluster.

    You have several types of clusters, each are designed to do a specific task, although you can easily mix-n-match for different purposes.

    1. Server clusters. Bunches of machines running together, providing services that compliment each other.

    For example you have a file server that is mirrored to another that is hooked up to a different part of a Lan/Wan backbone in order to improve service. Lot's of databases are clusters like this.

    2. High avaiblity clusters.

    You have a machines that are backups of other machines. If one machine fails a backup is activated instantly and replaces the failed machine without ANY loss in services.

    Sort of like a RAID harddrive setup. Hotswappable computers, that sort of thing.

    Google is the first 2 types. It has several clusters with nodes. Each node is made up of a few computers, if a node fails then another backup can back it up instantly, giving the techs time to correctly fix the issue. The computers each take some of the burden, too, so that it seems that they would have to be running mega-machines to provide the performance when in reality they just run a bunch of PC-style computers.

    3. Computational clusters. Clusters that are designed to pool their resources to create a single big computer that is used to proccess large amounts of data and intense mathmatical functions.

    2 types of these are Beowolf clusters and OpenMosix clusters.

    OpenMosix cluster is easy to setup if your a little bit familar with linux and even have knoppix cluster cdroms you can build ones quickly and easily.

    Beowolf is used for big number crunching and programs that use it are generally written to run a specific cluster, although libraries and tools are portable.

    Used lots in astromony for example. 10-12 PCs in a college lab can make a nice number crunching machine.

    There are some clusters that do all 3, lots can do only 1 or 2 of the types easily. Different types can compliment each other.

  35. Itanium vs Opteron by vlad_petric · · Score: 4, Insightful
    Itanium's instruction set is actually a lot more geared towards scientific computing than server benchmarks. Scientific stuff usually is made of very regular code, that is quite easily schedullable by the compiler. Server stuff is generally memory-bound and very irregular, so the processor usually gets less than one instruction executed per cycle - bundling instructions (static schedulling by the compiler) is completely pointless.

    "Big Iron" is a very vague term - server benchmarks behave very differently than scientific computation as far as performance is concerned; if you don't believe me I can easily point you to a couple of research papers analyzing them.

    The humongous on-die caches makes the Itanium perform well on servers, and definitely not the instruction-set architecture. So "WAS DESIGNED FOR" is only 50% true.

    --

    The Raven

  36. Linux support by linuxguy · · Score: 2, Insightful

    Intel provides excellent Linux support for Itanium. Also if you use the Intel compiler, which Lawrence Livermore does, you get considerable speed boost on Intel CPUs.

    See: http://www.llnl.gov/linux/linux_basics.html#compil ers

    Intel can afford to provide little niceties like this. Can AMD? I doubt it.

  37. Re: A Better Way To See It by value_added · · Score: 2, Insightful

    Ed Note: Unless the author wishes to narrow his/her audience to a small subset of Slashdot users, standard formatting and non-cutesy sentence case is always appropriate.

    There are basically three type of clusters:

    1. Shared Nothing: In this, each computer is only connected to each other via simple IP network: no disks are shared. and each machine serves part of data. These cluster doesn't work reliably when you have to aggregations. For example, if one of the machine fails and you try to to "avg()" and if the data is spread across machines, the query would fail, since one of the machine is not available. Most enterprise apps cannot work in this config without degradation. For example, IBM study showed that 2 node cluster is slower and less reliable than 1 node system when running SAP IBM on windows and unix and MS uses this type of clustering (also called federated database approach or shared nothing approach).

    2. Shared Disk Between Two Computers: In this case, there are multiple machines and multiple disks. Each disk is at least connected to two computers. If one of the computer fails, other takes over. no mainstream database uses this mode, but it is used by hp-nonstop. Still, each machine serves up part of the data and hence standard enterprise apps like SAP etc cannot take clustering advantage without lot of modification.

    3. Shared Everything: In this, each disk is connected to all the machines in the cluster. Any number of machines can fail and yet the system would keep running as long as at least one machine is up. This is used by Oracle. All the machine sees all the data. Standard apps like SAP etc can be run in this kind of configs with minor modification or no modification at all. This method is also used by IBM in their mainframe database (which outsells their Windows and Unix database by huge margin).

    Most enterprise apps are deployed in this type of cluster configuration. The approach one is simpler from hardware point of view. Also, for database kernel writers, this is the easiest to implement. However, the user would need to break up data judiciously and spread across machines. Also adding a node and removing a node will require re-partitioning of data. Mostly only custom apps which are fully aware of your partitioning etc will be able to take advantage.

    It is also easy to make it scale for simple custom app and so most of TPC-C benchmarks are published in this configuration. Approach 3 requires special shared disk system. The database implementation is very complex. The kernel writers have to worry about two computers simultaneously accessing disks or overwriting each others data etc. This is the thing that Oracle is pushing across all platforms and IBM is pushing for its mainframes. Approach 2 is similar to approach 1 except that it adds redundancy and hence is more reliable.

    So what type are we talking about here?

  38. But... by Trejkaz · · Score: 2, Funny

    But Windows only has 50% less TCO if your time is worthless.

    --
    Karma: It's all a bunch of tree-huggin' hippy crap!
  39. Of course... by MaestroSartori · · Score: 2, Interesting

    ...you realise that it isn't a linear scale. Trying to make a G5 cluster which achieved 4.8 gigaflops per processor would take more than the 4400 processors, and thus would easily take more than 300 more processors than are used for the Itanium cluster.

    300 processors. Thats 150 dual-processor boxes. I can't be bothered working it out now, but how far that goes to eliminating the power & heat advantage the G5 has would be interesting to find out...

    --
    Game dev and music blog
  40. Re:LLNL's usefulness by slamb · · Score: 2, Informative
    Why no hydrogen cars? Well, it could have something to do with hydrogen being a net-loss fuel; it takes more energy to make than it provides.

    That's thermodynamics. It's true for any fuel. It's even true for oil and nuclear energy - the difference being only that the energy wasn't put in during our lifetime. (And in the case of nuclear, that the pre-existing energy is all but inexhaustible.)

  41. Not any more! by brodin · · Score: 2

    >It's all for reserved for Doom III on longhorn.
    Sorry, I'd played Doom III yesterday at E3. That's joke is (in your best Iron Chef voice) o-vah!

  42. can't believe nobody mentioned favourite vendor! by nighty5 · · Score: 2, Funny

    $2,863,104 in license fees going SCO's way!

    I can see the investors now rubbing their 2 cents together....

  43. Top500 list not updated yet by mrjb · · Score: 2, Insightful

    This is the official top 500 list of supercomputers (not updated yet although thunder is mentioned as '*possibly* the second-most powerful computing machine on the planet'). Linux moving up to second place (from fifth a bit ago, iirc), woohoo! Only one left to beat!

    --
    Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
  44. Your forgetting by IAmAMacOSXAddict · · Score: 2, Interesting

    It also has twice the processors, to generate the X2 times speed that they claim. Something tells me, now that VA is recieving the XServer G5 cluster nodes, that they may want to add some more units. they can put 48 units in each rack now, rather than 12 of the full size G5 Desktop form factor. According top my primitive calculations that would allow them to run 4 times as many machines in the same space (would be over 8000 CPUs. I figure that will likely kick the crap out of this new linux cluster...

    --
    MacOSX, because making *NIX better is a lot better than waiting for Micro$loth to fix Windows