Slashdot Mirror

← Back to Stories (view on slashdot.org)

On the Supercomputer Technology Crisis

Posted by simoniker on from the superprocessor-to-the-rescue dept.

scoobrs writes "Experts claim America has been eating our 'supercomputer feed corn' by developing clusters rather than new supercomputer processors and interconnects. Forbes says America is playing catch-up and that the new federal budget items are too little too late. Cray is laying people off due to decreased federal spending and claims lower margin products have forced them to create products based on commodity parts. Red Storm, one of their new Linux-based products, is being delayed to next year."

13 of 347 comments (clear)

  1. it makes sense by dncsky1530 · · Score: 4, Insightful

    when you can build a top 5 supercomputer for under 6 million dollars, using off the shelf parts. Why spend the hundreds of millions of dollars?

    1. Re:it makes sense by Otter · · Score: 4, Insightful
      If you RTFA, an administration panel on high-end computing claims that clusters are inappropriate for certain tasks. I don't necessarily trust the claims of what I assume is an industry-heavy panel, but then I don't necessarily trust the supercomputing expertise of a bunch of Lunix fanboys "administering a network" in their parents' basement either.

      My inclination is to let the market sort itself out, although if supercomputer makers go under, they won't necessarily reappear the moment they're needed.

      --
      What I'm listening to now on Pandora...
    2. Re:it makes sense by badboy_tw2002 · · Score: 5, Insightful

      Then trust the fact that not all problems are easily attacked from a parallel perspective. This means problems where working on one section of the dataset affects large amounts of data in other sections. There's a lot of locking and waiting for tasks in other parts of the system to be completed; and a lot of data transfer/need for shared memory, which if you're bussing between cluster components, its going to be slow.

      This doesn't mean that clusters don't have some use in these regards, it just means that for these types of problems no one has figured out an efficient parallel algorithm to use on them.

    3. Re:it makes sense by ch-chuck · · Score: 4, Insightful

      because, sometimes you need two strong oxen instead of 10240 chicken.

      --
      try { do() || do_not(); } catch (JediException err) { yoda(err); }
    4. Re:it makes sense by grawk · · Score: 5, Insightful

      As someone who works for a supercomping center, I can say that some things work VERY well on cheap unix based clusters. I am the primary admin on a 5 TFLOP cluster. We've also got a Cray X1, and while it's only 2.6 TFLOPs, it will eat my IBM's lunch when it comes to some specificly tuned tasks. Much in the same way that we can outperform mac clusters that have significantly higher floating point performance because of the speeds of the interconnects. Supercomputing is about a LOT more than just raw CPU power.

  2. Expected fallout from the Beowulf takeover by beee · · Score: 4, Insightful

    This is an expected and predicted fallout from the recent rise in popularity of beowulf clusters. Slowly but surely managers are realizing, yes, it is possible to have a supercomputer on mass-market hardware, running a free OS.

    Don't see this as bad news... it's a sign that we're winning.

    --


    + Donald Gunth
    + Email: dgunth@quicktek.net
    "Caffeine is the greatest lubricant ever created." -ESR
    1. Re:Expected fallout from the Beowulf takeover by susano_otter · · Score: 4, Insightful

      Since clusters are so much cheaper than mainframes, it's often the case that clusters still offer better performance for the money spent than a mainframe would, even if the cluster isn't really optimized the way the mainframe is, for the task at hand.

      That being the case, wouldn't it make more sense to invest heavily in R&D to solve the cluster's problems and remove its limitations, than to invest heavily in R&D into next-gen mainframes?

      --

      Any sufficiently well-organized community is indistinguishable from Government.

  3. Law of Diminishing Returns by Billobob · · Score: 4, Insightful
    It appears to me as if we have reached the point where supercomputers aren't really as practical as they were before. Fewer and fewer industries need and prefer supercomputers to a cluster of cheap PCs, and the market is simply heading in that direction - nothing really unique happening here other than capitalism.

    Of course people are going to cry that companies like Cray are falling by the wayside, but the truth is that their services simply aren't as needed as they were in years past.

    --
    If you have to ask, you'll never know.
  4. "Feed' Corn? by jmckinney · · Score: 4, Insightful

    I think that should have been "Seed Corn."

  5. Expert complains: by Anonymous Coward · · Score: 4, Insightful

    Free market sucess might lead to us actually having to pay for our own supercomputer research that we use in profit making ventures.

  6. You are all missing the point by Anonymous Coward · · Score: 5, Insightful

    Its the fact that clusters require higher skill to program efficiently for than do single processor systems. Plus you have all of the wasted processing power used for communication between the nodes. Granted, many problems lend themselves well to distributed computing (essentially what a cluster is, but the nodes are closer and communicate faster), but there are also problems that are handled better by a smaller amount of specialized hardware. The other point is that by using off the shelf parts, we are not really innovating in this space like we should be. We are allowing the commodity computer market determine the direction of the supercomputer market.

  7. Without a market you can't survive long term by wintermute42 · · Score: 4, Insightful

    There seems to be some historical revisionism going on regarding the demise of the "supercomputer industry". People are coming out of the woodwork now saying that lack of government support caused the great supercomputer die off.

    As Eugene Brooks predicted in his paper Attack of the Killer Micros, the supercomputer dieoff was caused by the increasing performance of microprocessor based systems. Many of us now own what used to be called supercomputers (e.g., 3GHz Pentinum processors, capable of hundreds of megaFLOPs).

    The problem with supercomputers is that high performance codes must be specially designed for the supercomputer. This is very expensive. As people were able to fill their needs with high performance microprocessors they quit buying supercomputers.

    Many people who need supercomputer levels of performance for specialized applications (e.g., rendering Finding Nemo or The Lord of the Rings) are able to use walls of processors or clusters.

    There are, of course, groups where putting together off-the-shelf supercomputers will not suffice. But these groups are few and far between. As far as I can tell they consist of the government and a few corporations doing complex simulations. The problem is that this is not much of a market. Even if the government funds computer and interconnect architectural research, there does not seem to be a market to sustain the fruits of this research.

    In the heyday of supercomputers there were those who argued that when cheap supercomptuers were available the market would develop. The problem is, again, programming. High performance supercomputer codes tend to be specialized for the architecture. Also, no supercomputer architecture is equally efficient for all applications. It is difficult to build a supercompter that is good at doing fluid flow calculations for Boeing and VLSI netlist simulation for Intel (the first applications tends to be SIMD, the second, MIMD). The end result of these problems tends to suppress any emerging supercomptuer market.

    The reality right now seems to be that those who are doing massive computation must build specialized systems and throw a lot of talent into developing specialized codes.

  8. What tasks require high-speed interconnects? by Bruce+Perens · · Score: 5, Insightful
    One of the nice things about clusters is that they encourage people to consider how to decompose a problem so that it can work without a large high-speed shared data memory. Some of the older supercomputers were important because scientists hadn't done this work because there wasn't the economic incentive back then. Now there is one.

    So, what tasks still require a high-speed shared data memory? Answer that, and you'll understand where you can still sell a supercomputer.

    Bruce

    --
    Bruce Perens.