Slashdot Mirror
Cray CTO: Linux clusters don't play in HPC
jagger writes "Linux clustering was touted as the next big thing by many vendors last week at ClusterWorld Conference & Expo 2004. But supercomputer vendor Cray Inc. scoffed at the notion of putting Linux clusters in the high-performance computing (HPC) category. "Despite assertions made by Linux vendors, a Linux cluster is not a high performance computer," said Dr. Paul Terry, CTO of Cray Canada."
While Paul Terry makes some good points, in his statements, including the partial quote from the post, "Despite assertions made by Linux vendors, a Linux cluster is not a high performance computer, said Dr. Paul Terry, CTO of Cray Canada. "At best, clusters are a loose collection of unmanaged, individual, microprocessor-based computers."
Remember to take this with a grain of salt. The inflammatory nature of the comment is nothing more than a marketing ploy to increase visibility of, and sell, the new Cray XD1
Kinetic stupidity has a new brand leader: Allen Zadr.
A quote I've seen before:
"If you were plowing a field, which would you rather use? Two strong oxen or 1024 chickens?"
Maybe he meant penguins?
Nuke Gay Whales for Jesus.
Company officer claims competitor isn't as good as his product. Film at 11.
Oracle disclaim MySQL and PostgreSQL as "toy databases", Microsoft claims that "Apache cannot be used for real web serving", and Sun announces that "Intel and Linux simply cannot be used for enterprise computing".
So all those supercomputing labs that use Linux clustering (that invented Linux clustering, even) have been wasting their time?
Ceci n'est pas une signature
I guess that the simple problem is just that the algorithm applied is usually not suitable for massively parallel computing.
Regardless of whether I agree with the article or not I feel compelled to point out that:
The 1100 node Apple G5 cluster in virginia has yet to run any real scientific code. So far it has only ran benchmarks.
In other news...
"Despite assertions made by Toyota salesmen, a Lexus sedan is not a luxury car," said Bill Taylor, CEO of Mercedes-Benz.
This space intentionally left blank.
Clusters can get high performance on some types of tasks. But sometimes, you need fine-grained parallelism that just isn't available on a cluster.
On the other hand, high performance usually comes through special hardware. And on that hardware, I think Linux could be the right thing (modulo some patches).
A message from the system administrator: 'I've upped my priority. Now up yours.'
It all depends on the problem you are trying to solve. I have been doing some work of late that would not complete in my life time on the 108 node cluster that we have. But when programmed for and run on two Cray X1s I should complete inside of a week.
Granted there are many codes (and more every day) that will run on clusters, the big iron will never die.
Just because we love Lunux doesn't mean that clusters are HPCs.
There are real issues that differentiate mainframe/supercomputers from large, powerful, clusters.
Of course this all depends on your definition of an HPC. But I believe that it's reasonable to say that if parts of your computer are connected with low bandwidth connections (10/100,gigabit) they just can't handle the same kinds of transactions that a computer with parts that are connected by 10 gigabit or 1000 gigabit connections or whatever it is nowadays.
As far as I know if you're deploying a large database it's still advisable to have a big huge IBM mainframe or a Unisys box or a Sun 10k instead of 4,8 or 16 clustered 8 proc machines.
My point is there are valid arguments for not including clusters of commodity hardware in the HPC category.
In my mind they aren't High Performance Computers... they are High Performance Clusters of Commodity Computers.
~foooo
How well a cluster will do depends on the application that it is performing. Some problems can be divided into several small problems with little reliance on other parts of the problem (SETI / Encryption breaking). These things can be easily distributed to hundreds or thousands of "small" boxes for processing and are what a beowulf cluster would be good at.
Other applications require the breakneck interconnect speeds that large Cray / Sun / etc.. build on. When the data being calculated on one CPU requires data from CPU2 to continue its calculations you don't want to have it wait for 100mbit or even 1gbit ethernet speeds. Even quicker interconnects such as SCALI are going to be slowed by PC bus speeds.
Cray fills an important niche for those who can afford it.
The comment was stupid, yes, but not all jobs that you'd use supercomputers for can be broken down into many threads as others can. A linux cluster will do well for some jobs, a cray box will do well for others. There *will* be times when a Cray system is so far superior to anything you could do with Linux that it becomes the only real option.
However, dismissing linux cluster technology automatically is dumb. In many cases, it provides more than enough cpu power and I/O bandwith to support your reason for getting a supercomputer, and probably at less cost than the other options.
Its all a matter of determining what you need the computer to do, determining your budget, and get the best system in your budget for the uses you have for it. Sometimes that will be a Cray, sometimes a Linux cluster.
Who is this guy and what does a company like Cray know about... oh... never mind.
Clusters can rival a supercomupter when they are assigned is a task that's suitable for distributed computing. That is, work units can be divided up and worked on in any sequence... the result of segment 45 doesn't depend on knowing the result of 44 and such. Effectively, you can have the sum of all of the processors minus just a little overhead for the clustering.
What Cray's rightfully pointing out is that for most business applications, however, distributed computing is not a viable option. When processing on a transaction basis, the transactions often need to posted in the exact order they were recieved, which means they must be taken serially. In those situations, the programs can't multithread work out to the other processors so well, and the cluster will end up running at roughly the speed of just one processor while the others waste clock cycles waiting for something to do.
The cluster isn't the solution to everything. Nor is the supercomputer. You've gotta think about the job, then figure out which tool is right for the task.
In truth, such machine will always have a certain performance advantage over traditional clusters. The question is, will the price point be low enough to invalidate the idea of just adding more boxes to the traditional cluster.
"Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
While I certainly disagree that you can't build a very high performance computer out a cluster of computers (Linux or otherwise), there is a lot of merit to the fact that clusters just don't scale well for certain classes of applications. Hence the renaissance of the vector supercomputer (ala the Earth Simulator ).
Obviously, this guy is plugging the new Cray X1 architecture, which really is quite promising. For instance, check out this paper by some folks at Oak Ridge National Lab that appeared in Supercomputing 2003.
Of course, since this is Slashdot, I expect that there will be a deluge of posts decrying everything about the new Cray machine because it commits the cardinal sin of NOT USING LINUX. Oh, the horror!
You're right in saying that the Virgina Tech cluster is the 3rd fastest supercomputer (LINPACK tests). I think that for some other tasks however, it would be slower. Sure, they use infiniband as an interconnect (very fast & low latency), but that doesn't change the fact that it's many separate nodes, each with its own memory. So if one processor were to access some memory on a different node, it would slow down things a little.
So depending on the task at hand, the cluster might perform very well, or perhaps a little less well. Cray supercomputers are a big number of processors all in the same machine, and more importantly all sharing the same memory. Each processor has the same delay to access any memory content.
The argument in favor of clusters, however, is that it's still cheaper to throw more computers in than to buy a Cray that would perform the same task in less time.
In the end, there's a lot of marketing involved in all of this...
Hope this helps (and that I'm not completely wrong!),
Maan
I happen to work in a facility that has large had both large supercomputers (cray t3e, j90, sgi) and linux and *nix based clusters (beowulf/linux, compaq/Tru64). The Cray CTO is correct that you can't just call every linux cluster out there HPC. Just about anyone with networking and linux knowledge can build a linux cluster.
What really makes a difference between an HPC cluster and your normal every day cluster is the hardware interconnects used. There is a comment in the artical that refers to not using I/O for memory and message passing. I am not quite sure what he means by that, but I am guessing that he is saying that the network is not used for shared memory/message passing (MPI/openMP/SHMEM).
If a cluster can limit the impact of latency between nodes either through smarter software or faster interconnects then I can't see any reason not to concider a linux cluster as HPC.
Clusters without smarter software tend to be a real difficult coding platforms. Some developments with things like globally shared memory might make the difference, but there will still be the problem of latency between nodes.
Oh, here's the TOP500 list, btw.
So depending on the task at hand, the cluster might perform very well, or perhaps a little less well.
Surely what you meant to say is that, depending on the task at hand, a cluster might perform very well, or perhaps perform attrociously. :-)
Clusters tend to work well when the various nodes don't need to communicate very often but you need lots of cycles for the subtasks, while dedicated supercomputers tend to perform very well in tasks requiring vast amounts of internode communications bandwidth along with large numbers of cycles. If you need vast bandwidth and relatively low numbers of cycles, your pricepoint is likely a mainframe. And if you don't need either, you get a cheap desktop machine.
Certain problems parallelize well on a cluster ... others don't. Some don't parallelize at all, and a cluster won't do you a darn bit of good. The different machines are designed for different uses ... and one should be careful not to push a "one size fits all" solution. The Cray guy clearly got it wrong on that point, and likely knows it, but he was marketting, not teaching a course in choosing hardware for the task at hand.
The reason that Cray only holds 19th right now is because they have only deployed X1 systems using up to 256 nodes. When the number of nodes is increased, you will certainly see the Cray moving up the top 500 list -- the architecture is VERY scalable.
The how to from way back in the day.
e r- formats/html_single/Beowulf-HOWTO.html
http://www.ibiblio.org/pub/Linux/docs/HOWTO/oth
has a great explanation using a grocery story analogy that makes it really easy to understand what kind of tasks will work well and what kind will suck. And unlike the cheerleaders that have been showing up since clusters became a big business is very balanced about it.
Still worth reading.
Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
Cray could easily be at or close to the top of the top500 list, their X1 architecture will extend that far. However, for a lot of really important supercomputing codes, it's no contest: The cray will trounce the clusters (linux or otherwise). Those #19 crays are only 256 processors. To get similar performance a stack of xeons requires thousands of processors. Some tasks just can be split appart that easily.
A cray processor has eight floating-point units running at 800Mhz. The big Mac cluster (for example) uses G5 processors which have 2 FPUs at 2000Mhz. Thus the cray has a ~40% advantage. However, the G5 processor has ~4GB/s memory bandwidth. The Cray has ~50GB/s memory bandwidth. If you have a problem that needs to do a HUGE amount of math on a tiny amount of data, the G5 will rock. If you have a problem that needs to do a HUGE amount math on a GINORMOUS amount of data, buy the cray. (for a GINORMOUS amount of money too)
Similaraly infiniband (ala the big mac) is really hot in the cluster interconnect space because it gives 2.5GB/s per node. The Cray gives you 51GB/s.
You need to move a little data, buy a cluster. You need to move a lot of data, buy the Cray.
There's no one solution for all problems.
"No, a Kilochicken is a 1000 chickens. You're thinking of a kibichicken. Check it out at http://www.nist.gov" Somebody had to, right? Right?
doesn't this CTO of cray remind you of someone?
"There IS no Linux in high-performance clusters."
"There IS no Americans in Iraq."
OMG! It's the former Iraqi mis-Informed-ation minister!
Especially when 2004 has been dubbed the year of the penguin, it's wreckless to claim that Linux can't be used in HPC's.
Hell, just look at the current top500 list. There's no Cray in the top 10 but there are two Linux based clusters there (and one based on OSX [FreeBSB based]).
Here's a few:
NCSA's IA32 Linux cluster
NCSA's IA32 Linux cluster
Space Simulator Clust at Los Alamos (SS51G based; makes me proud as I have a SS51G too)
Beowulf - used in many Linux clustering projects
Linux clusters at Los Alamos (they seem to have more than one)
Virginia Tech's Supercomputer X
For the love of Christ people, it's a simple thing.
Format links like this: <a href="http://somelink">link text</a>
It takes virtually no extra time and we don't have to trim the fucking slashcode spaces.
Oh, and here's the link.
LOAD "SIG",8,1
LOADING...
READY.
RUN