Slashdot Mirror
10-TFlop Computer Built from Standard PC Parts
OrangeTide writes "Using PCI host adapters and Xeon processors, engineers at Lawrence Livermore National Labs have achieved 10-TFlops relatively cheaply. More information can be obtained from this article at EETimes." Lately, Linux seems to be the operating system of choice for new supercomputers, and this one's no different. It's cool to see big iron made cheaply.
A commodity supercomputing cluster of these! (There has to be a better name for it, but I'm new here on Slashdot).
... which was specifically developed for running Doom III.
Like Teddy with an elephant gun.
As long as it doesn't see a full moon, then it won't turn into...
Beowulf!
zzziiiippp!!!!(I love the smell of nomex in the morning!)
I love the PCI bus. I think its one of the coolest things ever. T_RDY anyone?
>The 1- to 10-teraflops processing range is opening up a revolutionary capability for scientific applications
In the not too distant future, that kind of processing power could very well be available in home PCs. Imagine what that would do to...well, I mean, dang it, what the heck will we do? Game frame rates can only go so high. Even realism of 3D graphics may have it's limits. Oh sure, we'll find something, but it's difficult for us to imagine now...
---
Open Source Shirts
Were I work (I can't say where... I've signed papers...) we have replaced an SGI 'super computer' (or mini computer, whatever, a big number crunching beast of silicon!) with a Beowulf cluster. This not only gives us great scaleability, but also lots of FLOPS per dollar (or rather, krona :^P).
Then the world will finally see the 4000 Playstation 2's that Saddam used to build a supercomputer
Look, all the 'cool' people are doing Linux, right? But BIG IRON is clearly trying to suck you and your money in. Oh, it's so cheap, but they don't tell you about the hidden costs, do they?
Just say no to BIG IRON!
[o]_O
can be found here.
So the Teraflops they're mentioning are just a theoretical upper bound, don't get too aroused when you see it.
The Raven.
The Raven
Using PCI host adapters and Xeon processors, engineers at Lawrence Livermore National Labs have achieved 10-TFlops relatively cheaply.
Imagine a Beowulf cluster of those! Er... Nevermind.
"I'd rather have a full bottle in front of me than a full frontal lobotomy"
The important part isn't the number of FLOPS (to get those you can just keep buying more PCs until you reach the desired number) but the performance in applications which are not 'embarassingly parallel'. In other words how good is the interconnect between machines? The article talks about a new network to replace Gigabit Ethernet.
-- Ed Avis ed@membled.com
The system has a few unique features that the lab says will facilitate applications performance, including a fast, custom-made network that taps into an enterprisewide file system.
"This network approach is nice because we can use a standard PCI slot on each processor node, which gives a 4.5-microsecond latency," he said, as opposed to 90-s latency for Gigabit Ethernet."
The boards are linked by a network assembled by Linux Networx into a clustered system that will have 960 server nodes.
The file system, called Lustre, uses a client/server model. Large, fast RAM-based memory systems support a metadata center, and data is represented across the enterprise in the form of object-storage targets. "Being able to share data across the enterprise is an exciting new capability
I think this is especially interesting, because it seems to glue together pieces from traditional clustering and distribted or metacomputing. Is there some site for this project with more details?
...that Apple's glorious supercomputers are obsolete?
:/
Damn...
Extraordinary Vacations. Exceptional Prices
Extraordinary Vacations. Exceptional Prices
So please explain this. I mean, I have two linux boxes in my room and each has a free PCI slot. What do I need to to to network them over directly over PCI?
"This type of capability elevates computer simulation to the same level as physical experiment and theory, so it is going to allow us to do groundbreaking scientific work,"
;*)
Yes, why don't they stop doing real physical experiments and just use this Linux-cluster instead? As we all know here, Linux is far better than the reality
(MS-Certs: Admit it! You know Linux is, you're just being stubborn!!)
I have a lot of movies to convert to DIVX...
The boards are linked by a network assembled by Linux Networx into a clustered system that will have 960 server nodes. - Do you call this relatively cheap?? it sure doesn't sound very cheap.
Hopefully the Cell will soon be out on the market. A cluster with 10 computers using the Cell will overrun this, and it would also be some hundred thousand dollars cheaper =)
I'LL SEE YOU IN METAMOD!!!
"Lately, Linux seems to be the operating system of choice for new supercomputers, and this one's no different. It's cool to see big iron made cheaply."
That's because nobody has the money to invest in a more expensive solution!
I told you my other Boxen was a 1000 node beowulf cluster... But no one believed my sticker...
Sigs? We don't need no stinking sigs!
Great, now I can do my spreadsheets and word processing way faster than before... :)
But I sure as hell won't!
*Sets Zayin on fire.*
Tell me about Beowulfs NOW!
;)
Chas - The one, the only.
THANK GOD!!!
LLNL offered me an interview but I turned it down because I already had one scheduled in redmond for the same day. Ah such is life.
"There is no reason anyone would want a computer in their home." - Ken Olson, President, chairman and founder of Digital Equipment Corporation, 1977.
Anyways, what I'm trying to point out is that it is actually becoming very convinient to build a super computer with lots of PCs that just lie idle. I am not sure if Saddam has heard about cheap linux systems. But what if he could build a super computer cluster?
Boy this gets interesting and scarier at the same time.
Uuh, I mean null-card connection. I have never really looked at the PCI spec from an eletrical engineer standpoint, but there are probably power leads, data leads, timing leads, and ground leads on there.
The data leads should be easy...TX to RX. Although they may use a full-duplex lead where the data shares the bus based on clock pulses.
The power could be dropped, as both machines already have the proper power requirements. The ground leads could be tied together if you wanted, but dropping them shouldn't have too much impact on the final outcome.
The tricky part would be the clock pulses. In order to keep the data integrity, you need to have both bachines on the same clock. The easy way would be to take the crystal from one motherboard and wire it to the other. Same crystal, same clock pulse.
Then drivers would be needed to make the other computer look like an attached device. Shouldn't be too difficult. Just take a NIC driver and modify it...heavily.
I think an easier option would be to share data across the IDE bus. Make an IDE driver look like a NIC driver and send IP across IDE. In fact, I remember Linux Journal publishing an article about someone doing IP over SCSI about 2 years ago. Get some SCSI cards and make your own version of a CDDI network ring.
I'd rather you do it wrong, than for me to have to do it at all.
They put tux on amphetamine?
I guess the heroine billg has been taking has finally rotted his brain.
If you would supply me with contact information, maybe we could discuss this. I feel that moderators should not doubt he facts supplied by the submitters, but rather value the information as if it is true.
I think that it is nice to see moderators yelling, flaming and behaving generally bad (yes, I mean that you do that). Tell me who you are and we can have a discussion concerning me working or not.
As for your comments about my employment. I have not said that the company I work for is "Extremely Important(tm)", I just said that we have needs for lagre computations and have invested in a Linux-based system. I am not " karma whoring by spouting bullshit", if you do not like my comments, simply mark me as a foe, and add -2 to my comments, do not mod me down simply because you have problems trusting people!
So, you put together N processors worth M mflops each and focus only on suitably parallizable computations so then you can claim M*N performance. Yeah? Really? So what?
The average geek is an idiot.
This just in...a computer has just been built with...and you won't believe this: computer parts!
Slashdot.org is there!
I'll form my OWN solar system! With blackjack! And hookers!
Who needs something like 10 Teraflops? I mean seriously.. If you are a dabbler and want to have that, then go ahead, but any serious party would not build their own computer if they need superpowers, they would buy it from professionals like IBM.
It's ten times cheaper, but still roughly a hundred times more expensive than most people can afford.
What is most hilarious about all this is that three years ago the same people at the Lab who put together this cluster, Livermore Computing, insisted that Linux was a toy....That it had no future in scientific computing....That it was a hobbyist's OS.
I sure hope they love the taste of crow....
"I was doing my nuclear simulations on the ASCII White and it was like BEEP BEEP BEEP...and like half my work was gone..."
"I'm tired of all this 'Aren't humanity great' bullshit. We're a virus with shoes" - Bill Hicks
I'm sure they have a hidden 2 TB directory somewhere full of p0rn videos.
Is Saddam such a bad guy if he's using linux and clustering ........... MMMMMmmmmmmmmmmmmmm that's a hard one!
Like to see them try to get a win on this.
Does any /. reader have any info on this? Is this a network / distributed filesystem? Why did they choose to write a new filesystem rather than pick from any of the existing filesystems out there? More importantly, is this code publicly available?
There is no such thing as luck. Luck is nothing but an absence of bad luck.
This is not a troll. I really want to know. Is it again because "no-one has ever been fired for buying Intel"?
This is such excellent news! Soon all the custom big-iron builders like Sun, IBM, HP and SGI will be out of business and the world will be ruled by x86 machines and their derivatives, and all we'll be left with is clusters of intel and AMD machines. All those pesky 64-bit, complicated, incompatible, proprietory RISC architectures will go away, and their single system image multi-CPU boxen and chewing-gum-and-string home-made clusters will rule! God, I love progress :-) Oh, and all those high-performance compilers and other dev tools will disappear to, and all those highly-evolved UNIX derivatives.
Anyone have any experience using (Open)MOSIX? I have a partially CPU-bound application (automatic part is IO-bound, manual part is CPU-bound) in Perl, Apache and MySQL. Anyone got experience with this stuff?
:)
For those who don't even know what MOSIX is, it is a kernel patch that essentially creates a virtual computer out of several boxes. They claim they will scale your application as long as you have multiple processes (they migrate them as needed) - without any coding on your part.
Since I'm looking for extra performance with limited resources, this looks like a potentially easy way out
Stop the brainwash
My porn is realistic enough, now it needs to be sorted, weeded for duplicates, and cataloged. And unfortunately there's too much for a human to do. Come to think of it, maybe the computer could appreciate it too. What was that movie again? Hardware?
--Jimmy has fancy plans; and pants to match.
The title says it all. Big Iron is _engineered_. No matter how big or how spiffy a Beowulf cluster is, it's still just a bunch of PC motherboards kludged together with a bunch of network cards. There is a reason Crays are expensive - they are _worth it_ from a performance standpoint, because not every problem lends itself easily to the solution of a Beowulf cluster. Some problems require the exchange of a lot of data between a lot of nodes, and a little math will show that it won't take much data interchange to saturate even a GigE switch. Adding more machines is not going to help; craftily designing and overengineering the network _might_, but by the time you get this whole damned thing glued together well enough to approximate a Cray's performance, you'll have spent enough to have just flat-out bought a Cray in the first place.
As others have noted, while this thing may have a theoretical peak performance of 10 TFLOPS, I'm willing to bet that number goes down like Monica Lewinsky on Quaaludes when you feed this magical supercomputer a problem that's _not_ suitable for distributed.net (i.e. one where computations on one node are dependent on computations on another node, like fluid-dynamics problems, turbulence, etc.)
Yeah, it's interesting as a curiosity, but this is by no means spectacular. Beowulf is good for what it's good for, which is a "poor-man's supercomputer" that works well for coarsely-parallel problems that don't require a lot of internode communication. It's not the Philosopher's Stone, folks.
-SD
I am Chaos. I am alive, and I tell you that you are Free. -Eris
Now that "Beowulf Clusters" are commonplace, heck, even Walmart sells them now, isn't this just a bit sad?
Far better would have been to use a couple of million cheap XT's (heck people pay you to take them away) and build a "Beowolf Cluster" which would smoke this pathetic excuse for an abacus into the stratosphere!!
I thought this was a much more interesting story Quantum encryption http://www.eet.com/at/news/OEG20021111S0036 "Supposedly at the stage of being rolled out For its part, MagiQ says that its Navajo system is currently at the alpha stage and promises real beta sites on selected campuses in the United States in the first quarter. Both companies are also talking about secure through-the-air communications with satellites. "
The Singularity is closer than you think
Quant
You'd think the EETimes would catch something like this:
nearly the same performance as the ASCII White system
No, it's ASCI White. Accelerated Strategic Computing Initiative, not the text format.
Which means there's a 4 GB hard limit on the amount of RAM a process can use, and a big performance hit if a node has more than 4 GB RAM. Of course with 10 TB, there's some room to spare, but if a calculation is not very parallelizable, you're still limited to the speed of one node.
The distributed memory Crays (T3D, T3E) are just the same: boards and network cards. The processors they use are not faster than the last generation PC processors. The difference are the NICs, that have about 10 times more bandwith and 10 times less latency (compared with standard fast ethernet cards).
There is the difference. As you say, for certain problems, this means that the whole machine is about 10 times faster than a Beowulf.
However, if/when conventional NICs are fast enough, specially in terms of latency, both systems can be equivalent again. In the meantime, a lot of people are trying to develop parallel algorithms that minimize the number and size of the messages, allowing to use cheap PCs as supercomputers.
hej e8johan -
Guys, Give him a break! My fiance' graduated with a Masters' from Chalmers. It's considered one of the more prestigious engineering schools in Europe. It's the Swedish "MIT". I think a little is being lost in the translation.. Don't mistake sarcasm for criticism. Slashdot is truely an international affair; Don't push our guests away.
(my Swede and I reside in Indiana, and I'm a homegrown Ohio boy)
THIS SPACE INTENTIONALLY LEFT BLANK.
There's a distinction to be drawn between big iron and a lot of small iron. People who've never used big iron never draw that distinction, nor do people who're trying to publicise their latest and greatest cluster, but there is a difference. A cluster of fast 32 bit PCs networked with gigabit ethernet does not big iron make.
2/3rds the cost of the three year computer lifetime is the electricity and cooling system. When TOC is counted a transmeta based cluster or the super-dese SGI cluster announced yesterday is cheaper.
It's very impressive what they've built, and I'm not knocking it, but I nearly split my sides at some of the quotes not directly related to the speed or hardware architecture of the thing:-
:)
"We have been using the File Transfer Protocol over Gigabit Ethernet, but now we will be able to read files directly from any available disk,"
Well - like wow - NFS/CIFS anyone. They've been ftp'ing docs to each other? ROFL
"Being able to share data across the enterprise is an exciting new capability. It will allow more collaboration among research projects,"
Ahh my sides are splitting - "shock news, scientists discover file sharing" heheh. Don't these guys have a file server? Guys listen up - you didn't need to design a world beating clusterbeast with 10Tflops just to share some files! LOL all that power just to let Larry from Sub-Atomic Meddling dept. look at a paper from Dave from the Induced Super Novae Working Comittee heheheh. These guys need to get out more: imagine their annoyance when they made this big announcement only to discover that not only has Novell, Microsoft/SAMBA, Unix/NFS done this already - they did it with only one CPU in the server!
"This network approach is nice because we can use a standard PCI slot on each processor node, "
Hmm like any network card you care to mention then really... Heheh "Hey like.. this network stuff is like - cool man!" What next? They invent a board with a button for each character they need to type? Priceless.
I'm sure it's great but I only just stopped laughing at those quotes. I can only imagine (or hope) it's a case of clueless journo mis-quoting or quoting out of context or just completely missing the point of the project.
"We have been using the File Transfer Protocol over Gigabit Ethernet, but now we will be able to read files directly from any available disk."
translation
We used to use FTP over Gig-E but came up with something more L337.
Trolling is a art,
Does it play Ogg Vorbis???
Chalmers the Swedish equivalent to MIT? Not really. In some areas, they are good, but in other areas, other schools are better. For example, when it comes to CompSci and related, Linköpings Tekniska Högskola(Linköpings Institute of Technology) and Linköpings Universitet are ranked as the best in Sweden.
And for your own health's sake, don't say things like that anywhere near students from KTH(Royal Institute of Technology). Or the opposite, for that matter...
Ahem!
Great work with that new supercluster! You guys are doing great, getting the most teraflops for your dollar!
Ummm...since you don't need it anymore...would you mind letting me have that ASCI White machine?
"Provided by the management for your protection."
It's sad that as much as Bernard Daines has done over the years to create newsworthy press for EE Times that they can't even mention his company has put this together.
For more information on why this cluster is much more than a bunch of Motherboards and NICs thrown into rows and rows of 42RU racks visit http://www.lnxi.com .
Finally this thing made slashdot! I've been trying to tell y'all about this for months, but the editors haven't found it newsworthy - see my journal (7/26 and 10/28) for links to additional articles and the home page for the cluster.
a race between engineers trying to make faster and better computers and Microsoft trying to make more bloated and processor-heavy operating systems. So far, Microsoft is winning.
Technoli
One of the benefits of computers is the ability to solve a problem with iteration rather than trying to come up with a classic "equation" and solve it. When I first entered the job market I had a trusty Pickett N4ES slide rule (and an N600-ES pocket slide rule) and had to first explain a problem with an equation and then solve the equation (from the "inside out" which was why HP calculators with RPG were so popular with engineers when they first came out versus the TI models... but I digress).
With the introduction of the HP-35 calculator (the "electronic slide rule") we could solve problems by just crunching the numbers at our desks. With the availability of programmable calculators (HP-67/97 and HP-41 - both of which I still use... but then I still use the slide rules too) we could program them to iterate through problems.
Not as elegant, certainly. But lots more efficient. And I'm sure that most of us have lost some of our old abilities to "see" problems in math... and perhaps some students never really learn that. But the jobs still get done and the tools still keep making it easier. I'm thinking about a Beowulf cluster for our office, actually.
No one ever had to evacuate a city because the solar panels broke!
The interesting thing about this setup is that it doesn't work like the traditional supercomputer. It's more like a community of totally independant computers all willing to work on the same problem.
The system employs a whole lotta control nodes that spend their whole time trying to assign work out to the worker nodes. The problem then becomes not just parallelizing the work but coordinating the workers. Apparently with this cluster design, it's not all as cut-and-dried as with a "real" supercomputer. They have been able to do some really cool stuff, though. Like, for example, any computer in the cluster can address the memory on any other computer.
The admins I talked to said they weren't really sure just how fast the system could go, because they could never get it to operate at full capacity. They said the fastest they'd gotten it to go was 4T-Flops, but they figured they were only at %40 theoretical capacity.
"With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea...."
RFC 1925
Have you ever heard of the Flat Neighborhood Network topology? Just because you can't figure out how to design a network for a cluster supercomputer, it doesn't mean nobody can. Apparently, you also seem to think you know Cray better than Cray. Not only is Cray now marketing Linux server clusters, it's also planning on building a cluster supercomputer using 16000 AMD Opteron processors. Perhaps you should jump in and inform them that clusters have gone the way of the dodo. Either you're wrong, or many highly paid Cray employees and various PhD's are.
*applies Ockham's Razor*
I've seen it, and I have the pictures to prove it. Its a pretty cool system. Wanna see the pics? Email me at jrhelgeson@hotmail.com with the subject of TFLOP and I'll forward the supposedly classified pics to you. The pics were taken at the NOC where the supercomputer was assembled prior to shipment to the Lawrence Livermore Nat'l Labs.
Good security is based upon reality and common sense. Common sense is a function of having common knowledge.
I think it is time to start thinking about raytracing in PCs. Imagine 3d environments raytraced in real time!!! now that needs a lot of computing power, and it is serious reason why processing power is not enough yet.
Raytracing is so much better than simple rendering. Just take a look around the Internet...beautiful raytraced images, almost real, in contrast to the rendered ones we have in current games.
And of course, there is always DOOM III, IV, etc...
For simple floating point the G4 is not that fast. For vector operations using Altivec the G4 is very fast.
That would be wonderful, except you can't metamod Overrated moderations. =(
I'll easily use up all that computing power with my single program that simulates single cell and multi cell bacteria colonies and that uses mutation to create more specialized and complicated creatures.
You can't handle the truth.
The term "cluster" has been used in so many contexts that it no longer has any meaning. OpenMOSIX cluster. MPI cluster. Disk cluster. Web server cluster. Is the definition of cluster simply any group of computers either on a LAN or the internet performing vaguely related tasks?
(+1, Troll)
Seriously, just because you write together a room full of computers, it doesn't mean much.
Sure if you specifically write an app to do parallel computing over fibre-channel nodes, and you have all these networked computers work on the same problem, it might turn out what seems like 10-TFLOPS. But it's NOT a mainframe.
Hell lets say we call all the machines doing SETI@HOME a cluster? How many TFLOPS do you think that could do? or IS doing? By "buzz word" logic Seti@home network might be one of the fastest mainframes in the world.
--zuchini
Is that old news as it was posted in July on CNET, here which reportedly had 962 nodes and was/is an "Evolocity" cluster.
... (NetworX built it with 959 nodes and 2050 processors) in-addition-to "Pink" named after "Pink"
:?)
Is this one for the sister lab(LANL):
A powerful new 2,048-processor, $6 million Linux supercomputer is being built for the Los Alamos National Laboratory to run unclassified analyses and
-or- it it "Pink"
-or- is the post about another one that going to LLNL and has nothing to do with "Pink"?
Is that 2, 3 or 4 this year?
This is getting strange.
Extreme Linux at http://www.extremelinux.org/ seems to have vaporized (behind the LANL firewall, I suppose).
It also seems kind of odd that LinuxBIOS is not mentioned in these articles and is, IMO, a critical part of the builds as well as being an outstandingly innovative use of the Linux kernel.
I think I need a Beowulf cluster to figure this out and tell me ?WHY? LLNL and LANL are suddenly in need of more SC's that are in addition to the ones that they already have.
In other words, the computers are networked via high-speed SCSI links, to increase bandwidth and thoroughput. I have always thought this was possible, and had probably already been done, but this is the first time I have seen such a thing written up about (in other words, it probably has been done in the past, and I just didn't read about it).
I am thinking the SCSI cards here are being used in a "poor-man's MYRINET" fashion, in order to get past the bottleneck of ethernet NICs and switches. Now, if they only made (or, maybe they do?) a SCSI "switch" (are those called crossbar switches?) for the thing, you could go to a star topology instead...
Reason is the Path to God - Anon
and be a slashdot editor as well.
Can you help???
It's called usenet.
All that and no gameplay, just like all the other new FPS-games that are coming..
Search for Incredible Coincidence at home
Determine the structure of the Universe!
Why did GEAR crush RDP?
It's ASCI, as in Advanced Strategic Computing Initiative (that somehow has mutated into "Advanced Simulation and Computing Program). http://www.llnl.gov/asci/
Federal authorities say they will seek the death penalty, but many observers think it is more likely that they will settle for a generous campaign contribution.
Given a choice between free speech and free beer, most people will take the beer.
A gangster assembled an engineer, a chemist, and a physicist. He explained
that he was entering a horse in a race the following week and the three
assembled guys had the job of assuring that the gangster's horse would win.
They were to reconvene the day before the race to tell the gangster how they
each propose to ensure a win. When they reconvened the gangster started with
the engineer:
Gangster: OK, Mr. engineer, what have you got?
Engineer: Well, I've invented a way to weave metallic threads into the saddle
blanket so that they will act as the plates of a battery and provide
electrical shock to the horse.
G: That's very good! But let's hear from the chemist.
Chemist: I've synthesized a powerful stimulant that disolves
into simple blood sugars after ten minutes and therefore
cannot be detected in post-race tests.
G: Excellent, excellent! But I want to hear from the physicist before
I decide what to do. Physicist?
Physicist: Well, first consider a spherical horse in simple harmonic motion...
- this post brought to you by the Automated Last Post Generator...