Slashdot Mirror

"For those with a short memory, NEC is a "baby bell" (an AT&T spin-off)."

BZZT! NEC http://www.nec.com/ is Nippon Electric Corporation, an immense japanese conglomerate founded in 1899. You're probably thinking of NCR, which was swallowed, never digested, and subsequently regurgitated by AT&T.

Your third real option is probably Fujitsu or IBM. The issue isn't just the interconnects, as you can buy those from Myricom http://www.myri.com/ or Quadrics.http://www.quadrics.com/ PNNL did this for their monstrous Itanium-2 system. It's also memory bandwidth, disk throughput, and that some jobs really require vector processors.

What Microsoft brings to the table, beyond incompatibility, overhead, and confusion, is really beyond me at the moment. They have a possibility of making an impact on high-throughput systems, but I can't see what they offer high-performance users.

If you look up how the top500 benchmark, and most of the others, slapping together a heap of boxes doesn't get you anything. To actually get a decent score on parallel DP linpack, or simulation codes used as benchmarks, you need a fast, very low latency interconnect between the nodes, excellent synchronization, and fast disk access.

Even the allegedly "off the shelf" systems contain an awful lot of not off the shelf hardware. Case in point would be PNNL's Itanium cluster http://www.emsl.pnl.gov/capabs/mscf.shtml/ (at 1000 or so nodes). At SC2003 I chatted with people I know from there, and they mentioned that they had four (4) Quadrics http://www.quadrics.com/ interconnect cards Per Node, plus extra switches, in order to get the bandwidth up high enough. Even a cheap cluster will add Myrinet (at about $1500/node when the switch is factored in), and start worrying about topology after the first few dozen nodes are installed.

There are clusters (basically networks of workstations), and then there are supercomputers.

Actually this is a very good point. Cray computers use Linux, Quadrics computers use Linux, and I'm sure anyone who has made an "Imagine a Beowulf cluster" post has read this. If I was managing Microsoft, I'd want people to think Windows is a powerful OS, so the last place I'd want it to be overtaken on is the world's more powerful computers. Especially by a freely available OS like Linux. Thus...the upcoming supercomputer Windows.

BTW the Windows-for-supercomputers story is already rising up on the Google lists. Check on or before the 7th link.

So essentially Microsoft is back to taking the approach that if they close their eyes tight enough, everything will be OK?

'Super-Linux' Cluster Declared Third-Fastest Computer On Earth

fastest computer system in the US

NCSA Linux Cluster Among Fastest Computers in the World

Two Linux clusters on Top 10 list of fastest computers

At least 5 of the top 10 systems are running Linux, starting at number two with Thunder at the Lawrence Livermore National Laboratory. The others are IBM BlueGene/L clusters at places #4 and #8, Tungsten at NCSA at #5, MPP2 at Pacific Northwest National Laboratory at #9, and probably also the Dawning 4000A at the Shanghai Supercomputer Center as #10, though I'm not 100% sure about this last one.

I don't think the fat tree died with Thinking Machines. For example, MCR at LLNL uses a Quadrics fat tree. I imagine many sizeable clusters (way more than 64 nodes) use one. There's one link here, and the MCR link here but you can probably google for quadrics and fat tree to find some more. I'd be surprised if fat trees didn't show up in Myrinet / other interconnects, but you typically need to have a sizeable cluster before there's any point in calling it a fat tree.

(Oh, and if you meant something else entirely by fat tree, I apologize. I'm not too familiar with the particulars of the CM5 fat tree, so the Quadrics one is the only usage I'm aware of.)

I thought LFNW was awesome. I especially enjoyed the Pacific Northwest National Labs High Performance Linux Cluster talk given by Timothy A Witteveen of PNL. It is one hell of a machine. It is one hell of a machine. 9.4 teraflops and a 53 terrabyte SAN running NWLinux. It placed 5th on the last Top 500 List.

That 53 terrabyte SAN is one contiguous filesystem using lustre. The use of QSNet2/Elan4 interconnects make the use of terbyte data sets with lots of internode communication more efficient than past machines. These interconnects provide a peak bandwidth of 340 MB/sec in each direction. But even more impresive than the bandwidth is the latency, between 2 us and 5 us. Compare that with ethernet latency measured in tens or even hundreds of ms.

During the presentation Tim went over two examples of simulations performed on their cluster that could not be accomplished on other machines. These examples were outside of my domain of knowledge, but one involved simulating the behavior of water molecules and the other was an extremely detailed protein folding simulation.

One last bit that was interesting was their methodology for updating the machines. They have over a thousand and took some time to determine an efficient means to keep the machines up to date. It was determined that reimaging the maches was faster than applying patches. They utilize a multicast approach in which allows them, theoretically, to reimage all of the machines in 28 minutes. They do not always reach this theoretical maximum, but they reimage a thousand machines PDQ.

The meet & greet in the commons was fun. Pogo Linux had a free drawing for a loaded AMD64 system. There was plenty of swag to be had from all sorts of folks. It was cool to stop and chat with one of the Helix developers. All in all there was a really good group of folks gathered.

I know Slashdot is full of trolls, but I must say I am a little suprised at how many crappy comments LFNW is getting. All I have to say to those filled with negativity is screw you. Here in the Pacific Northwest we have some cool stuff going on. Whether it is the 3 new lugs in Seattle, OSDL in Portland, the Linux Cluster at PNL, SeattleWireless, PersonalTelco or LFNW there are exciting things happening up here. I think a couple people on this site need to take their heads out of their arses and take a look around. Folks up here are using linux, and getting stuff done.

There are better ways to connect systems. eg: Myricom and Quadrics (More money, but faster)

Not quite. See the Quadrics webpage. This machine is using the Qsnet product. You stick a card in a PCI slot on each machine (it's not obvious if you *have* to have a switch in between or not for only a 2-machine network), and use the provided library to send data around.

Therefore, it does not appear to be transparent.

Also, from using a similar interface that is ~$4k per, I'd guess that these are Officially Not Cheap, although maybe if you buy in the quantities that these guys do, you can make out okay.

Really, for most user-level applications, it's hard to beat Fast Ethernet. I paid $26 each for my cards, and $90 for the hub.

Bus 0, device 12, function 0: PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 3). Master Capable. Latency=64. Min Gnt=4.

But you can't use this to connect a rack of computers. For one thing the max cable length for connecting two busses would be just a few inches. For putting PCI cards in 1.75" high 1U rackmount cases, there are PCI risers with a short ribbon cable that connects to the PCI slot. Even these short cables often cause timing problems. For instance, with the riser, cards may only work in the first one or two slots that will otherwise work in all the slots.

But even if you could cable all the computers together on one giant PCI bus, it would still be a bad idea. A good 24 port gigabit ethernet switch (~$2000) has a 480MB/sec switching fabric, to support full speed full duplex on each port. 32 bit 33Mhz PCI is only about 132 MB/sec, not nearly as fast. You'd need a 64 bit 66 Mhz PCI bus to keep up. And there are more expensive gbit switches with more ports that have 100 Gbit/sec fabric. And this is just gbit ethernet, the slowest and cheapest of the high speed interconnects used in modern Beowulf clusters.

There are faster ways to connect computers than gigabit ethernet. The EE times article is very untechnical, but this one has some more information. LLNL has used a very fast and very expensive interface called quadrics. This is probably the fastest way to connect computers in a Beowulf. People like Cray/SGI and IBM have faster things still, but they cost real big bucks. Other ways to connect a Beowulf are the above mentioned gigabit ethernet (~$100-$250 a node for up to 24 nodes), myrinet (~$1400-$2000 /node up to 128 nodes), and SCIhardware and software (~$1400-$2100 /node). Myrinet uses a switch like gigabet ethernet and the largest switch they have is 128 ports. SCI is switchless, each card has multiple cables (1-3), and is connected in into a ring, 2D or 3D torus.

For those of you who are wondering what they mean by high performance networks inside the Q machine..

The Q machine utilizes dual-rail Quadrics card according to this. Dual rail refers to using two NI cards (each one on a separate 64b/66MHz PCI bus so they can get the most out of the I/O system of the host).

I hadn't heard of Quadrics so I looked them up. At the web site you find out that they're a switched network that gets 340 MBytes per second between applications and with latencies around 3-5 microseconds. Compare this to 100Mbps ethernet, which gets 10MBytes/s and latencies of 70+ microseconds and you'll understand why the Q machine will run fine grained parallel apps that the green machine won't be able to touch.

Looking a bit through the literature, I noticed that Quadrics uses IEEE 1596.3 for its link signaling (400 MBaud, 10 bit). While they don't say it anywhere, this IEEE standard is the well-known SCI standard (scalable coherent interconnect.. pretty popular in Europe, but the US has been dominated by Myrinet..which I conicidentally use at school)..

Hope this gives some more detail about the arch..

Which has an excellent product page here. 2.35 usec latency for a short message. 340 MB/s peak, 210 MB/s sustained throughput. Fault tolerant redundant links. Tru64, Solaris and Linux support. I know nothing about this, but it sounds impressive to me.

The cluster will be built by quadrics who built a 128 nodes alpha cluster for Sandia Labs, of which they have a good picture on their Company Overview webpage.
---

The cluster will be built by quadrics who built a 128 nodes alpha cluster for Sandia Labs, of which they have a good picture on their Company Overview webpage.
---

The cluster will be built by quadrics who built a 128 nodes alpha cluster for Sandia Labs, of which they have a good picture on their Company Overview webpage.
---