Slashdot Mirror

Plus you can get 48 10Gb ports in 1U. No one else in the industry can touch that density

Not entirely true. Arista has little secret sauce. They're using merchant silicon. Stay tuned for the plethora of switches about to be released based on the Broadcom Trident ASIC (64 10 GigE switch on chip, manifesting itself as either 64 10 GigE or 48 10 GigE + 4 40 GigE). Some (like Force10's) are already announced. The differentiator in 10 GigE ToR switching is quickly becoming software.

Actually, I learned today that BSD is very much alive and fast to boot.
Force10 uses a version of NetBSD for their 10GB+ switches/routers.

That's less than $10k per 10Gb port for the hardware. Given that google returns about $20K per port per month, the hardware cost is pretty inconsequential.

Then again, I'm not a network engineer, so I may be reading these price sheets incorrectly. Of course, the trick is to use dark fibre and then find some way to attach to the internet, without having to pay the $80k for the 40Gbs bandwidth you're using.

Or pick up a older used L3 switch... Cisco 2948-GL3 @ Ebay The 3750's are nice, but they're priced out of reach for someone who is struggling trying to cobble together a router out of a PC. If you're going to spend the money on a decent L3 switch, check out the Force10 S50, they are a smaller vendor, but the quality is tops.

Actually network equipment vendors are getting pretty close to 60Gbps backplane speeds for individual line-cards.

Both Cisco, with it's76xx series, and Force10, with it's Exxx series, currently offer line-cards with 40Gbps switching capacity.

Regards.

Top500 classifies supercomputers as clusters, constellations, and MPPs (Massively Parallel Processors). BG/L is an MPP. It is obviously not a cluster since a single compute node is not a general-purpose machine. BG/L is specifically intended for massively parallel applications.

The difference between clusters and constellations is the ratio between the number of nodes (that can either work independently or be clustered) and the number of processors in a node.

Go to the Sublist generator of Top500 to see what machines belong to which category.

See, e.g., definitions here

I've always understood that Myrinet is one of the better latency products available.

And it has MacOSX Drivers:
http://www.myri.com/scs/macosx-gm2.html

Myrinet is used by 39% of the Top500 list published in November 2003
http://www.force10networks.com/applications/roe.as p?content=9

http://www.force10networks.com/ claim to have the higest port density.

A lot of posters in this article decry commodity hardware for sucking but they're not considering the long term. The tasks commonly performed by commodity hardware are starting to be more and more suited to scientific applications. We can already see how vector optimizations like Altivec and SIMD have worked their way into every desktop chip around.

As 3D rendering and sophisticated media codecs are becoming the primary reasons for upgrading a home PC, the front side bus and CPU (especially the CPU) have gained a disproportianate speed advantage compared to the rest of the PC. And when you start throwing in high speed network compenents like blade interconnects and 10Gig E, you can create an extremely powerful, scalable computer. Vendors like IBM love this because it means they can sell one design to everyone, just changing the number and specs of the nodes to fit their customer.

Speaking as someone who worked support on a large academic cluster where thriftiness is king, not only is the hardware cheaper, but it's mundane specs mean it's less likely to fail, when it does, the nodes are replaceable like lightbulbs, and you can run down to the local computer store in the event of a faulty IDE drive or memory stick. The support costs and reliablility are far superior to typical high performance solutions because support can mostly be done in house by existing staff.

And that ComputerWorld article is mostly bunk, OpenMP supports Fortran. The people who are writing large scale simulations in the first place are not only technical but also extremely bright. While it is a hassle, adding networking to the already heavy code optimization that they do is not that big of a deal.

Though there are problems that require a shared memory system, $2 million can buy so many more networked Xeons than shared memory SGI processors that the scientists in question should really consider their needs. While it may take much longer, their problem set can be much, much larger since they'll have more collective RAM and HD space. The system can also easily be partitioned, rebuilt, and shared in such a way that it's always in use by one or several people. In many cases, the tradeoff is worth it. And even then, shared memory systems will probably still have a lower price/performance ratio compared to clustering a smaller number of high memory database servers. Say for example, a group of Itaniums with 64Gig of ram each vs their SGI Origin equivalent in terms of memory footprint.

I think as time goes on, interconnect speed will increase to the point where clusters are very similar to traditional supercomputers. We can already see baby steps in this direction with blade equipment and various motherboards with gigabit NICs built into their own bus to avoid choking in PCI land. Infiniband and 10G over copper are the first major steps but switching and motherboards still have to catch up it seems. Instead of whining, the typical supercomputing vendors should be looking into merging standardizing their designs with those of clustered systems. Of course, I could see how the entrenched traditional vendors would want to legislate as much as possible to avoid having to compete with dozens of smaller engineering companies with less overhead and better ideas.

Now that I think of it, a lot of the posts I've read here are probably skeptical of massively parallel machines because they don't realize that the performance is all in the network. I've seen two clusters first hand with an identical number of identical nodes. One was being used as a compute farm where several hundred dual Xeons were attached to a rediculously large and overly expensive IBM foundry switch. The other was being used as a massively parrallel simulation machine, with dual gigabit nics attached to a bunch of 'cheap' 24 port switches in a 2D mesh. The second design cost much less because it wasn't built by IBM while still being about twice as fast.

Their command line is very cisco-like and their density is unbelievable. I just had the opportunity to test a couple of them for a customer of mine and was amazed.

I'm now recommending them to more of my customers.