Can Anyone Suggest a Good Switch?

wgadmin asks: "I am a sysadmin for a 500-node Linux cluster. We are doubling the size of our compute nodes, so, as a consequence, we need a new switch. We currently have a Foundry FastIron 1500 -- however, a) it doesn't have enough ports (only 208) and b) it is extremely unreliable. We want something that's solid as a rock. We process mostly serial jobs. And we will probably require ~320 ports. What's everyone using for their HPC clusters? There's so many high performance switches on the market, we hardly know where to start."

Comments

Several things left out.

[Zapman]

What level of interconnect do you want? (gig copper? gig fiber? 10/100?)

Or are you looking for something more specialized (HIPPI compliant or something similarly obscure?)

That said, if you're looking for in the ethernet space, we've been really happy with our recent Extreme Networks chassie's. Their black diamond 10k line is the newest release, and it looks awesome. It's really dense, they've got crazy levels of backplane bandwidth, and ours have been really reliable (granted, we have the previous generation of the gear). The chassies have blades (just like everyone else) that can speak 10/100, 10/100/1000 copper, gig fiber, 10 gig fiber, etc.

Re:Several things left out.

[wgadmin]

Sorry, I forgot to mention that we are interested in gig copper. We are exclusively interested in gig copper. And, as far as anyone has told me, we don't care about HIPPI compliance.

Extreme Networks

[Plake]

Extreme Networks has a great line of switches.

The Black Diamond 10808 would work great for the type of envrionment you have setup from the sounds of it. Also, Extreme is usually 20-40% cheaper then Cisco and Foundry for the equivilant appliance.

We currently use an Alpine 3808 with 192 100mbps ports and it's never had a problem with uptime and configuration is a simple and straightforward.

Re:Extreme Networks

[unixbob]

We also use Extreme Switches and I can vouch for their reliability and performance. Instead of going for the "one big switch" approach though, we've got a pair of Black Diamond 6808's with 1u 48 port Summit 400 edge switches uplinked back to the core switch (excuse the marketing terminology). This makes cabling much tidier when you have a high number of servers as you can locate the edge switches all around the server room then just have the cables from the Summit's in the rack with the Black Diamond. It makes deploying new kit much easier, and tracing cables much easier as well. You don't end up with the switch rack being a massive mess of untraceable patch cables. The only servers that are patched directly into the Black Diamonds are those using the NAS (because they need as much bandwidth as possible)

force 10

[complex]

http://www.force10networks.com/ claim to have the higest port density.

Re:force 10

[PSUdaemon]

Yes, these guys are awesome. We just got one of their switches for our cluster. All ports are line speed, no over subscription. They are also soon to announce some higher density line cards for their existing chassis in the upcoming months. Definitely give them a look.

Forget ethernet

[keesh]

Give serious thought to FC-IP and director-class fibrechannel kit. Performance-wise it'll thrash Ethernet, and there're various clever tricks you can do with directors clustered together via Open Trunking meaning that a bunch of 160 port boxes (a McData 6140 is your best bet here) will do as well as a larger single box.

Stackable 48 Ports

[DA-MAN]

I'm a sysadmin for a 3 large clusters in the same league, we use stackable 48 port Nortel switches. Each switch is 1u, and the interconnects don't use a separate port. The switches have wildly expensive support options, however because it just works we've never had to pay for support on them.

We use to have Foundry ourselves, but their switches were crap, they would suddenly become dumb hubs and lose their ip, etc.

We tried HP, but found their interface cumbersome and unfamiliar with weird networking related issues that would pop up.

Cisco's been rock solid, but very expensive.

Call your local supercomputing center

[beegle]

Send email to a few supercomputing centers. These places have tons of clusters, with lots of vendors throwing hardware at them. They're also often associated with schools, so they're not competitors and they actually -want- people to learn from what they've done.

To get you started:
http://www.ncne.org
http://www.psc.edu
http://www.sdsc.edu
http://www.ncsa.edu

Yeah, it's Pittsburgh-centric. Guess where I'm posting from. There's probably somewhere closer to you.

The things you want to figure out before calling:

-What's your budget? (Nice stuff tends to be more expensive)

-How much does latency matter? (Usually, lots. Sometimes, not so much. Put numbers here.)

-What's your architecture (at several levels of technical detail)? Can you use 64-bit PCI? Do you have to work with a proprietary bus? Can you use full-height, full-length cards? What OS -exactly- are you using? (Hint: "Linux" ain't close enough.) What version and vendor of PVM/MPI/whatever are you using, and can you switch?

Cisco 65XX

[arnie_apesacrappin]

If you're looking for Gig over copper, the 6509 will probably give you the density you want in a single device. It has 9 slots, one of which is filled by the supervisor module. If you want to upgrade to the 720 Gbps switch fabric, I think that takes another slot, but could very well be wrong. But with 7 available slots at 48 ports per 10/100/1000 blade you would have 336 connections.

The 6513 is basically the same thing but with four extra slots.

The 6509 chassis lists at $9.5K and the 6513 $15.25K. That's completely bare bones. The supervisor modules run anywhere from $6K to $28K at list. The 48 port 10/100/1000 modules list at $7.5K while a 24 port SFP fiber blade lists for $15K. You'll need two power supplies at $2K-5K each.

On the cheap end, to get the port density you're looking for out of Cisco, you'll pay about $70K list. But if you find the right reseller, you can see a discount of 30-40%.

All numbers in this post should be considered best guess, based on quotes I've gotten. They may be out of date. They are not official prices from Cisco. Take with the appropriate grain of salt.

HP Switches are very reliable, but run HOT!

[scum-o]

We're using the unmanaged HP procurve modular 1Gbps switches in our clusters, but they run VERY HOT when utilized (our switches get hammered 24/7 - like most clusters probably do) and we had some overheating issues with them. Our clusters aren't as large as yours, but I'd suggest going with a major manufacturer (IBM, HP, Cisco) if you're putting all of your eggs in one basket (switch-wise).

One thing is get a switch that's modular (most good ones are), but if something goes out, you'll only loose 8 or 32 nodes instead of the whole switch.

For a good switch

[Enrico Pulatzo]

try a hickory tree. Stings like hell and the mere thought is a deterrant for most rascals and rapscallions.