Slashdot Mirror
Can Anyone Suggest a Good Switch?
wgadmin asks: "I am a sysadmin for a 500-node Linux cluster. We are doubling the size of our compute nodes, so, as a consequence, we need a new switch. We currently have a Foundry FastIron 1500 -- however, a) it doesn't have enough ports (only 208) and b) it is extremely unreliable. We want something that's solid as a rock. We process mostly serial jobs. And we will probably require ~320 ports. What's everyone using for their HPC clusters? There's so many high performance switches on the market, we hardly know where to start."
Cisco is the de-facto brand of networking gear for standard stuff like Ethernet. How much better are these high-performance switches people are talking about as suggestions in the comments here? This is not a rhetorical question, I just realy want to know and I'm too lazy and uninterested to look into it myself, but not lazy enough to stop typing this slashdot post. Is it enough to be worth going with non-Cisco for HPC clusters that use Ethernet-based interconnects? I know Cisco isn't infallible, but for all kinds of reasons they're a good bet in networking gear come purchasing time, at least outside this HPC cluster business.
11*43+456^2
Do not under any circumstance buy Extreme switches. They are really a bunch of lower-bandwidth units coupled together, which results in crazy packet reordering problems if you are putting streams of any size across them.
One of our sites tried them.... and practically kicked them straight out of the window. If you care about throughput then you want to stay well clear.
Ever seen a 500-1000 port FC switch or trunked network? Any idea what such a beast, if it existed, would cost? What's the cost on 500-1000 FC interface cards for the PCs? You do know that most PCs now come with gigabit ethernet onboard, right?
Our department uses Extreme hardware throughout in a fairly large network deployment.
We have two Linux clusters -- Viking, a 512 processor P4 cluster, and Mars, a 400 processor Opteron cluster (being commissioned). We also host a number of large Sun machines, including SunSite Northern Europe.
The department also hosts a 250+ node teaching lab and several floors of staff and research desktops, each with 100Mbit+ to the desk and a 1Gbit uplink from each switch. At the middle of our network are two Black Diamonds, one of which is the new 10ks which we helped Extreme beta-test (now in production). A third BD is being set up at a second site with 10Gbit fibre links.
The rest of the network ('the edge') is also made up of Extreme switches, ranging from Summit 1s, 24s, 48s to some of their newer 400-series 1u switches on Mars and elsewhere. (The 400-series can run 10Gbit fibre uplinks, and each copper port runs at up to 1Gbit, which is ideal for a cluster environment.)
Plus, they're are all coloured a nice shade of purple.
isn't almost all the latency in your network from software? why not build a hypertree from cheap 24-port switches? At $200 a pop, you could make a
tree with 12 roots for $8000. spend more to get
more cross-section bandwidth, less for less. it
scales with your budget.
-I like my women like I like my tea: green-
You can run hybrid mode/CatOS on the sup720 (catos 8.x, I think). I also would not use the 6513 - only slots 9-13 actually run at full speed. You're limited to half the backplane bandwidth on slots 1-8.