Slashdot Mirror
Supercomputers - Does the Cabling Matter?
papaia asks: "Having watched, for a while, the development in the area of high-density server hardware solutions (i.e. blade servers), like IBM's 'top gun', and their increased presence in Data Centers, I have been wondering if anybody has had any experience (thus comments) in regards to how important - in such highly priced solutions - is (or could be) the [always neglected] cabling, connecting the servers. One such comment caught my attention, in this regard. Slashdot, how important is the server cabling infrastructure in your Data Centers, and how do you resolve the cable management aspect of it?"
any experience (thus comments) in regards to how important - in such highly priced solutions - is (or could be) the [always neglected] cabling
.....you werent even really trying
im sure you could fit more parentheses and brackets in there!
When you're running a large linux cluster, it is absolutely vital to plan your cabling and get it right. It just is, for access, fault troubleshooting, avoiding restricting airflow. Get cable tray, lay it to each rack site (you have planned out banks of rack sites and planned the airflow to your air con, right? - and please, tell me you have air con???).
Allow for larger racks so there's room for cables and PDUs. Really.
Keep things neat with a raised floor, cable tray underneath. A second tray overhead can be handy, but can interfere with moving equipment, so don't fit it unless you'll need it.
Be aware high speed cables, particularly fibre, have a minimum bend radius - go beyond it, and you have fucked up the cable. Also, be aware of sag due to weight of cables - compress the cable too much with the weight of the bundle against cable ties, and you can damage it.
Crosstalk is seldom an issue these days, but be wary of laying power and network too close - even if there is negligible interference, you're safer if they're separated by a decent amount.
Patch panels are useful - use them. Run cables to patch panels and patch panels ->switches, don't go machine -> switch directly (unless you're doing really high-end stuff (for 1GigE copper or fibre, patch panels are fine).
I think that it'd be fairly important. Even if only for reliability. Because speed and reliability are equally important.
Pretty Pictures!
This isnt really an answer - more a bit of trivia:
The ol' Cray-X supercomputer where round (their cabinets where placed in a circle) so that the length of the cabling could be kept down. Back then the synchronisation between pulses in different cables where a problem. And there really where a *snakepit* of cables between the cabinets.
Literally speaking the cabling won't matter at all. Whether the resistance of your wires is high or low the electrons are going to travel through it at the same rate. What may be a worry is freak occurances of inductance between wires which could possibly mess with your data, but I'm not sure how common that is. The signal is digital so it's going to be either a 1 or a 0 depending on the voltage of the line, and it's usually difficult to make the voltage do something as drastic as go from +5 to 0 or +5 to -5.
I always have to laugh anytime I go to the store and look at the number of things that say digital nowadays. Wal-Mart sells digital telephone cable for your computer to connect to the wall as if 5 feet of high quality cabling with gold plated tips is going to make a difference in the 30-odd mile trip to your ISP. A friend of mine boasted his "digital" headphones to me once and I had to beat him down to the opinion that "there must be something better about them, so I'm happy" (this sort of pacifist optimism is the bane of IT in general).
Best Buy sells Monster digital audio cables at something like $20 for a 5-ft cable. I had to argue with my father trying to convince him that the cheap RCA cables we already had back at home would be perfectly capable of communicated a digital signal the 5 inches between the DVD player and the receiver. I could have ripped two wires from a speaker cable to connect the two devices and would have gotten just as good sound.
People don't seem to want to realize that digital implies lossless or error-corrected. They don't understand that the "premium quality sound" transmitted between devices can be done using the cheapest electronics equipment available.
I'd save a fortune on car stereo...
Direct away from face when opening.
If nothing else, in an extremely complex environment, if you use a quality cable and quality connectors (skillfully attached) you can eliminate the bus as "one more thing to check" if you are getting unexplained slow downs. It is a nice way to shorten the troubleshooting to do list when you are up to your eyeballs in alligators and the pager wont stop buzzing.
...But I digress. TREMBLE PUNY HUMANS!ONE DAY MY SPECIES WILL DESTROY YOU ALL!
Quality cabling always will make a difference. Not quite a computer situation, but still similar, my friend was recently hooking up a new DVD player up to his new projector with component video inputs. He first just grabbed the first pair of RCA cables that he could find. The projector kept resyncing with the YPrPb inputs. Despite soomeone else's refusal to accept it, I told him to pull out some video monster cables. Once he did that, it eliminated the resyncing.
Cable quality will affect both digital signals and analog signals alike. A bad quality cable will generate a good share of dropped packets, or corrupted data, causing more resends or less accurate data. Also, take care if crimping your own cables, make sure you untwist wires as little as possible, and break the insulation and sheilding as little as possible.
With that said, don't be like a crazy audiophile (key word here crazy) and spend thousands of dollars just on cabling (I know an audiophile who spent 500 dollars on a 6" cable, when a $25 monster cable has the exact same specs. He claims to hear a difference, but I call b.s. on him.). Spending more means getting better, but only to a point.
In any setting the quality of cabling does matter. Or, more specifically, the quality of the connectors on the cable, and the quality of their installation.
Anyone who argues otherwise should recall that the first step in troubleshooting is almost invariably to check the cables.
While I am happy to use zip cord to wire my stereo speakers, I wouldn't trust dollar store cables for anything mission critical.
More important though is to document your cable runs, or even better tag each cable so that you have some idea where it begins and ends. You may know what goes into and out of that big ball of CAT-5 on the floor, but the guy who follows you will have no idea.
Three Squirrels
People don't seem to want to realize that digital implies lossless or error-corrected. They don't understand that the "premium quality sound" transmitted between devices can be done using the cheapest electronics equipment available.
Digital, maybe, but you don't want to raise the ire of the analog stereophiles: You'll get everything from Stereo cables make a difference to Debunking the Myth of Speaker Cable Resonance, not to mention forests worth of dead tree sacrifices for Speaker Cable Face Offs.
And please, please, please, please: Don't get them started on Solid State -vs- Vacuum Tube...
$500 for a 6 foot audio cable? Your friend prolly got some cheap low-end swill. :-)
/. thread has a few other interesting examples, including the one above.
Here's the real good stuff. I wish I weren't making this shit up.
This previous
Me? I just hook everything up using lamp cord.
Windows (TM) Supercomputer Uptime: 23:59:5BSOD!
Behold the Rat's Nest,
If your datacenter is 24/7, doing costly (financial), life critical (healthcare) or corporate production, then cabling ranks right up there with A/C and power. In fact all three of these are more important than apps or server platforms.
I mean, most signal cabling is now part of a network, (IP, FC, ESCON, Token Ring, etc.). A single cable failure can lead to a network failure which, like an A/C or power failure, affects a good portion of the datacenter.
I've seen poor cabling take out a datacenter on a couple of occassions. In one case, the engineers had loosely laid fiber cable for their network backbone under the computer floor. The cable draped over metalic power conduit. A year later, the datacenter contracted to have the power upgraded. The electrician pulled out the old conduit taking about ten fiber pairs with it. The company lost a good portion of their IP connectivity for several hours. Cabling is critical.
Cabling should be well thought out and properly run. The best systems I've witnessed are seperate trays under a computer floor for copper signal, fiber signal and a third for power. Cable runs go down the rows under the backs of the racks. All trays have proper feeds for each rack. All new cabling is quoted, and contracted before installation. Any equipment removal entails cable removal.
The best cable management system I've ever seen was at a TV station. The chief engineer kept several different cable lists depending on the cable function. Each cable was given a number. Once the cable was run, on his inkjet printer, he printed up cable labels using a Brady label sheet. The label identified the use, local connection, remote connection and number. There were never any problems disconnecting or reconnecting equipment.
Cables tell the story. If you are ever going to contract a datacenter for rack space, a visual check of the cabling will tell you more about the establishment than any brochure or spec sheet. If the cables are well run, you can bet - the power and A/C are properly spec'ed and redundant, their bandwidth adequate, and their building secure environmentally and physically.
machinator omnis sine licentia
I strongly disagree with putting cabling under the floor. Out of sight, out of mind works fine when you're talking a few cat-5 runs to a cubicle farm, but when connections are your main business, put them up where they won't be neglected.
The under-floor space can be used for AC power, if you use AC, but that's usually just for convenience outlets. Downflow air handling units that use the floor for air distribution are good too. I've even seen installations where the DC power cabling was run under the floor, and it simplified things greatly. But please, don't put your signal cables down there.
For one, it's easy to drop a tile into the floor while trying to remove it. One bad suction cup can cause the crushing or cutting of a cable. For another, it's awkward to feed cables through the little cutouts at the bottom of a cabinet. I've seen a lot of dirty or damaged connectors because of this.
If you're not bolting your cabinets to the floor, it also creates a shear point if the cabinet shifts. Please do bolt down your racks and cabinets, because they can tip.
Hiding the cabling also encourages poor workmanship. When someone has 20 feet of slack to store, and they throw it in a clump under the floor, it's a nightmare when another cable in the same area has to be pulled out. The initial infraction would've been noticed immediately if it'd been overhead, in plain view.
A well-designed overhead cable rack system is superior to any floor system. It's cleaner, because there's literally less dust colleting on it. Running cables overhead doesn't involve dragging them through a pile of connection-ruining crud. It's easier to install, because you don't have to contend with tile supports. It's easier to expand, because you can visualize the whole layout easily, and see where the congested areas are.
Furthermore, overhead rack is a natural companion to fiber trough systems, most of which are intended to be overhead. If you have a mix of fiber and copper, and most of us do, you owe it to yourself to plan a system that accomodates large amounts of both. As equipment density rises, the amount of cabling you'll need to bring to each rack also rises. Plan for that.
Also, plan for slack runout areas. The cables are never the exact length you need. Running them back and forth in the rack can create all sorts of tangle problems. Having a designated path to run your slack loop down can really make tearouts less dangerous.
Also, don't underestimate the sheer size of the cabling you're dealing with. I saw one particularly bad example, where a company had laid out their aisle of patch panels very carefully. The bottom of each bay was for panels that went to transport equipment, and the top was panels for other equipment. That way, most cross-connects could be made without leaving the bay. There were cable management rings to accomodate the occasional jumper that had to go between bays. It worked great.
Then they merged with another company, and the recordkeeping system changed. The new system made port assignments automatically, and it didn't respect the physical layout of what was where. Now the majority of jumpers were long, inter-bay runs. Over time as circuits got moved around, the management rings got filled, overoaded, and eventually stuffed to the point that the mass of wire was essentially solid. You could punch the bundle and it would go "thud". Pulling out a jumper was likely to burn through its neighbors simply due to friction, so they stopped pulling old ones out.
Eventually they added a dedicated piece of cable rack, and run all interbay jumpers up there. It was ugly, and awkward, but it worked. The initial system was much better, but relied on a level of care and planning that the new owners weren't willing to provide. Consider this: Will your successor's successor curse your name, or laud you for laying out a comfortable, expandable environment?
It's not the cables, it's how you cable. Kentucky bred a cabling strategy for their cluster. The Big Mac project at VT was supposed to release a software package that made cluster cabling easier, IIRC, but I can't find it anywhere.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
One thing we do with 'cat 5' cables is color-code different length cables, so black == 3 feet, green == 7 feet, yellow = 10 feet, orange == 14 feet, red == 25 feet, grayish-white == 50 feet, red with yellow boots == crossover cable. This has been helpful in a number of ways. -- it's unlikely a white cable will be to something else in that cabinet, crossovers are easily identified, longer cables are probably for servers further away from the switch/patch panel/whatever, it provides some color distinction in otherwise monochrome patch panels, it's easy to stock and order cables like this.
One thing I wish we did is have unique serial numbers on both ends of each (and every!) cable. While it's possible to trace cables using the tried-and-true tug-and-feel method, in reality it sucks and printed documentation is difficult to keep in sync with reality.
I've also seen cables color-coded for other purposes, but these haven't worked as well e.g.: one color is for network, another for KVM, another for switch uplinks, etc. This works well until you need a KVM cable, but don't have the right length in the right color so substitute "temporarily", blowing the scheme completely since 'temporary' is a synonym for 'permament' in most datacenters. another example: Use every color available randomly in the hope that there are only so many hot-pink cables with a green stripe in your datacenter making it easier to trace things. In reality this last example doesn't scale well and makes patch panels look really untidy.
As far as what I *think* you were asking, which is whether there is some qualitative difference between cables -- there is. Make sure you get 'certified' cables from a trusted vendor, preferably each one individually tested with the results pasted on a sticker on the (sealed) bag each cable comes in. Also make sure you get 'plenum' cables where necessary to comply with fire codes and just plain common sense. I'd say any permament infrastructure cables (not patch cables) should be plenum whether they are legally required to be or not -- if you have a fire you'd be better off without a few hundred extra pounds of fuel to keep it going. Beyond plenum/pvc and tested cables there isn't much else to stress over -- thank god "Monster" doesn't make patch cables with 24k gold connectors to hoodwink unsuspecting people -- if the cable tests good the rest doesn't matter.
"But actually trying to use m4 as a general-purpose langage would be deeply perverse" --ESR
What the gentleman meant is they need to be jacketed.
For in wall applications, you can't use plain old insulated copper anymore, you require something with a fire-resistant outer jacket.
Most companies accomplish it with PVC or PVC and foil.
.sig: Now legally binding!