Slashdot Mirror
Supercomputers - Does the Cabling Matter?
papaia asks: "Having watched, for a while, the development in the area of high-density server hardware solutions (i.e. blade servers), like IBM's 'top gun', and their increased presence in Data Centers, I have been wondering if anybody has had any experience (thus comments) in regards to how important - in such highly priced solutions - is (or could be) the [always neglected] cabling, connecting the servers. One such comment caught my attention, in this regard. Slashdot, how important is the server cabling infrastructure in your Data Centers, and how do you resolve the cable management aspect of it?"
When you're running a large linux cluster, it is absolutely vital to plan your cabling and get it right. It just is, for access, fault troubleshooting, avoiding restricting airflow. Get cable tray, lay it to each rack site (you have planned out banks of rack sites and planned the airflow to your air con, right? - and please, tell me you have air con???).
Allow for larger racks so there's room for cables and PDUs. Really.
Keep things neat with a raised floor, cable tray underneath. A second tray overhead can be handy, but can interfere with moving equipment, so don't fit it unless you'll need it.
Be aware high speed cables, particularly fibre, have a minimum bend radius - go beyond it, and you have fucked up the cable. Also, be aware of sag due to weight of cables - compress the cable too much with the weight of the bundle against cable ties, and you can damage it.
Crosstalk is seldom an issue these days, but be wary of laying power and network too close - even if there is negligible interference, you're safer if they're separated by a decent amount.
Patch panels are useful - use them. Run cables to patch panels and patch panels ->switches, don't go machine -> switch directly (unless you're doing really high-end stuff (for 1GigE copper or fibre, patch panels are fine).
Literally speaking the cabling won't matter at all. Whether the resistance of your wires is high or low the electrons are going to travel through it at the same rate. What may be a worry is freak occurances of inductance between wires which could possibly mess with your data, but I'm not sure how common that is. The signal is digital so it's going to be either a 1 or a 0 depending on the voltage of the line, and it's usually difficult to make the voltage do something as drastic as go from +5 to 0 or +5 to -5.
I always have to laugh anytime I go to the store and look at the number of things that say digital nowadays. Wal-Mart sells digital telephone cable for your computer to connect to the wall as if 5 feet of high quality cabling with gold plated tips is going to make a difference in the 30-odd mile trip to your ISP. A friend of mine boasted his "digital" headphones to me once and I had to beat him down to the opinion that "there must be something better about them, so I'm happy" (this sort of pacifist optimism is the bane of IT in general).
Best Buy sells Monster digital audio cables at something like $20 for a 5-ft cable. I had to argue with my father trying to convince him that the cheap RCA cables we already had back at home would be perfectly capable of communicated a digital signal the 5 inches between the DVD player and the receiver. I could have ripped two wires from a speaker cable to connect the two devices and would have gotten just as good sound.
People don't seem to want to realize that digital implies lossless or error-corrected. They don't understand that the "premium quality sound" transmitted between devices can be done using the cheapest electronics equipment available.
I'd save a fortune on car stereo...
Direct away from face when opening.