Stupid Data Center Tricks

jcatcw writes "A university network is brought down when two network cables are plugged into the wrong hub. An employee is injured after an ill-timed entry into a data center. Overheated systems are shut down by a thermostat setting changed from Fahrenheit to Celsius. And, of course, Big Red Buttons. These are just a few of the data center disasters caused by human folly."

Comments

bad article is bad

[X0563511]

The summary reads like a digg post, and has two different links that, in actuality, link to the exact same thing.

This needs some fixin'.

Re:bad article is bad

[macwhizkid]

Article also needs fixin' in the lessons learned from the incidents described. Look, I'm sorry, but if your hospital network was inadvertently taken down by a "rogue wireless access point", the lesson to be learned isn't that "human errors account for more problems than technical errors" -- it's that your network design is fundamentally flawed.

Or the woman who backed up the office database, reinstalled SQL server, and backed up the new (empty) server on the same tape. Yeah, a new tape would have solved that problem. Or, you know, not being a mindless automaton. Reminds me of a quote one of my high school teachers was fond of: "Life is hard. But life is really hard if you're stupid."

Re:Not using Cisco ACLs

[omglolbah]

Amusingly anyone who ever worked as tech crew at a lan party knows that this is the first thing you look for... :p

Re:Quad Graphics 2000

[Anonymous Coward]

Why the fuck was the button unlabeled? That's the REAL MISTAKE.

Re:Not using Cisco ACLs

[Gumbercules!!]

I have to agree with this guy. As soon as IP addresses started being assigned incorrectly, the first thing I would be doing is checking the DHCP server. ipconfig /all on a windows box (so may 3 seconds of typing) would give this answer.

More to the point, though - why was another DHCP allowed on the network? Can your switches not block or refuse to route DHCP traffic from the wrong host?? Otherwise every single student who brings in their own wifi box is going to shut down the network.

Re:Power strips (with on/off buttons) are bad

[Velox_SwiftFox]

Covering those power strip buttons with a hardened glob fixing them in the "on" position is what an electric glue gun is for.

Re:Not using Cisco ACLs

[blair1q]

Or unplug it.

The slow part is figuring out that that's the problem. The first time it happens to you.

Which is why it's good to have oldbies around, to whom lots of weird shit has happened.

Fun with PIX

[mkiwi]

I had fun with a company awhile back. They are about 300 employees and ~90mil/year, so this is a small corporation.

Anyway, the company was trying to get a VPN tunnel established to their China office, and they were having a hell of a time at it. The employees on the China side had no IT experience so everything was done remotely.

It just so happens that one of the Chinese employees was recruited to make a change to the PIX firewall on the China side in order to get everything working. To our astonishment, it worked, and we had a secure VPN tunnel established.

The problem was accounts in the US started to get locked out, alphabetically, every 30 minutes. Our Active Directory was getting tons of password crack attempts from inside our internal network. I was using LDAP to develop an application at the time, so naturally I was suspect for causing all these lockouts.

Fast-forward a week. We look at the configuration of the Chinese firewall and it allowed all access from any IP address on the Chinese side. In other words, crackers were trying to get into our systems through our VPN tunnel in China. In effect, our corporate LAN had been directly connected to the Internet. Once we figured that out, I was free to go back to work and the network lived to see another day, but that incident caused major trouble for all our employees.

Moral of the story: Don't trust a Chinese firewall.