Slashdot Mirror
Grading Telco & ISPs During the Blackout of 2003?
alt_cognito asks: "Our company runs natural gas generators here in Novi MI and when the power went out we didn't miss a beat. Nine hours later, our telco blinked and our T1 service went down despite the lines being run to different locations and ISPs (UUNet, LDMI). Service did not return until power had returned to the upstream offices. I was under the impression that these locations would be run by similar power generation. How did your telco/ISP perform?"
Our generator worked fine (we were powerless till about 5am friday) and a dedicated line to Motient stayed up the entire time. But our UUNET T1 was down until about 6pm friday. and the entire time it was impossible to even get a Worldcom person on the phone to find out if they even had an idea as to when they would get thier act together. so basically I give Motient an A+ (well they were outside the blacked out area but I still got updated on thier network status inside the area) and I give Worldcom a big fat F.
Here in Canada, Bell Sympatico was up from when I lost power to when I got it back, I would just assume they were up all along. All ISPs I believe have failsafe setups that were never tested and maintained and that showed very well during the blackout.
At my company the IBM eSeries servers were backed by a smart UPS that showed 17 hours remaining, 15 seconds before shutting down everything in cold blood. It was all scandisks booting back up there on Windows 2000 machines.
I maintain some small servers with no UPS in a few locations, and while one Solaris server crapped out, you had to manually do the fsck thing, all the FreeBSD servers were back up as the power came without a hitch. I have to learn to setup Solaris on sparc so it fsck itself without asking for an input.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
...and without a contract, I doubt the workers will be working very hard to fix the damage caused. I have multiple T-1 lines down all over Manhattan, most on UUNet/WorldCom. The word is Verizon has a couple CO's that are still out, and I hope they know what they're doin....
"... I declare our city to be a free and independent state to be named Tri-Insula!" --Fernando Wood, Mayor of NYC 1861
It's great that you have two different providers. Eliminating the single point of failure is important. But, most people miss the semi-hidden single point of failure, the local telco. The problem is that your two ISPs likely don't own the copper that those T-1s run on. That copper is owned and operated by your local telco. Your ISP just contracts with the telco to provide you their service over the local telcos loop.
It is likely that you will never find out exeactly what happened but, from what you describe it sounds like; the lights went out, the local Central Office(CO) where the local loop for your T-1s went onto UPS backup or generator and after a few hours the UPS or the generator ran out of juice. Once the CO ran out of juice your T-1s went dead. So, you lost connection with the ISPs. More than likely the ISPs themselves never blinked.
The only way to avoid this problem it to use two different local loop providers which is usually going to be hard to find unless you are in a large metropolitan area. The other thing to do is get the local loop lines from different COs which will be like pulling teeth from your local telco.
Planning and preparedness, unfortunately, does not guarantee against failure.