Slashdot Mirror
Grading Telco & ISPs During the Blackout of 2003?
alt_cognito asks: "Our company runs natural gas generators here in Novi MI and when the power went out we didn't miss a beat. Nine hours later, our telco blinked and our T1 service went down despite the lines being run to different locations and ISPs (UUNet, LDMI). Service did not return until power had returned to the upstream offices. I was under the impression that these locations would be run by similar power generation. How did your telco/ISP perform?"
Our generator worked fine (we were powerless till about 5am friday) and a dedicated line to Motient stayed up the entire time. But our UUNET T1 was down until about 6pm friday. and the entire time it was impossible to even get a Worldcom person on the phone to find out if they even had an idea as to when they would get thier act together. so basically I give Motient an A+ (well they were outside the blacked out area but I still got updated on thier network status inside the area) and I give Worldcom a big fat F.
It's great that you have two different providers. Eliminating the single point of failure is important. But, most people miss the semi-hidden single point of failure, the local telco. The problem is that your two ISPs likely don't own the copper that those T-1s run on. That copper is owned and operated by your local telco. Your ISP just contracts with the telco to provide you their service over the local telcos loop.
It is likely that you will never find out exeactly what happened but, from what you describe it sounds like; the lights went out, the local Central Office(CO) where the local loop for your T-1s went onto UPS backup or generator and after a few hours the UPS or the generator ran out of juice. Once the CO ran out of juice your T-1s went dead. So, you lost connection with the ISPs. More than likely the ISPs themselves never blinked.
The only way to avoid this problem it to use two different local loop providers which is usually going to be hard to find unless you are in a large metropolitan area. The other thing to do is get the local loop lines from different COs which will be like pulling teeth from your local telco.
Planning and preparedness, unfortunately, does not guarantee against failure.