apologies if this was posted earler... from:http://www.dotcomscoop.com/worldcom.html#outa ge
LATEST NEWS
Thursday October 3, 2002 @ 4:31 PM EDT
Have fun with this one:
FLASH SUMMARY FLASH NUMBER 20021003023 ORGANIZATION UUNET SEVERITY LEVEL CATA CRITERIA 100,000 AOL User drop IMPACT LOSS OF DATA AND CONNECTIVITY NETWORK IP SUBNETWORK UUNET CORE NETWORK STATE VA CITY ASHBURN COUNTRY USA
OUTAGE START DATE 10-03-2002 OUTAGE START TIME 12:29:00 GMT OUTAGE END DATE none OUTAGE END TIME none GMT DURATION EQUIPMENT N/A CAUSE OF OUTAGE N/A CORRECTIVE ACTION TICKET SOURCE REMEDY TICKET NUMBER 60138 LEC/OCC TICKET PVC/CKT Affected
WEBSITE oasis.wcomnet.com
COMMENTS
10-03-2002 12:52 GMT UUNET NOC has identified a catastrophic outage. Multiple routers across the core network are unreachable. TAC and NOC engineers working to isolate the problem. Information bridge VNET # 211-5675 PIN # 236044
10-03-2002 13:29 GMT UUNET reports 1462 dedicated customers affected. Senior TAC engineers still working to isolate/resolve the outage.
10-03-2002 14:13 GMT UUNET NOC reports 13 routers and 240 customers are still down. NOC and TAC still working to restore those devices and customers. No firm reason for outage.
10-03-2002 14:49 GMT UUNET NOC reports continuing to reload CISCO routers to restore line cards. Approximately 500 T1's and 100 multi-meg customers are down.
10-03-2002 15:25 GMT UUNET NOC is continuing to troubleshoot disabled line cards. At this time there are 3 GW routers and 5 BR routers that are down. There are 332 T1 customers and 60 multi-meg customers that are affected.
AOL has reported 15,000 new user drops since the last update. CISCO and Juniper representatives are involved in the troubleshooting process.
10-03-2002 15:46 GMT UUNET NOC reports they are continuing to work with Cisco to determine the cause of the line card outages. Dedicated customers at the T1 and Multi-Meg level continue to lose connectivity.
10-03-2002 16:08 GMT UUNET NOC reports currently have 97 Multi-Meg and 824 T1 customers down as line cards continue to become disabled. Ashburn NOC reloading routers as necessary to restore service. 11 GW's and 9 BR's are down. Filters are in place to capture crash information in order to provide core dumps to vendor representatives to isolate reason for outage.
10-03-2002 16:54 GMT UUNET NOC reports they continue to gather card crash information from the various filters set throughout the network. 8158 T1 customers down, 360 Multi-Meg, 14 BR's and 45 GW's are down. Current strategy is to gather more info from filters to isolate reason for outage.
10-03-2002 17:43 GMT UUNET NOC reports they are working with the vendor on the router crash data. They are also reviewing route updates from 12:00 GMT to identify a possible bad route update that may be causing the line cards to become disabled. There are 608 Multi-meg customers, 10 BRs and 34 GWs down at this time.
10-03-2002 18:32 GMT UUNET NOC reports the network has been stable for approximately 50 minutes. The NOC is working on restoring 11 BRs and 22 GWs. There are approximately 2800 T1 and 326 Multi-Meg customers that remain down.
Slashdot Mirror
apologies if this was posted earler... from:http://www.dotcomscoop.com/worldcom.html#outa ge
LATEST NEWS
Thursday October 3, 2002 @ 4:31 PM EDT
Have fun with this one:
FLASH SUMMARY
FLASH NUMBER 20021003023
ORGANIZATION UUNET
SEVERITY LEVEL CATA
CRITERIA 100,000 AOL User drop
IMPACT LOSS OF DATA AND CONNECTIVITY
NETWORK IP
SUBNETWORK UUNET CORE NETWORK
STATE VA
CITY ASHBURN
COUNTRY USA
OUTAGE START DATE 10-03-2002
OUTAGE START TIME 12:29:00 GMT
OUTAGE END DATE none
OUTAGE END TIME none GMT
DURATION
EQUIPMENT N/A
CAUSE OF OUTAGE N/A
CORRECTIVE ACTION
TICKET SOURCE REMEDY
TICKET NUMBER 60138
LEC/OCC TICKET
PVC/CKT Affected
WEBSITE oasis.wcomnet.com
COMMENTS
10-03-2002 12:52 GMT
UUNET NOC has identified a catastrophic outage. Multiple routers across the core network are unreachable. TAC and NOC engineers working to isolate the problem. Information bridge VNET # 211-5675 PIN # 236044
10-03-2002 13:29 GMT
UUNET reports 1462 dedicated customers affected. Senior TAC engineers still working to isolate/resolve the outage.
10-03-2002 14:13 GMT
UUNET NOC reports 13 routers and 240 customers are still down. NOC and TAC still working to restore those devices and customers. No firm reason for outage.
10-03-2002 14:49 GMT
UUNET NOC reports continuing to reload CISCO routers to restore line cards. Approximately 500 T1's and 100 multi-meg customers are down.
10-03-2002 15:25 GMT
UUNET NOC is continuing to troubleshoot disabled line cards. At this time there are 3 GW routers and 5 BR routers that are down. There are 332 T1 customers and 60 multi-meg customers that are affected.
AOL has reported 15,000 new user drops since the last update. CISCO and Juniper representatives are involved in the troubleshooting process.
10-03-2002 15:46 GMT
UUNET NOC reports they are continuing to work with Cisco to determine the cause of the line card outages. Dedicated customers at the T1 and Multi-Meg level continue to lose connectivity.
10-03-2002 16:08 GMT
UUNET NOC reports currently have 97 Multi-Meg and 824 T1 customers down as line cards continue to become disabled. Ashburn NOC reloading routers as necessary to restore service. 11 GW's and 9 BR's are down. Filters are in place to capture crash information in order to provide core dumps to vendor representatives to isolate reason for outage.
10-03-2002 16:54 GMT
UUNET NOC reports they continue to gather card crash information from the various filters set throughout the network. 8158 T1 customers down, 360 Multi-Meg, 14 BR's and 45 GW's are down. Current strategy is to gather more info from filters to isolate reason for outage.
10-03-2002 17:43 GMT
UUNET NOC reports they are working with the vendor on the router crash data. They are also reviewing route updates from 12:00 GMT to identify a possible bad route update that may be causing the line cards to become disabled. There are 608 Multi-meg customers, 10 BRs and 34 GWs down at this time.
10-03-2002 18:32 GMT
UUNET NOC reports the network has been stable for approximately 50 minutes. The NOC is working on restoring 11 BRs and 22 GWs. There are approximately 2800 T1 and 326 Multi-Meg customers that remain down.