Users Report of Nationwide CenturyLink Internet Outage (ktvb.com)

Are you having trouble accessing your CenturyLink internet today? Well, you're not alone. From a report: Several Treasure Valley residents reached out to KTVB Thursday morning to report they had no access to their CenturyLink internet or in some cases, their phone services. Some KTVB employees experienced the outage as well. A KTVB employee who uses the internet service provider called CenturyLink customer service. The customer service associate said they are aware of the outage for the 208 area code and technicians are working on a fix. He added there were similar outages in other states across the country. Downdetector.com indicated issues began being reported a little before 2 a.m. MST.

As far as a cause, the CenturyLink representative could not comment. The associate said the outage could last anywhere from 24 to 48-hours. CenturyLink released a statement to Newsweek explaining that its network "is experiencing a disruption affecting customer services." "We know how important these services are to our customers and we are working to restore services as quickly as possible," CenturyLink's statement continued.

Comments

On-site recovery needed in DCs

[Etcetera]

Per one of the better Reddit threads on this, the internal issue was pretty much the worst-case scenario for losing access to everything everywhere at once, and without alternate mechanisms for getting into places:

As of 2021 GMT

Tier IV Equipment Vendor Technical Support continues to work with CenturyLink Field Operations and Engineering to restore visibility and apply the filter to devices in Atlanta, GA and Chicago, IL. While those efforts are ongoing additional logs have been pulled from the devices in Kansas City, MO and New Orleans, LA following the restoral of visibility and the necessary filter application to obtain additional pertinent information now that the device is remotely accessible.

As of 1916 GMT

Efforts to regain visibility to sites in Atlanta, GA and Chicago, IL remain ongoing. Once visibility has been restored the filter will be applied to limit communication traffic between sites which was causing CPU spikes that in turn prevented the devices from functioning properly.

As of 1828 GMT

On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.

they have no DNSSEC & other problems

[johnjones]

centurylink.com has no DNSSEC
centurylink.com have insecure cipher suite on their mail servers : IDEA-CBC-SHA
centurylink.com does not support DKIM records

  centurylink router config has no rpki for bgp so can be hijacked

they do however have a status page : https://status.ctl.io/