Slashdot Mirror

← Back to Stories (view on slashdot.org)

Users Report of Nationwide CenturyLink Internet Outage (ktvb.com)

Posted by msmash on from the outage-reports dept.

Are you having trouble accessing your CenturyLink internet today? Well, you're not alone. From a report: Several Treasure Valley residents reached out to KTVB Thursday morning to report they had no access to their CenturyLink internet or in some cases, their phone services. Some KTVB employees experienced the outage as well. A KTVB employee who uses the internet service provider called CenturyLink customer service. The customer service associate said they are aware of the outage for the 208 area code and technicians are working on a fix. He added there were similar outages in other states across the country. Downdetector.com indicated issues began being reported a little before 2 a.m. MST.

As far as a cause, the CenturyLink representative could not comment. The associate said the outage could last anywhere from 24 to 48-hours. CenturyLink released a statement to Newsweek explaining that its network "is experiencing a disruption affecting customer services." "We know how important these services are to our customers and we are working to restore services as quickly as possible," CenturyLink's statement continued.

26 of 87 comments (clear)

  1. On-site recovery needed in DCs by Etcetera · · Score: 5, Informative

    Per one of the better Reddit threads on this, the internal issue was pretty much the worst-case scenario for losing access to everything everywhere at once, and without alternate mechanisms for getting into places:

    As of 2021 GMT

    Tier IV Equipment Vendor Technical Support continues to work with CenturyLink Field Operations and Engineering to restore visibility and apply the filter to devices in Atlanta, GA and Chicago, IL. While those efforts are ongoing additional logs have been pulled from the devices in Kansas City, MO and New Orleans, LA following the restoral of visibility and the necessary filter application to obtain additional pertinent information now that the device is remotely accessible.

    As of 1916 GMT

    Efforts to regain visibility to sites in Atlanta, GA and Chicago, IL remain ongoing. Once visibility has been restored the filter will be applied to limit communication traffic between sites which was causing CPU spikes that in turn prevented the devices from functioning properly.

    As of 1828 GMT

    On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.

    --
    Hire a Linux system administrator, systems engineer,
  2. Re:I'm in the effected area... by Anonymous Coward · · Score: 1

    AFFECTED area, not effected. Sheesh

  3. Re: I'm in the effected area... by Anonymous Coward · · Score: 1

    yeah, thatâ(TM)s the problem, millenials canâ(TM)t spell for a goddamn thing so someone misspelled

    router bgp

    as

    write erase

  4. Bigger than a ...... by niftymitch · · Score: 1

    Utah government is impacted.
    Problems from NY to Calif.

    Public utilities that should be reliable... Hmmm....
    Perhaps there is a busted link managed by a furlough fed DHS/IT contractor guy.

    --
    Truth is stranger than fiction, but it is because Fiction is obliged to stick to possibilities; Truth isn't. Mark Twain.
  5. Re: It's cable TV... Are you not being entertained by zlives · · Score: 2

    don't worry soon to be renamed as Bell after they merge with Veri&T

  6. they have no DNSSEC & other problems by johnjones · · Score: 5, Informative

    centurylink.com has no DNSSEC
    centurylink.com have insecure cipher suite on their mail servers : IDEA-CBC-SHA
    centurylink.com does not support DKIM records

      centurylink router config has no rpki for bgp so can be hijacked

    they do however have a status page : https://status.ctl.io/

  7. Glad I don't work at teleNetwork anymore by RickyShade · · Score: 1

    I sure don't miss those days.

  8. If your CenturyLink DSL goes down by 93+Escort+Wagon · · Score: 1

    Could you tell the difference?

    --
    #DeleteChrome
  9. Yesterday by ArkiMage · · Score: 1

    Our service was fine today. They had a ~6hr outage yesterday though. This is for business fiber connectivity. We BGP peer with them and another carrier though, so no actual downtime.

    Their last ticket update:

    Our local repair group field technician found a dirty fiber connection at the 2nd optical patch panel in the Henderson, NC central office. The field technician cleaned the dirty fiber to clear the automatic power reduction alarm. We are showing two way traffic on the higher level interface has restored. We will hold our ticket for 24 hours to ensure stability.

  10. Re:Wait, wait, let me get this right by Ungrounded+Lightning · · Score: 1

    Slashdot now depends on Reddit for actual tech news?

    No. The user "Etcetera" (14711) found and posted part of a relevant thread (and a link to the rest) that happened to be on Reddit.

    Treating "Slashdot" and all its users (except yourself?) as a monolith composed of identical bricks is an example of stereotyping. It clouds your thinking, and leads to silly posts.

    --
    Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
  11. Cause may be a backhoe incident by mjensen · · Score: 1

    I work on equipment that uses CenturyLink that stopped working today. I am told the issue is someone ran a backhoe which damaged their main fiber optics to their management system. CenturyLink offices have no internet to other offices, main databases are out.....

  12. CenturyLink Monopoly by WindowsStar · · Score: 1

    I hope that this outage cost Google, Facebook and every other billion dollar company 10 billion in lost revenue. Now they can now all go to the government and force a competitive ISP nationwide. This way we will have a choice. Since everything is CenturyLink we lost, Comcast, CableOne, Verizon, AT&T and CellularOne services. WHY??? because they all have to use CenturyLink for their connections. All these companies have no access to the Internet but to use CenturyLink. This monopoly needs to be fixed.

    1. Re:CenturyLink Monopoly by jpaine619 · · Score: 1

      Since everything is CenturyLink we lost, Comcast, CableOne, Verizon, AT&T and CellularOne services. WHY???

      Um, not that I disagree with your post, but it's a bit confusing. Are you suggesting that AT&T uses CenturyLink for connectivity? If not, my apologies. But if so, then you are way off base. AT&T is a Tier 1 provider. They have interconnect agreements with other networks, but they own fiber running to every continent. I don't recall if that includes Antarctica. So yeah.. I didn't have any issues with my AT&T fiber..

    2. Re:CenturyLink Monopoly by WindowsStar · · Score: 1

      Yes, many people do not know that many companies use CenturyLink's Fiber for their connections. This is bad for everyone because there is no real option for Internet. If you use AT&T or Comcast etc. in reality they are on CenturyLink so you are using CenturyLink. This monopoly has to stop. It would be nice to have 2 or 3 REAL alternative connections. Other ISP companies with their own network without having to rely on another. I was on the phone with all of them today asking what was wrong. Everyone of them said it is a CenturyLink issue and we are waiting on them. All our cellphones were down, and all networks were down, everything down for 9 hours today.

    3. Re:CenturyLink Monopoly by jpaine619 · · Score: 1

      Uh.. No guy.. AT&T does not use CenturyLink's fiber. I was an AT&T Construction Splicer for a decade. I've work on transcontinental trunks and small distribution fibers.. I can assure you that as a Tier 1 Provider they use their own fiber. As I mentioned before, they have interconnects and peering agreements with other providers, but that's how the whole internet works.. you have to connect to other systems. I can run a traceroute from San Diego to NY and it'll never leave AT&T's network..

    4. Re: CenturyLink Monopoly by jpaine619 · · Score: 1

      No, it does not. It most certainly does not. Your information is in error or you do not understand how the internet works. It is vastly more likely that it is the other way around. i.e. A CenturyLink customer may connect to a remote network and the data travels across AT&T (or Level 3 for example) network for the bulk of the trip and then hits CenturyLink for the last mile.

    5. Re: CenturyLink Monopoly by WindowsStar · · Score: 1

      I worked for CenturyLink & AT&T years back and I still have many engineer friends that work there. They "CenturyLink" are a monopoly. I have worked in this industry for 50 years and I know exactly how it is setup. While it is true that AT&T has its own Tier 1 Fiber the break up years ago caused them to lose a lot. You will find them on their own infrastructure in and around big cities, but once you are in the southwest that is no linger true. Please note a traceroute is NOT an exact way to see if traffic is routed the way you think. Core equipment is hidden from traceroutes, hand offs are hidden, firewalls are hidden, so you don't know exactly where the traffic really goes.

    6. Re: CenturyLink Monopoly by jpaine619 · · Score: 1

      The breakup occurred on January 1, 1984. That was before fiber was in any sort of common use, so they didn't lose anything. Besides, any fiber already in the ground would have belonged to AT&T Long Lines, which is the company that became AT&T. Long lines is the ONLY thing any fiber in the ground was being used for at that time. It was used for transcontinental trunk lines. It was too expensive to be running around cities for for local exchange distribution.

      AT&T isn't just a Tier 1 provider, they were the first. They do not rely on CenturyLink. They have peering agreements and interconnects. Whatever your beef with them, you're talking about shit you do not know as well as you think you know.

  13. Re: I'm in the effected area... by mermeid007 · · Score: 1

    QA is for wussies :)

  14. 48 hours!? by jpaine619 · · Score: 1

    Holy shit.. That's a long time to get internet connectivity back up. This isn't 1990.. internet is not a trivial service anymore.. Entire industries require it for supply / distribution / ordering / etc.. etc..

    1. Re:48 hours!? by thegarbz · · Score: 1

      You should see how long it took to get electricity up in the United States of America's territory of Puerto Rico.

  15. Re: Wait, wait, let me get this right by negRo_slim · · Score: 2

    Look at this NPC lads, this is a quite advanced script.

    --
    On the Oregon Cost born and raised, On the beach is where I spent most of my days
  16. Re: by astrofurter · · Score: 1

    That's not inflation - that's public policy.

  17. Re:I'm in the effected area... by omnichad · · Score: 1

    I'd rather have good speeds and horrible customer service than bad speeds and horrible customer service. You must not have ever dealt with CenturyLink before.

  18. Re: I'm in the effected area... by omnichad · · Score: 1

    millenials canâ(TM)t spell for a goddamn thing

    At least Millennials know how to keep their Unicode from leaking out and spilling all over Slashdot.

  19. Re: I'm in the effected area... by brasscount · · Score: 1

    +1 Witty BUT PROBABLY TRUE

    --
    Confidentiality, Integrity, Availability: without Availability the other two are assured, as is Bankruptcy.