Slashdot Mirror


Users Report of Nationwide CenturyLink Internet Outage (ktvb.com)

Are you having trouble accessing your CenturyLink internet today? Well, you're not alone. From a report: Several Treasure Valley residents reached out to KTVB Thursday morning to report they had no access to their CenturyLink internet or in some cases, their phone services. Some KTVB employees experienced the outage as well. A KTVB employee who uses the internet service provider called CenturyLink customer service. The customer service associate said they are aware of the outage for the 208 area code and technicians are working on a fix. He added there were similar outages in other states across the country. Downdetector.com indicated issues began being reported a little before 2 a.m. MST.

As far as a cause, the CenturyLink representative could not comment. The associate said the outage could last anywhere from 24 to 48-hours. CenturyLink released a statement to Newsweek explaining that its network "is experiencing a disruption affecting customer services." "We know how important these services are to our customers and we are working to restore services as quickly as possible," CenturyLink's statement continued.

87 comments

  1. Re:Democrats posing as Russians did it! by Anonymous Coward · · Score: 0

    Republicans posing as humans did it!

  2. I'm in the effected area... by Anonymous Coward · · Score: 0

    I'd post something witty, but I'm located in the effected area.
    Hopefully, they'll find out who tripped over the extension cord.

    Oh, as a fallback I using my "Smoke Signal" modem to post...

    CAP === 'electro'

    1. Re:I'm in the effected area... by Anonymous Coward · · Score: 0
    2. Re:I'm in the effected area... by Anonymous Coward · · Score: 0

      CenturyLink = utter crap. At least where I live.

      They won't run anything faster than 1.5 megabits out to my area, and they expect a whopping 50 bucks a month for that. They offer a discount saying that that exact same discount can be renewed each year, but when you call to renew it the next year they say that they are no longer offering that discount, and you can get it for 45 a month if you bundle with a bunch of crap you will never use.

      The only reason I put up with their shit at all was because I had no other choice. Then, eventually, a new option moved in. Comcast.

      Frying pan, fire.

      Ugh.

    3. Re:I'm in the effected area... by Anonymous Coward · · Score: 0

      Why frying pan, fire? Comcast likely will offer you something a couple of orders of magnitude faster than what you currently have. That's a bigger difference than between what you have now and a 28.8 phone modem.

    4. Re:I'm in the effected area... by Anonymous Coward · · Score: 1

      AFFECTED area, not effected. Sheesh

    5. Re: I'm in the effected area... by Anonymous Coward · · Score: 1

      yeah, thatâ(TM)s the problem, millenials canâ(TM)t spell for a goddamn thing so someone misspelled

      router bgp

      as

      write erase

    6. Re:I'm in the effected area... by Anonymous Coward · · Score: 0

      Because Comcast is monstrously horrible in how it treats its clients.

      If I had a gun and only two bullets, and I was locked in a room with Hitler, Osama Bin Laden, and Comcast, I'd shoot Comcast. Twice.

    7. Re: I'm in the effected area... by Anonymous Coward · · Score: 0

      And then scripted it to run on every router in the company without testing first :)

    8. Re:I'm in the effected area... by Anonymous Coward · · Score: 0

      Bummer. Where I live in Vancouver, WA I'm getting 40Mbps for $40 a month.

      Although it was hard to read the comics this morning when they were having there problem.

    9. Re: I'm in the effected area... by mermeid007 · · Score: 1

      QA is for wussies :)

    10. Re: I'm in the effected area... by Anonymous Coward · · Score: 0

      Affected.

    11. Re: I'm in the effected area... by Anonymous Coward · · Score: 0

      Disable IPv6? In Windows it's trivial enough just to uncheck the IPv6 checkbox (at least on Windows before Windows 10), then it's IPv4 only and not dual link or trying for an exclusive IPv6 connection. Other devices may vary, I just did troubleshooting on occasional failed to connect until reboot every time I got a new IP (as shown by a new lease in IP config after reboot/reconnect) and/or the router changed its channel... and I'm not on CenturyLink.

    12. Re: I'm in the effected area... by Anonymous Coward · · Score: 0

      QA is for wussies :)

      Slow down there Microsoft!

    13. Re:I'm in the effected area... by omnichad · · Score: 1

      I'd rather have good speeds and horrible customer service than bad speeds and horrible customer service. You must not have ever dealt with CenturyLink before.

    14. Re: I'm in the effected area... by omnichad · · Score: 1

      millenials canâ(TM)t spell for a goddamn thing

      At least Millennials know how to keep their Unicode from leaking out and spilling all over Slashdot.

    15. Re: I'm in the effected area... by brasscount · · Score: 1

      +1 Witty BUT PROBABLY TRUE

      --
      Confidentiality, Integrity, Availability: without Availability the other two are assured, as is Bankruptcy.
  3. It's the NSA by Anonymous Coward · · Score: 0

    They're trying to keep everyone from selling off their stock.

  4. Doesn't seem nationwide by Anonymous Coward · · Score: 0

    I'm a Century Link customer in Oregon, no issues at all.

    1. Re: Doesn't seem nationwide by Anonymous Coward · · Score: 0

      How good a customer are you?

    2. Re: Doesn't seem nationwide by Anonymous Coward · · Score: 0

      A very very bad customer. He/she actually has counterfeit Century Link service; however, since it's up, it's better than the real deal.

    3. Re:Doesn't seem nationwide by Anonymous Coward · · Score: 0

      Your not in the Portland area then because it was affected.

    4. Re:Doesn't seem nationwide by Anonymous Coward · · Score: 0

      packer nation has been unaffected as well.

  5. On-site recovery needed in DCs by Etcetera · · Score: 5, Informative

    Per one of the better Reddit threads on this, the internal issue was pretty much the worst-case scenario for losing access to everything everywhere at once, and without alternate mechanisms for getting into places:

    As of 2021 GMT

    Tier IV Equipment Vendor Technical Support continues to work with CenturyLink Field Operations and Engineering to restore visibility and apply the filter to devices in Atlanta, GA and Chicago, IL. While those efforts are ongoing additional logs have been pulled from the devices in Kansas City, MO and New Orleans, LA following the restoral of visibility and the necessary filter application to obtain additional pertinent information now that the device is remotely accessible.

    As of 1916 GMT

    Efforts to regain visibility to sites in Atlanta, GA and Chicago, IL remain ongoing. Once visibility has been restored the filter will be applied to limit communication traffic between sites which was causing CPU spikes that in turn prevented the devices from functioning properly.

    As of 1828 GMT

    On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.

    1. Re: On-site recovery needed in DCs by Anonymous Coward · · Score: 0

      And how much of that is just bullspit? I cannot wait for new service

    2. Re: On-site recovery needed in DCs by Anonymous Coward · · Score: 0

      A lot of high level language which, having worked in a Carrier NOC and in Engineering support roles, seems to indicate that either they had several core routers or firewalls get compromised, or someone really fucked up some routing tables.

    3. Re: On-site recovery needed in DCs by Anonymous Coward · · Score: 0

      I read a lot of this stuff and I see the stray typos and mods, crossouts, redlines, various things that one might consider imperfections in the language but I just think these are all how this document got to be the way it is right now

  6. DNS by Anonymous Coward · · Score: 0

    There was a lot of slowness until about noon today, when I seemed to lose access entirely.
    But it turned out most of the problem was just using CenturyLink's DNS. When I switched to 8.8.8.8, everything started working again.
    Obviously that's not a long-term solution, but I'm telecommuting today and tomorrow.
    Fortunately, UTOPIA is coming to my area in the fall. I'm almost free from CenturyLink!

    1. Re: DNS by Anonymous Coward · · Score: 0

      That's the only good thing about Utah. All the rest isn't

    2. Re: DNS by Anonymous Coward · · Score: 0

      Where is Jessica Hyde?

  7. Are you having trouble accessing your CenturyLink? by Anonymous Coward · · Score: 0

    Not at all, I don't even know what that is.

    And I don't care.

  8. It's cable TV... Are you not being entertained? by Anonymous Coward · · Score: 0

    so here is why Cable TV is different than Telephone..
    Their whole mindset is focused on entertainment - worst comes to worst, they have to credit your bill for a day's service if it was down more than 24 hrs. Someone's annual metrics will be affected.

    They do not have a "safety of life" aspect to the service internalized - it never existed. Your being able to call 911 makes no difference. While the traditional landline telcos are getting pretty lame, and heading rapidly towards the entertainment model - there's hundreds of thousands of people within the company who have decades of "keep the dial tone on" thinking.

    1. Re:It's cable TV... Are you not being entertained? by Anonymous Coward · · Score: 0

      CenturyLink is not a cable TV company. It's a phone company.
      Around here CenturyLink used to be QWest used to be USWest used to be Mountain Bell.

    2. Re:It's cable TV... Are you not being entertained? by Anonymous Coward · · Score: 0

      Yeah, and it doesn't look like any of the old Mountain Bell states are affected.

    3. Re:It's cable TV... Are you not being entertained? by Anonymous Coward · · Score: 0

      How did you make that mistake? TFS mentions only Idaho.

  9. Re: Are you having trouble accessing your CenturyL by Anonymous Coward · · Score: 0

    Why should you? I also called and they confirmed what this article says. Resolution in 24-48 hours

  10. Re: It's cable TV... Are you not being entertained by Anonymous Coward · · Score: 0

    Oh the only people complaining are elderly women knitting or sewing on a couch

  11. Re: It's cable TV... Are you not being entertaine by Anonymous Coward · · Score: 0

    It is garbage =)

  12. Re: It's cable TV... Are you not being entertained by Anonymous Coward · · Score: 0

    How did they get away from naming these phone conpanies sensible names

  13. Bigger than a ...... by niftymitch · · Score: 1

    Utah government is impacted.
    Problems from NY to Calif.

    Public utilities that should be reliable... Hmmm....
    Perhaps there is a busted link managed by a furlough fed DHS/IT contractor guy.

    --
    Truth is stranger than fiction, but it is because Fiction is obliged to stick to possibilities; Truth isn't. Mark Twain.
    1. Re: Bigger than a ...... by Anonymous Coward · · Score: 0

      Centurylink has all the appeal of a blue light special. You k ow why they (derp) call it a blue light special? Because it is exactly the right shade of blue to put a customer into a mode where they buy the very most expensive goods no matter what might try to distract them. And I can tell you Kmart is full of distractions, one more than the previous :)

    2. Re: Bigger than a ...... by Anonymous Coward · · Score: 0

      Centurylink has all the appeal of a blue light special. You k ow why they (derp) call it a blue light special? Because it is exactly the right shade of blue to put a customer into a mode where they buy the very most expensive goods no matter what might try to distract them. And I can tell you Kmart is full of distractions, one more than the previous :)

      Kmart is full of níggers. That is all.

  14. Re: It's cable TV... Are you not being entertained by zlives · · Score: 2

    don't worry soon to be renamed as Bell after they merge with Veri&T

  15. Re: Are you having trouble accessing your Century by Anonymous Coward · · Score: 0

    A helicopter hit a power line and power outages for three square blocks caused an internet hub to come down. I guess they should take a trip out west some day and see how these more robust cable companies are managing

  16. Maybe they need to click the ad by Anonymous Coward · · Score: 0

    https://arstechnica.com/tech-policy/2018/12/centurylink-blocks-internet-access-falsely-claims-state-law-required-it/

  17. Re:Democrats posing as Russians did it! by Anonymous Coward · · Score: 0

    Republicans should stop trying to act like Democrats

  18. they have no DNSSEC & other problems by johnjones · · Score: 5, Informative

    centurylink.com has no DNSSEC
    centurylink.com have insecure cipher suite on their mail servers : IDEA-CBC-SHA
    centurylink.com does not support DKIM records

      centurylink router config has no rpki for bgp so can be hijacked

    they do however have a status page : https://status.ctl.io/

  19. Keep calm by Anonymous Coward · · Score: 0

    At least nothing important depends on connectivity. Consider it a vacation and time for some more constructive thinking.

  20. TRUMP indicted. by Anonymous Coward · · Score: 0

    popcorn and wine on tap. sit back and enjoy.

    1. Re: TRUMP indicted. by Anonymous Coward · · Score: 0

      Mark Zuckerberg, Sundar Pichai, Kamala Harris, and Hillary Clinton will be sharing a cell in the gulag. Yay gulag!!1!

  21. If only by AHuxley · · Score: 0

    Some kind of network existed that did not place everything on the very end of the network.
    Some kind of ring, loop, circle like shape that kept people connected when one part of the network was not networking.

    --
    Domestic spying is now "Benign Information Gathering"
  22. Glad I don't work at teleNetwork anymore by RickyShade · · Score: 1

    I sure don't miss those days.

    1. Re:Glad I don't work at teleNetwork anymore by Anonymous Coward · · Score: 0

      You and me both. Nearly 2 decades at that place and I'm glad I am not part of this firestorm.

      Still wearing the shirt today though... :/

  23. Re: It's cable TV... Are you not being entertained by Anonymous Coward · · Score: 0

    It's affecting parts of Montana and Colorado, which were both old MT Bell in a past life. But just landline and dsl, not cellphones.... because their cell service in those areas actually uses towers which backhaul via other companies like Charter, Comcast, Verizon, and At&t.

  24. If your CenturyLink DSL goes down by 93+Escort+Wagon · · Score: 1

    Could you tell the difference?

    --
    #DeleteChrome
  25. Re: ... so be sure to check our website for update by Anonymous Coward · · Score: 0

    It is a fine website. Glad I got to see it. A lot of websites have a lot less attention to detail

  26. Yesterday by ArkiMage · · Score: 1

    Our service was fine today. They had a ~6hr outage yesterday though. This is for business fiber connectivity. We BGP peer with them and another carrier though, so no actual downtime.

    Their last ticket update:

    Our local repair group field technician found a dirty fiber connection at the 2nd optical patch panel in the Henderson, NC central office. The field technician cleaned the dirty fiber to clear the automatic power reduction alarm. We are showing two way traffic on the higher level interface has restored. We will hold our ticket for 24 hours to ensure stability.

  27. Wait, wait, let me get this right by Anonymous Coward · · Score: 0

    Slashdot now depends on Reddit for actual tech news?

    Holy shit

    1. Re:Wait, wait, let me get this right by Ungrounded+Lightning · · Score: 1

      Slashdot now depends on Reddit for actual tech news?

      No. The user "Etcetera" (14711) found and posted part of a relevant thread (and a link to the rest) that happened to be on Reddit.

      Treating "Slashdot" and all its users (except yourself?) as a monolith composed of identical bricks is an example of stereotyping. It clouds your thinking, and leads to silly posts.

      --
      Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
    2. Re: Wait, wait, let me get this right by negRo_slim · · Score: 2

      Look at this NPC lads, this is a quite advanced script.

      --
      On the Oregon Cost born and raised, On the beach is where I spent most of my days
    3. Re: Wait, wait, let me get this right by Anonymous Coward · · Score: 0

      Huh? Unfunny misapplication of a once-funny but already tired meme.

  28. Cause may be a backhoe incident by mjensen · · Score: 1

    I work on equipment that uses CenturyLink that stopped working today. I am told the issue is someone ran a backhoe which damaged their main fiber optics to their management system. CenturyLink offices have no internet to other offices, main databases are out.....

    1. Re:Cause may be a backhoe incident by Anonymous Coward · · Score: 0

      Hell, my home $40/m fiber connection with no install cost is on a redundant self-healing passive ring. Once the fiber leaves my block, it takes two paths. Requires two backhoes to take it out. How can an ISP be so incompetent to have their mission critical fiber not have properly implemented redundancy?

  29. CenturyLink Monopoly by WindowsStar · · Score: 1

    I hope that this outage cost Google, Facebook and every other billion dollar company 10 billion in lost revenue. Now they can now all go to the government and force a competitive ISP nationwide. This way we will have a choice. Since everything is CenturyLink we lost, Comcast, CableOne, Verizon, AT&T and CellularOne services. WHY??? because they all have to use CenturyLink for their connections. All these companies have no access to the Internet but to use CenturyLink. This monopoly needs to be fixed.

    1. Re:CenturyLink Monopoly by jpaine619 · · Score: 1

      Since everything is CenturyLink we lost, Comcast, CableOne, Verizon, AT&T and CellularOne services. WHY???

      Um, not that I disagree with your post, but it's a bit confusing. Are you suggesting that AT&T uses CenturyLink for connectivity? If not, my apologies. But if so, then you are way off base. AT&T is a Tier 1 provider. They have interconnect agreements with other networks, but they own fiber running to every continent. I don't recall if that includes Antarctica. So yeah.. I didn't have any issues with my AT&T fiber..

    2. Re:CenturyLink Monopoly by WindowsStar · · Score: 1

      Yes, many people do not know that many companies use CenturyLink's Fiber for their connections. This is bad for everyone because there is no real option for Internet. If you use AT&T or Comcast etc. in reality they are on CenturyLink so you are using CenturyLink. This monopoly has to stop. It would be nice to have 2 or 3 REAL alternative connections. Other ISP companies with their own network without having to rely on another. I was on the phone with all of them today asking what was wrong. Everyone of them said it is a CenturyLink issue and we are waiting on them. All our cellphones were down, and all networks were down, everything down for 9 hours today.

    3. Re: CenturyLink Monopoly by Anonymous Coward · · Score: 0

      There are other peers between AT&T's networks. CenturyLink is probably just the one expected to be able to handle the traffic. Used to be more peers before CenturyLink bought them up.

      It is like passing through towns while driving on the highway, except those "towns" may or may not be owned by the same company. AT&T -> CenturyLink - > Comcast -> Google (YouTube).

    4. Re:CenturyLink Monopoly by jpaine619 · · Score: 1

      Uh.. No guy.. AT&T does not use CenturyLink's fiber. I was an AT&T Construction Splicer for a decade. I've work on transcontinental trunks and small distribution fibers.. I can assure you that as a Tier 1 Provider they use their own fiber. As I mentioned before, they have interconnects and peering agreements with other providers, but that's how the whole internet works.. you have to connect to other systems. I can run a traceroute from San Diego to NY and it'll never leave AT&T's network..

    5. Re: CenturyLink Monopoly by Anonymous Coward · · Score: 0

      Oh, yes it does. Yes, AT&T indeed uses Verizon, Centurylink, lighthouse, zayo, and any other carrier for last mile fiber, or short interconnects at various locations.

    6. Re: CenturyLink Monopoly by jpaine619 · · Score: 1

      No, it does not. It most certainly does not. Your information is in error or you do not understand how the internet works. It is vastly more likely that it is the other way around. i.e. A CenturyLink customer may connect to a remote network and the data travels across AT&T (or Level 3 for example) network for the bulk of the trip and then hits CenturyLink for the last mile.

    7. Re: CenturyLink Monopoly by WindowsStar · · Score: 1

      I worked for CenturyLink & AT&T years back and I still have many engineer friends that work there. They "CenturyLink" are a monopoly. I have worked in this industry for 50 years and I know exactly how it is setup. While it is true that AT&T has its own Tier 1 Fiber the break up years ago caused them to lose a lot. You will find them on their own infrastructure in and around big cities, but once you are in the southwest that is no linger true. Please note a traceroute is NOT an exact way to see if traffic is routed the way you think. Core equipment is hidden from traceroutes, hand offs are hidden, firewalls are hidden, so you don't know exactly where the traffic really goes.

    8. Re: CenturyLink Monopoly by jpaine619 · · Score: 1

      The breakup occurred on January 1, 1984. That was before fiber was in any sort of common use, so they didn't lose anything. Besides, any fiber already in the ground would have belonged to AT&T Long Lines, which is the company that became AT&T. Long lines is the ONLY thing any fiber in the ground was being used for at that time. It was used for transcontinental trunk lines. It was too expensive to be running around cities for for local exchange distribution.

      AT&T isn't just a Tier 1 provider, they were the first. They do not rely on CenturyLink. They have peering agreements and interconnects. Whatever your beef with them, you're talking about shit you do not know as well as you think you know.

  30. Re: ... so be sure to check our website for update by Anonymous Coward · · Score: 0

    TDS in every. single. thread.

    This might require some medication in the water supply to clear up.

  31. Re: by Anonymous Coward · · Score: 0

    Old POTS telephones had 5x9s uptime due to government regulation. Now the government is unable to impose any measure of minimum quality on the internet which also affects telephone communication through VOIP use. The reason is inflation raising the cost of having reliable services, to the point that the US is unable to afford having the infrastructure (and education and health-care) it was able to have in the past.

  32. SS7 links down as well by Anonymous Coward · · Score: 0

    SS7 connectivity was/is down as well. No Long distance calling from Alaska. We lost OC3 connectivity at our interconnect in Seattle.

    1. Re:SS7 links down as well by Anonymous Coward · · Score: 0

      OC3.. Impressive in 1990.. Pathetic now. I have an OC-24.. Suck on that, bitch.

    2. Re: SS7 links down as well by Anonymous Coward · · Score: 0

      Go back to Faceboot, petulant child.

  33. 48 hours!? by jpaine619 · · Score: 1

    Holy shit.. That's a long time to get internet connectivity back up. This isn't 1990.. internet is not a trivial service anymore.. Entire industries require it for supply / distribution / ordering / etc.. etc..

    1. Re:48 hours!? by Anonymous Coward · · Score: 0

      So called Engineers are working on it. I bet 99% are technicians and there is a file note or email about single point of failure. As Christmas - must be an authorised change. If so fire the numbnut that approved it with no recovery plan.

    2. Re:48 hours!? by thegarbz · · Score: 1

      You should see how long it took to get electricity up in the United States of America's territory of Puerto Rico.

  34. Re:Democrats posing as Russians did it! by Anonymous Coward · · Score: 0

    Republicans should stop trying to act like Democrats

    What, and turn liberal?! I don't think so!

  35. Re: by astrofurter · · Score: 1

    That's not inflation - that's public policy.

  36. Since when is Treasure Valley considered nationwid by Anonymous Coward · · Score: 0

    Title of this story claims a nationwide outage.... actual article talks about an outage at a very specific area code in Treasure Valley.

  37. Re:Since when is Treasure Valley considered nation by Anonymous Coward · · Score: 0

    The site linked is a local TV station that covers mainly the Treasure Valley. I don't know how you clicked on the link and read Treasure Valley but didn't read the words right next to it that said "and across the U.S.". Also, that specific area code services the entirety of the state.