Slashdot Mirror

← Back to Stories (view on slashdot.org)

CenturyLink's Nationwide Outage Affects Millions

Posted by timothy on from the experiencing-seems-a-gentle-verb dept.

halfEvilTech writes "CenturyLink, the nation's third largest telco network, is experiencing an outage of its broadband service nationwide, leaving its support systems overwhelmed and even causing its website to hit a few snags this morning. The company, which at last count has 5.8 million broadband subscribers, has no estimates yet on how long it will take to restore service." CenturyLink is the company that will be providing the Defense Department with the equivalent of Internet2.

55 of 105 comments (clear)

  1. Once upon a qwest by Reverand+Dave · · Score: 4, Insightful

    After CenturyLink "partnered" with Qwest they pretty much became a pariah in my book. Qwest was just a terrible terrible mess and apparently CenturyLink is keeping their spirit alive.

    --
    I got here through a series of tubes
    1. Re:Once upon a qwest by Anonymous Coward · · Score: 1

      NO management was lost at our location, and frankly there needs to be some that are cut. I don't see ANY noticeable improvement in ANYTHING under the new company.

    2. Re:Once upon a qwest by sl3xd · · Score: 4, Insightful

      It's Qwest's famouse Spirit of Service

      For CenturyLink, it was probably a good deal: They get to be a Tier-1 peer, instead of having to pay extortion fees like TIER-2 and 3.

      It was a very good deal for Qwest's customers. They went from being limited to 1.5 Mbit to being able to buy "up to" 40 Mbit...

      I dumped CenturyLink/Qwest long before then, but my brother supposedly got close to 30 Mbit measured.

      Like most telecom idiots, CenturyLink has a 12-month "introductory rate", and they won't negotiate. Since all of their competitors do the same, the practice has become switching networks every year, after the introductory rate expires. The same applies if you have Cable or Satellite TV; customers just switch every year for a lower rate.

      I really don't see how being so boneheaded helps either company, but that's telecom in the USA.

      --
      -- Sometimes you have to turn the lights off in order to see.
    3. Re:Once upon a qwest by Hatta · · Score: 4, Insightful

      Qwest was the only telco to refuse warrantless wiretaps during the Bush era. I was happily a Qwest customer until they got bought out by CenturyLink. I would switch immediately to any telco that guaranteed refusal of any unwarranted requests. Unfortunately, none exist.

      --
      Give me Classic Slashdot or give me death!
    4. Re:Once upon a qwest by Nyder · · Score: 1

      It's Qwest's famouse Spirit of Service

      For CenturyLink, it was probably a good deal: They get to be a Tier-1 peer, instead of having to pay extortion fees like TIER-2 and 3.

      It was a very good deal for Qwest's customers. They went from being limited to 1.5 Mbit to being able to buy "up to" 40 Mbit...

      I dumped CenturyLink/Qwest long before then, but my brother supposedly got close to 30 Mbit measured.

      Like most telecom idiots, CenturyLink has a 12-month "introductory rate", and they won't negotiate. Since all of their competitors do the same, the practice has become switching networks every year, after the introductory rate expires. The same applies if you have Cable or Satellite TV; customers just switch every year for a lower rate.

      I really don't see how being so boneheaded helps either company, but that's telecom in the USA.

      I had Qwest, then Centurylink took over and they kicked me off the service because I "download" too much.

      I was pissed, but in hindsight, i got the better of the deal.

      --
      Be seeing you...
    5. Re:Once upon a qwest by thomasw_lrd · · Score: 1

      CenturyLink jacked my rates up, but I called them, they gave me a new deal. What's surprising about it is, in my BFE town, they are the only game in town, unless I pay over $100 a month for cable plus internet from Suddenlink.

      --
      21st Century Renaissance Man
    6. Re:Once upon a qwest by ahoffer0 · · Score: 1

      I saved a bundle switching from Comcast to Century Link. I get 8 Mbps/down which is fine for my Netflix and other streaming needs. If the price goes up, I'll just switch back to Comcast on one of their introductory plans. Then again, in a year from now, it is likely that HSPA+ and LTE wireless broadband will be competitive with Comcast and Century link in terms of price for my personal bandwidth needs.

      It's a good time to be a customer.

      PS: I'm lucky. My Century Link connection is still up.

    7. Re:Once upon a qwest by bbcisdabomb · · Score: 1

      People tend to switch every 2 years, as that's when the contracts expire. Other than that, you're totally right.

      --
      Please put some pants on before you post again.
    8. Re:Once upon a qwest by Jawnn · · Score: 1

      Our data networking and VOIP service were spotless while QWest was running the show. After the move to CenturyLink customer service (which, to be fair, was never great) because a joke. I do not exaggerate, it took weeks to have simple issues resolved. Stability and reliability of their network has, clearly, also suffered, but it's the useless customer service that has us moving all of our traffic to a new carrier soon. Today's mess is just an underscore.

    9. Re:Once upon a qwest by Sperbels · · Score: 1

      Well, I've been nothing but happy with Qwest/Century Link. Have very fast DSL through them. My experience with Comcast has been: promised speeds that were never achieved (although, still generally faster than DSL), frequent service distruptions, and I feel like I'm talking to a car salesman whenever I talk to them. I recently had to go back to Comcast after moving. It took me 10 seconds to tell them exactly what I wanted, then it took another 10 minutes to tell them I didn't want all the extra stuff the kept trying to push on me. I'm not happy about being back with Comcast.

    10. Re:Once upon a qwest by hurfy · · Score: 1

      Can't tell the diference here....

      Outage didn't affect us...as far as i know...
      We were down due to local problems at the same time but the line next door worked.....until they fixed this line then the other went down...sigh. Somehow fixing and rebooting my line down the street killed the modem for next door...whatever.

      Generally been fine up until this year when they alternate breaking the dialtone and internet (never both,wtf) each month. 5 service calls this year between 2 accounts...not impressed.

    11. Re:Once upon a qwest by michaelmalak · · Score: 2

      Qwest was the only telco to refuse warrantless wiretaps during the Bush era.

      Actually, 1997, during the Clinton administration.

    12. Re:Once upon a qwest by Obfuscant · · Score: 1

      As an anecdote on the other side of the coin, Qwest installed my DSL on the wrong (second) phone line, and lied to me about getting a static IP to get me to sign up. They then lied to the PUC when I complained, saying they never told me any of that.

    13. Re:Once upon a qwest by Forever+Wondering · · Score: 1

      What about sonic.net? They may not be what you want, but they seem to come close:
      http://boingboing.net/2012/06/25/sonic-net-stopped-saving-logs.html

      --
      Like a good neighbor, fsck is there ...
    14. Re:Once upon a qwest by sl3xd · · Score: 1

      I suppose it depends on the area; the contract lengths for home Internet service are actually zero in my area - from both Comcast, and CenturyLink. The introductory rates are still good for 6 months to a year.

      State & local regulations, maybe?

      --
      -- Sometimes you have to turn the lights off in order to see.
    15. Re:Once upon a qwest by Reverand+Dave · · Score: 1

      When you have one master that punches and kicks you, then you get a master that just punches you, they do seem like a much more benevolent master despite the fact that they are both mistreating you, it's just a matter of degree.

      --
      I got here through a series of tubes
    16. Re:Once upon a qwest by airdweller · · Score: 1

      Same experience here. Comcast -> CL. 12Mbs. $30/m. No complaints so far.

  2. Not me by Tailhook · · Score: 1

    My DSL is up. It's either fixed or didn't involve my part of the system.

    CenturyLink (formerly Qwest in my case) has been very reliable during the past year. You get the rate you pay for as well.

    --
    Maw! Fire up the karma burner!
    1. Re:Not me by tilante · · Score: 1

      Mine was down this morning, but I can ping my home box and SSH to it now from work. Looks like it's fixed in my area (Tallahassee, Florida). My past experience is that CenturyLink is reliable - at least, more so than Comcast, which is my only other broadband option that doesn't involve satellite or the like.

    2. Re:Not me by jerk · · Score: 1

      Same here. I was streaming video last night as I went to bed and have been logged on to my VPN from work all morning. If it went down, it was between 11:30pm and 8am MST.

    3. Re:Not me by tipo159 · · Score: 1

      Mine is up as well. No sign that there was a problem today.

  3. Re:nobody posted yet... by Cenan · · Score: 2

    Apparently a wide one.

    --
    ... whatever ...
  4. No estimates by Rich0 · · Score: 5, Insightful

    Having dealt with having to provide estimates on service restoration at work, my experience is that by the time you can figure out what it would take to restore service, it won't be long before service is actually restored.

    I'm not saying that providing status updates isn't good practice. However, it is usually rare that you'll get an ETA on something being fixed. Maybe if they discover it is a broken line and they actually have to dig it up and fix it and that will take hours you might get an ETA. Usually root cause analysis is 95% of the work in problem solving.

    Reminds me of a story at work when some developers decided to actually try to embrace the outsourcing model that was being pushed by management. They sent a list of bugs to the outsourced development team and asked for estimates to fix them. They replied, "no problem, just tell us which lines of code to modify and how and we'll take care of it." Now, THAT is a value-add!

    1. Re:No estimates by geminidomino · · Score: 1

      Odd. When I called the support line (which I always do in the case of an outage, to see if the recording tells me it's a known issue) at around 9AM Eastern, the Estimate restoration was 7PM. Obviously, they beat that by several hours.

      Maybe it's a regional thing, but I usually get estimates, even if they do go a little heavy on the Scotty coefficient.

    2. Re:No estimates by Rich0 · · Score: 1

      Well, they can always make something up to keep you from calling back.

      A manager in our operations support group once explained the situation to some internal customers - if the system is down, do you want everybody available to be fixing the problem, or do you want them to spend their time telling people that it is under control? (Mind you, this was only for internal customers.) The ultimate decision was that for very short outages they'd just deal with it, and once it was taking more than 15min they'd start issuing communications. For a short outage few would even notice it in the first place or bother to call it in, and announcing it after the fact didn't really provide any value-add.

      Now, if we were contracting with an outside company we'd insist on knowing about all outages as part of our SLA - we'd care about them after the fact because we'd want to know if we're getting value for our money. Where internal customers are concerned the data would get reported to management, and they'd decide if the support group was doing a good job. The random company workers really didn't have decision rights over where the work was done - it isn't like we let individual departments contract for their own support.

  5. They promised the same rate f or 5 years ... by johnlcallaway · · Score: 2

    .. they didn't promise the same access!

    --
    I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
  6. "CenturyLink, the nation's third largest telco" by PolygamousRanchKid+ · · Score: 1

    My DSL is up. It's either fixed or didn't involve my part of the system.

    My DSL is up. It's either fixed or didn't involve my nation of the system.

    --
    Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
  7. Its not the whole united states . by Anonymous Coward · · Score: 3, Informative

    Western states in the former Qwest footprint seem to be up and running ok. East Coast Legacy Centurylink seems to be not having a good time of it though.

    Suffice it to say, the sky is not falling and Centurylink is not pulling a "u-verse" on it.

    1. Re:Its not the whole united states . by 0racle · · Score: 1

      My east coast former Embarq DSL has been working all day.

      --
      "I use a Mac because I'm just better than you are."
  8. DNS? by neghvar1 · · Score: 1

    I don’t know exactly what is causing the problem, but when I used to be on Time Warner, we once went down for 4 days due to a DNS issue. I just modified the router settings to go to Google’s DNS servers and my problem was solved.

    1. Re: DNS? by RussR42 · · Score: 1

      Here.

  9. Glad I switched to Comcast last year by rwyoder · · Score: 1

    I had CL DSL for over 10 years.
    I'm only 2600' from the CO, but the best they would sell me was 7mbs.
    So I switched to Comcast and get 22mbs, *and* native IPv6, for less than I was paying CL.

    1. Re:Glad I switched to Comcast last year by Anonymous Coward · · Score: 5, Funny

      "Glad I switched to Comcast last year"

      Those aren't words heard very often...

  10. My service near STL is up. by Anonymous Coward · · Score: 2, Informative

    I didn't notice any downtime.

    Centurylink keeps raising their prices and lowering their bandwidth. I started out at 25 mbit fiber to the house. Then it went to 20, then 15, finally 10.

    Prices went up in the meantime so I downgraded to 3 mbit. However they implement the cap seems to work worse than I'd expect. They also require a 1 year contract to get the "discount" rates and it has a $200 cancellation fee even though I don't lease a router and I've been a customer for 5 years.

    Overall the local office has top notch support people that are real humans. The national office... not so much.

  11. Glitch by rot26 · · Score: 1

    That always happens when they turn up the pipe to Bluffdale.

    --



    To ensure perfect aim, shoot first and call whatever you hit the target
  12. Perfect by ThatsNotPudding · · Score: 1

    CenturyLink is the company that will be providing the Defense Department with the equivalent of Internet2.

    Sleep Tight!

    1. Re:Perfect by Keruo · · Score: 1

      Maybe they were trying to implement security by air-gap and unplugged the wrong cord, accidentally achieving it.
      Security by accident - now there's new sales pitch for their sales team to use.

      --
      There are no atheists when recovering from tape backup.
  13. Re:nobody posted yet... by Anonymous Coward · · Score: 1

    FierceTelecom - "CenturyLink suffers Internet outage in 21 states"

    This issue started at 4AM 5/7/2013 and lasted until just a few moments ago, at least in the Tallahassee/North Florida region. The outage took out a good bit of their national network, not just the local region.

    Article - http://www.fiercetelecom.com/story/centurylink-wades-through-nationwide-internet-outage-customers-21-states-af/2013-05-07

    Video - http://cdn.l2net.com/content/video/CTLFL-Outage-BGP-05072013.mp4

  14. Re:Century Link by AmongTheBoulders · · Score: 1

    My 1.5 Mb Century-link DSL connection is also still working normally. I have not noticed any problems now or from earlier this morning.

  15. Ah, Centurylink by holmedog · · Score: 1

    I've used them for years and years (only broadband in my area). In general it's been a good service, but when they screw the pooch they go above and beyond. I'll never forgive them for leaving me with huge latency for 3 months. If I ever have a choice in broadband they will be gone.

    http://ask.slashdot.org/story/12/03/06/175223/ask-slashdot-what-is-an-acceptable-broadband-latency

  16. What outage? by rootmon · · Score: 1

    I have been working since 8 AM Eastern Time as a telecommuter and my CenturyLink DSL has been up without so much as an SSH session disconnecting all day. I live in SW Florida and my colleagues tell me they're having problems but perhaps the outage is not as widespread as publicized or it's affecting DNS and I use OpenDNS instead of my ISP's DNS for filtering sites I don't want my kids to browse such as adult content.

    In my experience, Cable modems were far less stable than DSL. I had Comcast for a while and it was much worse. So please don't listen to the cable modem trolls. Overall DSL is a more reliable technology and I've been using Spring/Embarq/CenturyLink for 12 years now with very few hiccups which they did address quickly, except for a move where I used DirectWay/Hughes and then Comcast for a while. DirectWay/Hughes was good if you have no other options but satellite just has too much latency for some apps like voip or skype, and Comcast was terrible especially during peak hours.

    --
    "As flies to the wanton boys are we to the gods; they kill us for sport." - William Shakespeare, King Lear
  17. Nationwide? by tyrione · · Score: 1

    Wow. Then I wonder what service I'm using if not CLink in the PNW? I'm not experiencing the outages so I guess `nationwide' is a bit over-the-top.

  18. Re:Century Link by Zibodiz · · Score: 1

    Same here. I had trouble accessing 1 website yesterday (pretty unlikely to be related), but I've had no trouble at all since. I am, however, on part of the original CenturyTel network, not on the Qwest wires. I'm guessing it's probably just one of their sub-companies that was affected.

  19. Re:nobody posted yet... by Penguinisto · · Score: 2

    Are the majority of /. readers using Centurylink?

    I used to (had no choice at a previous place I was renting out on the Oregon Coast) - service was crap but usable. However, it was their billing department who turned a 'first two months free!' promotion into a 'I'm sorry sir, but you owe us $126 before we can re-instate your service', in spite of never missing a payment. The nanosecond Charter showed up in the neighborhood a month later, I switched so fast that I could have almost not dropped a packet.

    All I can say is - never again. Compare Centurylink's $70/mo for 3.5 Mb/sec and maybe 70-75% uptime, with Charter's $30/mo 30 Mb/sec and 99.999% uptime.

    Fuck Centurylink.

    --
    Quo usque tandem abutere, Nimbus, patientia nostra?
  20. Re:nobody posted yet... by ttucker · · Score: 1

    Are the majority of /. readers using Centurylink?

    I used to (had no choice at a previous place I was renting out on the Oregon Coast) - service was crap but usable. However, it was their billing department who turned a 'first two months free!' promotion into a 'I'm sorry sir, but you owe us $126 before we can re-instate your service', in spite of never missing a payment. The nanosecond Charter showed up in the neighborhood a month later, I switched so fast that I could have almost not dropped a packet.

    All I can say is - never again. Compare Centurylink's $70/mo for 3.5 Mb/sec and maybe 70-75% uptime, with Charter's $30/mo 30 Mb/sec and 99.999% uptime.

    Fuck Centurylink.

    We went Cox Business and never looked back!

  21. Re:Century Link by sycodon · · Score: 1

    Seems to be fucking up a lot of news sites. They apparently host Ad servers.

    --
    When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
  22. clueless by no-body · · Score: 1

    I am complainimng to Centurylink for about a week about their service overwhelmed. Latency times of 700+ ms right out my door - countless emails with traceroute logs bring no change - clueless customeer support. With Qwest nthing like that happpened!!!

  23. So ... Re-using AOL Disks Again? by hduff · · Score: 1

    That caused the Three-Mile Island disaster. They should have learned their lesson.

    --
    "I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
  24. Re:nobody posted yet... by RanchNachos · · Score: 1

    $30/month for charter is the promo price. expect it to at least double.

  25. Re:nobody posted yet... by Ol+Olsoc · · Score: 1

    "...the people that it's affecting are unable to read or comment easily"

    Actually, the outage ended about 15 minutes before this story hit slashdot; at least in Las Vegas.

    Whne is the internet going to move to the cloud? Then these problems will go away.

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  26. Re:nobody posted yet... by x_t0ken_407 · · Score: 1

    zomg if I had mod points...

  27. Re:nobody posted yet... by ixidor · · Score: 1

    and evan after id doubles it is still cheaper than CenturyLink 6065

  28. DSL isn't even available in my suburban area by zummit · · Score: 1

    I can't even get Qworst ... I mean Century Link DSL at my home office in a major suburb of Denver not 5 miles from downtown! WTF???

  29. Re:nobody posted yet... by Penguinisto · · Score: 1

    Don't care - $60/mo for 30mb versus $70/mo for 3.(something)mb that breaks down all the time (and being billed as if it were a 6mb package)? No effing way.

    --
    Quo usque tandem abutere, Nimbus, patientia nostra?
  30. Centurylink 12hr outage occurred May 3, NOT May 7 by hardwarefreak · · Score: 1

    The media reporting of this outage as occurring on May 7 is flat wrong. They're off by 4 days. Centurylink's network outage began at ~12:25 CST May 3 and ended at ~01:00 CST May 4. Following are the entries from my mail server log that show the start, end, and duration of the outage. Note the flurry of queued deliveries occurring after IP routing was restored at 01:02 May 4. You can clearly see there are no client connections between 12:20 CST May 3 and 01:02 CST May 4, about 12.5 hours. I called the DSL support line once at about 13:30 May 3 after performing local troubleshooting and eliminating local equipment as the cause. I received "all circuits are busy, please try again later." Receiving that message when calling a telco is like walking into a grocery store and finding the shelves empty. I instantly realized the severity of the outage, and gave myself the afternoon of May 3 off, knowing it wouldn't be fixed for many hours.

    BTW, how does one go about modifying the headline? It needs to inform readers that the outage date/time in the referenced articles is wrong by 3 days.

    May 3 12:15:04 greer postfix/smtpd[24985]: connect from unknown[199.15.233.160]
    May 3 12:15:04 greer postfix/smtpd[24985]: NOQUEUE: reject: RCPT from unknown[199.15.233.160]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [199.15.233.160]; from= to= proto=ESMTP helo=
    May 3 12:15:04 greer postfix/smtpd[24985]: disconnect from unknown[199.15.233.160]
    May 3 12:17:01 greer postfix/anvil[24898]: statistics: max connection rate 2/60s for (smtp:59.55.253.97) at May 3 12:09:06
    May 3 12:17:01 greer postfix/anvil[24898]: statistics: max connection count 1 for (smtp:168.100.1.7) at May 3 12:08:13
    May 3 12:17:01 greer postfix/anvil[24898]: statistics: max cache size 2 at May 3 12:08:41
    May 3 12:17:52 greer postfix/smtpd[25008]: connect from rheocrat.us-deals-atlanta.com[91.90.205.207]
    May 3 12:17:52 greer postfix/smtpd[25008]: NOQUEUE: reject: RCPT from rheocrat.us-deals-atlanta.com[91.90.205.207]: 554 5.7.1 : Client host rejected: Mail not accepted from Ukraine; from= to= proto=ESMTP helo=
    May 3 12:18:09 greer postfix/smtpd[25008]: disconnect from rheocrat.us-deals-atlanta.com[91.90.205.207]
    May 3 12:20:30 greer postfix/smtpd[25013]: connect from unknown[173.232.112.158]
    May 3 12:20:31 greer postfix/smtpd[25013]: NOQUEUE: reject: RCPT from unknown[173.232.112.158]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [173.232.112.158]; from= to= proto=ESMTP helo=
    May 3 12:20:31 greer postfix/smtpd[25013]: disconnect from unknown[173.232.112.158]
    May 3 12:23:51 greer postfix/anvil[24898]: statistics: max connection rate 1/60s for (smtp:91.90.205.207) at May 3 12:17:52
    May 3 12:23:51 greer postfix/anvil[24898]: statistics: max connection count 1 for (smtp:91.90.205.207) at May 3 12:17:52
    May 3 12:23:51 greer postfix/anvil[24898]: statistics: max cache size 1 at May 3 12:17:52
    May 4 01:02:00 greer postfix/smtpd[26610]: connect from mail88.us2.rsgsv.net[72.26.195.88]
    May 4 01:02:00 greer postfix/smtpd[26610]: NOQUEUE: reject: RCPT from mail88.us2.rsgsv.net[72.26.195.88]: 554 5.7.1 : Client host rejected: Access denied; from= to= proto=ESMTP helo=
    May 4 01:02:00 greer postfix/smtpd[26610]: disconnect from mail88.us2.rsgsv.net[72.26.195.88]
    May 4 01:02:26 greer postfix/smtpd[26614]: connect from oss.sgi.com[192.48.182.195]
    May 4 01:02:26 greer postfix/smtpd[26610]: connect from oss.sgi.com[192.48.182.195]
    May 4 01:02:26 greer postfix/smtpd[26614]: F174E6C1B3: client=oss.sgi.com[192.48.182.195]
    May 4 01:02:27 greer postfix/cleanup[26615]: F174E6C1B3: message-id=
    May 4 01:02:27 greer postfix/qmgr[15493]: F174E6C1B3: from=, size=6358, nrcpt=1 (queue active)
    May 4 01:02:27 greer postfix/smtpd[26614]: disconnect from oss.sgi.com[192.48.182.195]
    May 4 01:02:27 greer postfix/smtpd[26610]: 172866C1B8: client=oss.sgi.com[192.48.182.195]
    May 4 01:02:27 greer spamd[23679]: spamd: connection from localhost [127.0