CenturyLink's Nationwide Outage Affects Millions
halfEvilTech writes "CenturyLink, the nation's third largest telco network, is experiencing an outage of its broadband service nationwide, leaving its support systems overwhelmed and even causing its website to hit a few snags this morning. The company, which at last count has 5.8 million broadband subscribers, has no estimates yet on how long it will take to restore service."
CenturyLink is the company that will be providing the Defense Department with the equivalent of Internet2.
Are the majority of /. readers using Centurylink?
After CenturyLink "partnered" with Qwest they pretty much became a pariah in my book. Qwest was just a terrible terrible mess and apparently CenturyLink is keeping their spirit alive.
I got here through a series of tubes
I am on Century-link and haven't even had a hiccup in my connection.
I cannot find any notice of outage on their website.
My DSL is up. It's either fixed or didn't involve my part of the system.
CenturyLink (formerly Qwest in my case) has been very reliable during the past year. You get the rate you pay for as well.
Maw! Fire up the karma burner!
Enough said.
This is what happens when you don't maintain your infrastructure. This is the company which is still advertising 1.5mbps DSL has "lightning fast" in many areas. It's obvious that once they took over Qwest they continued their "hands-off" approach to maintenance and upgrades. Sad, but not surprising.
South-Western MO went back up around 11:30 Central time.
Maybe century link can just ask china let them have their network back.
Having dealt with having to provide estimates on service restoration at work, my experience is that by the time you can figure out what it would take to restore service, it won't be long before service is actually restored.
I'm not saying that providing status updates isn't good practice. However, it is usually rare that you'll get an ETA on something being fixed. Maybe if they discover it is a broken line and they actually have to dig it up and fix it and that will take hours you might get an ETA. Usually root cause analysis is 95% of the work in problem solving.
Reminds me of a story at work when some developers decided to actually try to embrace the outsourcing model that was being pushed by management. They sent a list of bugs to the outsourced development team and asked for estimates to fix them. They replied, "no problem, just tell us which lines of code to modify and how and we'll take care of it." Now, THAT is a value-add!
I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
What nation?
My DSL is up. It's either fixed or didn't involve my part of the system.
My DSL is up. It's either fixed or didn't involve my nation of the system.
Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
Western states in the former Qwest footprint seem to be up and running ok. East Coast Legacy Centurylink seems to be not having a good time of it though.
Suffice it to say, the sky is not falling and Centurylink is not pulling a "u-verse" on it.
I don’t know exactly what is causing the problem, but when I used to be on Time Warner, we once went down for 4 days due to a DNS issue. I just modified the router settings to go to Google’s DNS servers and my problem was solved.
WHICH NATION'S? Not everyone lives in your country.
I had CL DSL for over 10 years.
I'm only 2600' from the CO, but the best they would sell me was 7mbs.
So I switched to Comcast and get 22mbs, *and* native IPv6, for less than I was paying CL.
I didn't notice any downtime.
Centurylink keeps raising their prices and lowering their bandwidth. I started out at 25 mbit fiber to the house. Then it went to 20, then 15, finally 10.
Prices went up in the meantime so I downgraded to 3 mbit. However they implement the cap seems to work worse than I'd expect. They also require a 1 year contract to get the "discount" rates and it has a $200 cancellation fee even though I don't lease a router and I've been a customer for 5 years.
Overall the local office has top notch support people that are real humans. The national office... not so much.
I'm a Century Link customer! My broadband is down?!
Sounds like it might be more like Internet0.
#DeleteChrome
That always happens when they turn up the pipe to Bluffdale.
To ensure perfect aim, shoot first and call whatever you hit the target
Sleep Tight!
-- you're doing it wrong.
This is the second time Centurylink has had a major outage... This time it appears NC is not included. While my service reliability with Centurylink in the past has degraded over the last three months it has been almost flawless.
My question is did they just grow to fast taken over to many differing networks and lose reliable control at some key point?
I've used them for years and years (only broadband in my area). In general it's been a good service, but when they screw the pooch they go above and beyond. I'll never forgive them for leaving me with huge latency for 3 months. If I ever have a choice in broadband they will be gone.
http://ask.slashdot.org/story/12/03/06/175223/ask-slashdot-what-is-an-acceptable-broadband-latency
You know who else didn't have DNS problems?
I have been working since 8 AM Eastern Time as a telecommuter and my CenturyLink DSL has been up without so much as an SSH session disconnecting all day. I live in SW Florida and my colleagues tell me they're having problems but perhaps the outage is not as widespread as publicized or it's affecting DNS and I use OpenDNS instead of my ISP's DNS for filtering sites I don't want my kids to browse such as adult content.
In my experience, Cable modems were far less stable than DSL. I had Comcast for a while and it was much worse. So please don't listen to the cable modem trolls. Overall DSL is a more reliable technology and I've been using Spring/Embarq/CenturyLink for 12 years now with very few hiccups which they did address quickly, except for a move where I used DirectWay/Hughes and then Comcast for a while. DirectWay/Hughes was good if you have no other options but satellite just has too much latency for some apps like voip or skype, and Comcast was terrible especially during peak hours.
"As flies to the wanton boys are we to the gods; they kill us for sport." - William Shakespeare, King Lear
Wow. Then I wonder what service I'm using if not CLink in the PNW? I'm not experiencing the outages so I guess `nationwide' is a bit over-the-top.
I am complainimng to Centurylink for about a week about their service overwhelmed. Latency times of 700+ ms right out my door - countless emails with traceroute logs bring no change - clueless customeer support. With Qwest nthing like that happpened!!!
That caused the Three-Mile Island disaster. They should have learned their lesson.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
The author of the story "Stacey Higginbotham" - Apparently Chicken Little now has aliases.
I was in an area where the service outage was readily observable. Was back up shortly after noon.
No days on end without service. Tech support was a bit laggy.
I subscribe to Centurylink's updates on facebook. I saw it on my phone this morning that they were addressing issues, and promptly updated us when it was time to reboot our modems to get online.
The reference to CenturyLink providing "internet2" connectivity to the government is quaint. If I recall correctly, centurylink bought qwest, and qwest had a bunch of smaller fiber purchases before they were qwest. Perhaps they were just the best choice to put it together because they already had the fiber?
Ah, right, we're slashdot, where its easier to whine with the rest of the folks rather than think for ourselves.
I can't even get Qworst ... I mean Century Link DSL at my home office in a major suburb of Denver not 5 miles from downtown! WTF???
The media reporting of this outage as occurring on May 7 is flat wrong. They're off by 4 days. Centurylink's network outage began at ~12:25 CST May 3 and ended at ~01:00 CST May 4. Following are the entries from my mail server log that show the start, end, and duration of the outage. Note the flurry of queued deliveries occurring after IP routing was restored at 01:02 May 4. You can clearly see there are no client connections between 12:20 CST May 3 and 01:02 CST May 4, about 12.5 hours. I called the DSL support line once at about 13:30 May 3 after performing local troubleshooting and eliminating local equipment as the cause. I received "all circuits are busy, please try again later." Receiving that message when calling a telco is like walking into a grocery store and finding the shelves empty. I instantly realized the severity of the outage, and gave myself the afternoon of May 3 off, knowing it wouldn't be fixed for many hours.
BTW, how does one go about modifying the headline? It needs to inform readers that the outage date/time in the referenced articles is wrong by 3 days.
May 3 12:15:04 greer postfix/smtpd[24985]: connect from unknown[199.15.233.160]
May 3 12:15:04 greer postfix/smtpd[24985]: NOQUEUE: reject: RCPT from unknown[199.15.233.160]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [199.15.233.160]; from= to= proto=ESMTP helo=
May 3 12:15:04 greer postfix/smtpd[24985]: disconnect from unknown[199.15.233.160]
May 3 12:17:01 greer postfix/anvil[24898]: statistics: max connection rate 2/60s for (smtp:59.55.253.97) at May 3 12:09:06
May 3 12:17:01 greer postfix/anvil[24898]: statistics: max connection count 1 for (smtp:168.100.1.7) at May 3 12:08:13
May 3 12:17:01 greer postfix/anvil[24898]: statistics: max cache size 2 at May 3 12:08:41
May 3 12:17:52 greer postfix/smtpd[25008]: connect from rheocrat.us-deals-atlanta.com[91.90.205.207]
May 3 12:17:52 greer postfix/smtpd[25008]: NOQUEUE: reject: RCPT from rheocrat.us-deals-atlanta.com[91.90.205.207]: 554 5.7.1 : Client host rejected: Mail not accepted from Ukraine; from= to= proto=ESMTP helo=
May 3 12:18:09 greer postfix/smtpd[25008]: disconnect from rheocrat.us-deals-atlanta.com[91.90.205.207]
May 3 12:20:30 greer postfix/smtpd[25013]: connect from unknown[173.232.112.158]
May 3 12:20:31 greer postfix/smtpd[25013]: NOQUEUE: reject: RCPT from unknown[173.232.112.158]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [173.232.112.158]; from= to= proto=ESMTP helo=
May 3 12:20:31 greer postfix/smtpd[25013]: disconnect from unknown[173.232.112.158]
May 3 12:23:51 greer postfix/anvil[24898]: statistics: max connection rate 1/60s for (smtp:91.90.205.207) at May 3 12:17:52
May 3 12:23:51 greer postfix/anvil[24898]: statistics: max connection count 1 for (smtp:91.90.205.207) at May 3 12:17:52
May 3 12:23:51 greer postfix/anvil[24898]: statistics: max cache size 1 at May 3 12:17:52
May 4 01:02:00 greer postfix/smtpd[26610]: connect from mail88.us2.rsgsv.net[72.26.195.88]
May 4 01:02:00 greer postfix/smtpd[26610]: NOQUEUE: reject: RCPT from mail88.us2.rsgsv.net[72.26.195.88]: 554 5.7.1 : Client host rejected: Access denied; from= to= proto=ESMTP helo=
May 4 01:02:00 greer postfix/smtpd[26610]: disconnect from mail88.us2.rsgsv.net[72.26.195.88]
May 4 01:02:26 greer postfix/smtpd[26614]: connect from oss.sgi.com[192.48.182.195]
May 4 01:02:26 greer postfix/smtpd[26610]: connect from oss.sgi.com[192.48.182.195]
May 4 01:02:26 greer postfix/smtpd[26614]: F174E6C1B3: client=oss.sgi.com[192.48.182.195]
May 4 01:02:27 greer postfix/cleanup[26615]: F174E6C1B3: message-id=
May 4 01:02:27 greer postfix/qmgr[15493]: F174E6C1B3: from=, size=6358, nrcpt=1 (queue active)
May 4 01:02:27 greer postfix/smtpd[26614]: disconnect from oss.sgi.com[192.48.182.195]
May 4 01:02:27 greer postfix/smtpd[26610]: 172866C1B8: client=oss.sgi.com[192.48.182.195]
May 4 01:02:27 greer spamd[23679]: spamd: connection from localhost [127.0