Slashdot Mirror
Time Warner Cable Experiences Nationwide Internet Outage
Wolfrider (856) writes Reports are coming in from multiple U.S. states that TWC is having a major Internet outage since early this morning. ... TWC's customer service reps are reportedly a bit overwhelmed by call volume at the moment, and no ETA yet.
According to CNET, most locations should be more-or-less back online as of 7 a.m. EDT or so (my TWC connection came back around 7:30 a.m. EDT). TWC says it's maintenance gone wrong: In response to a query by CNET, Time Warner Cable issued this statement: "At 430am ET this morning during our routine network maintenance, an issue with our Internet backbone created disruption with our Internet and On Demand services. As of 6am ET services were largely restored as updates continue to bring all customers back online."
this level of incompetence clearly deserves the penalty of death
my kids woke up this morning and couldn't watch their Phineas and Ferb cartoons. my oldest kid had to actually read a book to pass the time before camp and a trip to the museum
My time Warner went out sometime around 4:30 EST this morning, oddly enough it came back up before I was awake (IRC logged my disconnect) and with IPv6 service finally, it's a good chance they had a hiccup doing a mass switch to IPv6
TWC: "Surely only Comcast can save us! Please allow the merger!"
Government: "Well golly gee, what a great idea!"
All my liberal friends think I'm a conservative, all my conservative friends think I'm a liberal.
I am on Time Warner at home and when I got up this morning I found I couldn't check my email like I do every morning. I spent a bit of time on it trying to figure out what they had done to the connection and found I could not connect to their DNS servers, but I could connect to remote IPs and even trace route them just fine.
I had to head into work, so I didn't stay to actually call them... But I'm hardly surprised if they have screwed up their DNS. I had changed my DNS settings to use Google's DNS servers for a long time because theirs had issues. More recently I rebuilt my PC and I hadn't changed the settings again so they still use Time Warner's DHCP settings to pull DNS server settings.
we are all invisible unless we choose otherwise
When I subscribed to Comcast a while back, there was a 4 day outage. By the second day, I found out that it was due to an attack on the DNS servers. So I went into the gateway and changed mine to Verizon's for the time being.
If it was just DNS, I'll be a bit dissapointed. With stuff like "Internet Backbone" I'm hoping for some major BGP messup.
I mean, a couple hours isn't the worst TWC has ever done. Not even in the past year. This is just a wide problem. No one even bothers to note 12 hour fuckups on a regional scale.
Thanks, Comcast!
"TWC's customer service reps are reportedly a bit overwhelmed by call volume at the moment"
So it's just a normal day then? It seems that whenever I call one of these companies all I get is "due to unusually heavy call volume we are experiencing extended hold times".
I haven't had TW for years, but when I did, it was newsworthy when it was actually working.
Slow down, cowboy! It has been 4 hours since you last posted. You must wait another few hours.
Couldn't even ping 8.8.8.8, so I suspect it was more than just a problem turning URLs into numerical addresses.
It was so weird to have problems and still have all the lights on the modem lit up (except standby).
Unfortunately there's no novelty factor in having problems with it in general.
I see even classic Slashdot is now pretty much unusable on dial up anymore.
I was forced (by moving) to switch from TWC to Comcast. I can tell you from experience that everything that is bad about TWC is at least 5x worse with Comcast. TWC fixed this in a matter of hours; if it were Comcast they would have billed the customers for the problem and it wouldn't be working until at least Monday.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Did the NSA screw up again while tapping in into the TW infrastructure? I mean, if the manage to knock whole countries of the Internet by doing so, I wouldn't be surprised if they knocked a few ISP offline now and then.
Did they jump the gun on the comcast merger?
Several rumors about this outage are circulating, so being as I work at Time Warner i thought id dispell them.
1.Your call is important to us: this is a longstanding glitch in our callcenter software. at no point do any of us sincerely have any valued interest in you, or your communication.
2.We are working to resolve the issue quickly: not likely. Tina started her lunch break at like, 10 am. im posting this from the beerpong tavern up the road. My manager is asleep in the datacenter. nick and rob from networking didnt show up today because they had like, 11 vodkas last night and that means theyll be on from vpn around 6 pm to sign their timesheet.
3.This was caused by routine maintenance: again, a rumor. Dale and I wanted a kegerator in the NOC but we were already using the fridge in there for bottled beers. Instead we emptied a rack and made space. we're calling it cellar temperature but really its a bit warmer.
4.You are all valued customers: I dont know where people get this idea. have you tried calling? seriously most of the phone tree is a circular loop. we have maybe 4 phones in this office and its incredibly difficult to order pizza and wings while you "people" keep inundating us with sob stories about your internet.
anyhow sign up for the bundle, or 3 play or whatever it is the coked up hobo in marketing calls it these days. maybe that will help, or not, i dont know. that food truck with the salvadorian food is outside again and daddy needs a pupusa so, whatever.
Good people go to bed earlier.
Now I'm inclined to believe the typical explanation of "A wizard did it". God dammit, Dresden, I thought we told you to stay away from the datacenters.
It could also be that you chose to root your device, or to uninstall myriad floarware (bloat). Srsly, thought it was a server bug for TF2. Then for an hour-and-a-half it seemed as though the Android's networking must have gone terminal. Until getting back to the desk to find the pc in the same state. It's return was magical of course.
... try sending :80 though to LAN proxy?
The fact they have ONE backbone connection is an utter and epic failure. Who designed their network because that person needs to be fired.
Each major section or city needs it's own backbone connection. At least that is how we did it back in the old @home days, one failure can not bring down all services across the nation. Some cities had multiple backbones going into the area's OTN.
Do not look at laser with remaining good eye.
Comment removed based on user account deletion
No need to shoot them. They just need to be replaced. I hear comcast's network is ALWAYS up. I bet if we just got ahold of their admins... maybe through a merger... service would be WAY more reliable. Don't all you customers want more reliable service? Just support the merger!
This will never happen again once Comcast buys TWC.
TWC's customer service reps are reportedly a bit overwhelmed by call volume at the moment
Hopefully, their remaining customers, who evidently are willing to wait on hold for at least two hours under normal circumstances, will understand and will patiently wait on hold for ten hours for the privilege of being told "We're looking into it but there's nothing we can do right now."
Luckily, TWC has competition in my area. Every time I see an ad from them that seems cheaper than what I'm paying to their competitor, I remember the horrifying customer service, and the urge to save money goes away.
A self-inflicted single point of failure to a mission-critical network? TWC, you are now qualified to be a government contractor!
I'm a TW customer, and I didn't notice the difference because TW always sucks (and so does their sole competition in the area. F oligopolies).
Table-ized A.I.
You could edit your civilized +5 post, turning it into a troll fest. Yet still having the same +5 rating. To deal with that, /. would have to throw out the ratings for any edited post. Then those mod points would be wasted. More mod points would be needed. Users edit their posts once a minute. SkyNet would be the result.
I come here for the love
I was wondering where you had went, Ernestine! Should Comcast lack the vision and foresight to keep you on, we've got a spot for you.
Irving P. Freely,
SVP of Customer Relations, AT&T
Really? Moderated as flamebait. Really? Fuck you.
I can't believe how much chat there is about this outage, and so little talk about what diagnostics showed the problem was. It was a DNS issue, NOT a general routing issue.
It hit us here in Austin TX. It looked like a DNS outage... but I was using Google DNS. Routing was NOT down... I could still access a selection of web servers by direct IP address, and ping and traceroute. Rebooted my modem and router repeatedly. Modem acquired a link quickly, and status page showed it had a valid configuration. Modem's signal strength dropped from the usual minus-8-ish to minus-6-ish dBmV. Router acquired IP and WAN domain name effortlessly.
I tried OpenDNS, Earthlink, Dell, and some other public DNS servers I have in a list, but they didn't work either. All timed out. I didn't know what TWC's DNS servers were, so I zeroed them out in my router config, then rebooted. Well, DHCP picked up TWC's DNS servers like nothing was wrong. STILL had no working DNS resolution! And I could still access websites by direct IP address. I was also still receiving mail and web traffic to my own servers, though well below usual levels. The email server was rejecting all mail though, since it couldn't verify the sender's domain names.
Exhausted, I gave up and went to bed. Everything was simply working again when I got up.
The evidence suggests that TWC decided to *filter* DNS traffic, possibly even to aggregate and reroute all of it, and they screwed the pooch. I can't think of any legitimate reason they'd need to do this. I think I'm going to go back to running my own DNS server. Not much I can do about state-redirected DNS traffic other than tunnel it through a VPN perhaps.