Belkin Router Owners Suffering Massive Outages

An anonymous reader writes: ISPs around the country are being kept busy today answering calls from frustrated customers with Belkin routers. Overnight, a firmware issue left many of the Belkin devices with no access to the customer's broadband connection. Initial speculation was that a faulty firmware upgrade caused the devices to lose connectivity, but even users with automatic updates disabled are running into trouble. The problem seems to be that the routers "occasionally ping heartbeat.belkin.com to detect network connectivity," but are suddenly unable to get a response. Belkin has acknowledged the issue and posted a workaround while they work on a fix.

Comments

Why a fixed hostname?

[Orgasmatron]

Old cable modems sucked. Mine would often lock up, needing a power cycle to resume working. Very annoying when I was at work.

The quick and easy solution is to monitor the connection status and flip a relay to reboot the modem. But how to monitor the connection? Setting a single host or IP seemed like a bad idea because it would have added an extra, and totally unnecessary, single point of failure.

Instead, my home router (slackware box with 2 ethernet cards) collects the IPs that I connect to (by watching the conntrack stuff in /proc/ ), and if it can ping them, adds them to the ping list. It then pings random selections from that list to verify connectivity. IPs are removed if they are unreachable for a while (until it decides the connection is down; no point purging the whole list because of an outage).

Took me a couple of hours to set up and debug, back in like 2002 or 2005 or whenever I wrote it. I presume that there is some free software to do the same task by now.

Monitoring a single fixed hostname is foolish, at best. And this is like the 3rd or 4th big story (that I can think of) about home routers acting badly because of hardcoded values.

Re:Why a fixed hostname?

[ledow]

I wrote scripts to do ADSL load-balancing / failover using kernel patches that allowed all kinds of things that aren't in the base kernel. Problem was that our ADSL modems were sucky and wouldn't bring the connection back up when it could have done, so I stuck a £5 Velleman electronics kit and a couple of relays into a box, and with USB control we could reset them from the router itself.

It ran a school for 5+ years, even able to stick 3G sticks into the list and let it failover to them when a dead connection was spotted. And, handily, the 3G sticks worked as a perfect "text-to-fix" receptor and also sent out and received text messages on behalf of the school at the same time. Hell, in one emergency, we even just bought a shed-load of SIMs and every time we hit 1Gb data on a SIM, it turned it off, we changed the SIM for the next and threw the first one away (to get around stupid low data limits). We literally didn't have anyone know we'd done it, from inside the school, except myself and the bursar who bought the SIMs. Everything just worked seamlessly.

But, just watch number of packets incoming on connection. It's much easier. If your external DNS is down, you aren't going anywhere anyway, without manual intervention. If the root DNS is down, you're fucked. If traceroute can't trace to your ISP's gateway, you're probably dead anyway. All of these work, there's no need to get too complex and ping-out.

So if you aren't getting DNS packets coming back from simple queries, you might as well consider the connection dead and move onto the next. That's what we did. Then a few second later you'd hear a click, the lights would flash on one of the modems, and in a couple of minutes it's would be back up and pass traffic again.

The biggest problem? We had to put TWO IP's on the external VPN list because you were never quite sure what line was up and handling the route for the VPN. It could be either. Plug both in, let the VPN client try both. End of problem.

Was so sad when I realised that I'd left the hardware and scripts for that at my previous workplace.

Re:Mod parent up.

[Trepidity]

Apple does that too, though on end-user machines. When connecting to wifi, it doesn't enable the connection until it first verifies you're really connected. It does that by trying to pull a specific known Apple URL. If it doesn't get the expected contents, it guesses you're behind a wifi hotspot's login wall, and pops up the "please log in" page. The intent of this is to make sure apps like Dropbox and your email and whatever don't think they're back online and start failing connections, in the time between when you connect to a hotspot wifi and when you log in. But it also means that if Apple's URL goes down, wifi connection will end up with extra hoops to jump through to get it to work.

Re:One rule comes to mind...

[mythosaz]

This is true for all people who understand the code of OpenWRT in its entirety.

Else it's simply a choice of picking who to trust.

Re:Bad Belkin

[AmiMoJo]

It's a feature. By pining Belkin's servers they can keep tabs on their customers. See how long they keep their old routers for, what reliability is like, if they replace it with another Belkin etc. Even just knowing the number of active users is valuable marketing data for them.

Re:Live by the cloud, die by the cloud.

[Animats]

Why not using that DNS server has fixed the heartbeat ping issue.

Their router may be trying to distinguish, as Windows and most things that connect through WiFi now have to, between real Internet connectivity and fake Internet connectivity. Fake Internet connectivity is when some WiFi access point hijacks all DNS requests to take you to some login web page or ad. So, many devices try to connect to some known site which produces a known response to verify that they can connect to the outside world.

It's the choice of "known site", and not having alternatives for it, that's the problem.

Arbitrarily breaking HTTP is a bad idea. Who knew?

[Behrooz]

Entertainingly enough, I've run into this issue before. You will encounter the same issue when trying to connect the affected Belkin routers through the Cisco Clean Access NAC here (AKA Campus Housing), because devices are quarantined in the VLAN ghetto until successfully authenticated and associated.

So, these terrible, terrible Belkin routers try to phone home, and when they are unsuccessful they redirect all HTTP requests to the router's administration page. Since sessions are required to authenticate via HTTPS, there is no way to login. Extensive investigation revealed no way to disable this behavior on the client side, SOP for anyone calling with connection problems involving a Belkin router became "Officially unsupported. Return it and get something else that isn't a Belkin."

I am beyond pleased that this incredibly foolish decision on Belkin's part has come back to bite them in general, and hilariously entertained to see that Belkin's temporary workaround was effectively "spoof DNS traffic to heartbeat.belkin.com to a server on your local network that will pingback to fix your ISP's broken clients"

Re:Mod parent up.

[h4ck7h3p14n37]

I used to work at Bank of America, they had their Internet facing routers set to ping microsoft.com and to remove themselves from the pool if the pings didn't come back. Sure enough, microsoft.com had issues one day and the entire BofA organization lost Internet access.

Re:Live by the cloud, die by the cloud.

[mcrbids]

Fake Internet connectivity is when some WiFi access point hijacks all DNS requests to take you to some login web page or ad.

So my company presents at trade shows. Trade shows often have Internet service available at ridiculous prices, and frequently, performance is horrible. Often, rather than pay that ridiculous price, we have a laptop set up with the same configuration as our servers, and run with a recent backup copied onto the laptop. This lets us demonstrate our products with a "sandbox" - same as we use for development - without having to bother with the on site Internet.

Our mobile "server" is set up to wildcard DNS to a locally hosted copy of our website. Other vendors, of course, see our hot spot and figure they can use it to get Internet service on somebody else's dime. When they find that all they can get to is our website and product, it's typical for them to get upset - more than once we've been accused of hacking!

Now, set up the hot spot with an SSID like "NoInternetHere" as a way of discouraging trouble.