Belkin Router Owners Suffering Massive Outages

An anonymous reader writes: ISPs around the country are being kept busy today answering calls from frustrated customers with Belkin routers. Overnight, a firmware issue left many of the Belkin devices with no access to the customer's broadband connection. Initial speculation was that a faulty firmware upgrade caused the devices to lose connectivity, but even users with automatic updates disabled are running into trouble. The problem seems to be that the routers "occasionally ping heartbeat.belkin.com to detect network connectivity," but are suddenly unable to get a response. Belkin has acknowledged the issue and posted a workaround while they work on a fix.

Comments

Oh hey, consumers!

[fuzzyfuzzyfungus]

Yeah, we sold an untold number of plastic boxes that don't work correctly if we ever stop responding to pings... Why would that ever be a problem? Companies never go bankrupt, deliberately kill old products, or 'change strategic direction'.

Re:Oh hey, consumers!

[gandhi_2]

I'm going to be naming all my deadman switches "heartbeat" now. (:

Re:Oh hey, consumers!

[baudilus]

They hired Infosys to write the firmware, obviously.

Re:Oh hey, consumers!

[i kan reed]

It was a part of their advanced simulation program that slipped out. This particular feature was designed to simulate what it's like to be a Comcast customer.

Re:Oh hey, consumers!

[LessThanObvious]

Exactly. It is just wrong to sell a product that will not continue to function if the company that sold it ceases to exist.

Re:Oh hey, consumers!

[ArmoredDragon]

I think the most awesome revelation about this whole thing is that you can stop a lot of people from accessing the internet if you simply DDoS heartbeat.belkin.com.

Re:Oh hey, consumers!

[sbrown7792]

Just as long as you ensure that your botnet doesn't use Belkin routers.

Live by the cloud, die by the cloud.

[Animats]

Did Belkin tell you their router was dependent on their site being up?

"When I die, the world ends." - Belkin policy

Re:Live by the cloud, die by the cloud.

[Animats]

Why not using that DNS server has fixed the heartbeat ping issue.

Their router may be trying to distinguish, as Windows and most things that connect through WiFi now have to, between real Internet connectivity and fake Internet connectivity. Fake Internet connectivity is when some WiFi access point hijacks all DNS requests to take you to some login web page or ad. So, many devices try to connect to some known site which produces a known response to verify that they can connect to the outside world.

It's the choice of "known site", and not having alternatives for it, that's the problem.

Re:Live by the cloud, die by the cloud.

[mcrbids]

Fake Internet connectivity is when some WiFi access point hijacks all DNS requests to take you to some login web page or ad.

So my company presents at trade shows. Trade shows often have Internet service available at ridiculous prices, and frequently, performance is horrible. Often, rather than pay that ridiculous price, we have a laptop set up with the same configuration as our servers, and run with a recent backup copied onto the laptop. This lets us demonstrate our products with a "sandbox" - same as we use for development - without having to bother with the on site Internet.

Our mobile "server" is set up to wildcard DNS to a locally hosted copy of our website. Other vendors, of course, see our hot spot and figure they can use it to get Internet service on somebody else's dime. When they find that all they can get to is our website and product, it's typical for them to get upset - more than once we've been accused of hacking!

Now, set up the hot spot with an SSID like "NoInternetHere" as a way of discouraging trouble.

Bad Belkin

[JohnFen]

No router should ever be dependent on phoning home to a server in order to work. (No router should be engaging in any communication at all that I haven't told it to!) This is BAD - Broken As Designed. I'm awfully glad that I don't use Belkin stuff.

Re:Bad Belkin

[Mike_EE_U_of_I]

Agreed. I am astonished that this happened. I thought the router companies would have learned their lesson after the SNTP nonsense over a decade ago. Clearly I was mistaken. For those that do not know about that incident, here you go:

http://pages.cs.wisc.edu/~plon...

Re:Bad Belkin

[AmiMoJo]

It's a feature. By pining Belkin's servers they can keep tabs on their customers. See how long they keep their old routers for, what reliability is like, if they replace it with another Belkin etc. Even just knowing the number of active users is valuable marketing data for them.

Mod parent up.

[khasim]

Yeah! Who the fuck thought that was a good idea?

Sounds more like "all of the internets is broken because this one site won't work" complaint I get all the time.

It's a ROUTER. If the physical link is up then try pushing packets through it. That's all.

If you want to show connectivity to a specific site then show that in the diagnostic page on that router. But keep pushing packets.

Re:Mod parent up.

[DougOtto]

Actually, according to links in the article, it DOES still push packets. The issue, specifically, is that it breaks DNS if it can't get it's heartbeat. It's still stupid but the device continues to be a router.

Re:Mod parent up.

[Trepidity]

Apple does that too, though on end-user machines. When connecting to wifi, it doesn't enable the connection until it first verifies you're really connected. It does that by trying to pull a specific known Apple URL. If it doesn't get the expected contents, it guesses you're behind a wifi hotspot's login wall, and pops up the "please log in" page. The intent of this is to make sure apps like Dropbox and your email and whatever don't think they're back online and start failing connections, in the time between when you connect to a hotspot wifi and when you log in. But it also means that if Apple's URL goes down, wifi connection will end up with extra hoops to jump through to get it to work.

Re:Mod parent up.

[ArhcAngel]

Yeah! Who the fuck thought that was a good idea?

IIRC it was Cisco.

Re:Mod parent up.

[h4ck7h3p14n37]

I used to work at Bank of America, they had their Internet facing routers set to ping microsoft.com and to remove themselves from the pool if the pings didn't come back. Sure enough, microsoft.com had issues one day and the entire BofA organization lost Internet access.

Re:Mod parent up.

[Alrescha]

"Apple does that too, though on end-user machines. When connecting to wifi, it doesn't enable the connection until it first verifies you're really connected. It does that by trying to pull a specific known Apple URL."

I'm sorry, but there must be more to this than your description.

Just for jollies I unplugged my cable modem and fired up my Macbook. It connected to Wi-Fi and I was up and running on my local network same as always.

A.

Re: Mod parent up.

[ArcadeMan]

I'm typing www.slashdot.org, but it's the website that's broken.

Re:Mod parent up.

[Anonymous Coward]

iPhones and iPads definitely exhibit this behavior, but I think only on Wifi networks that do not have any security enabled (which would be most public hotspots). When the login page pops up, it always has a apple URL before quickly switching to the login page's URL.

Re:Mod parent up.

[Kazoo the Clown]

If Microsoft decides to ping BofA to determine if the internet is alive, that'd be just about right.

Re:Mod parent up.

[Bengie]

Unless they were sending 100mb of ICMP traffic, I doubt the network admins where going to care. I would rather have the servers responding to the pings than the firewall having a huge list of IPs to check against. Blocking data is expensive, CPU wise. Faster just to forward it.

Re:Mod parent up.

[NoMaster]

It's a ROUTER.

No, it's a BELKIN. They've been pulling stupid shit like this for the last 10 years at least , so why anyone should still be surprised is beyond me.

Remember that - whatever you buy from them, it'll always be a Belkin first and <device> second.

Re:Mod parent up.

[ShaunC]

And adding to the list, Windows does it as well. It's called Network Connection Status Indicator.

NCSI is designed to respond to changes in network conditions, and examines the status of a network connection in a variety of ways. First it uses an active probe to determine the status. For example, in an active probe NCSI tests connectivity by trying to reach http://www.msftncsi.com/ a simple Web site that exists only to support the functionality of NCSI. Eventually, as other programs begin generating Internet traffic, NCSI switches to a passive monitoring process that assumes responsibility for detecting changes to the network status.

Every time a network configuration event occurs (meaning that something has changed in the network configuration), the NCSI process performs several tests to identify the network's connectivity status. The first step NCSI performs is a DNS query for www.msftncsi.com. The second step is and HTTP get request for http://www.msftncsi.com/ncsi.t.... This file is a plain-text file and contains only the text "Microsoft NCSI." Last it will perform a DNS query for dns.msftncsi.com.

The URLs used by NCSI can be changed via Group Policy, i.e. you can have it check for the presence of some local server, so that it doesn't bug the shit out of users on a network without external connectivity. Several weeks ago, Microsoft was having global DNS troubles, and many users reported seeing the "trouble" icon in the tray even though their internet connection was working just fine; the problem was that msftncsi.com wasn't resolving. Whoops.

Re:Mod parent up.

[JDG1980]

But all that means is that the user sees a yellow splat in the system tray. Not really a big deal. You can still actually access the Internet on Windows even if NCSI thinks it's down. The Belkin issue is a much bigger deal.

Re:Mod parent up.

[AmiMoJo]

The important difference is that Windows just puts up an icon saying "your connection looks broken", it doesn't stop resolving DNS like Belkin routers do.

I quit using Belkin years ago,

[pecosdave]

any type of device, they won't get my money for even a power strip.

They earned my boycott honestly years ago. I still haven't let them off the hook. Comments on that article exposed other reasons not to even give them the satisfaction of wiping your ass their products.

In retrospect

[XxtraLarGe]

It was probably short-sighted to write their main event loop like this:

while (!((date.day == 7) && (date.month == 10) && (date.year == 2014))) {
// rest of router code follows
}

Re:In retrospect

[BronsCon]

Just a nitpick, but it's more efficient to use OR...

while (date.year != 2014 || date.month != 10 || date.day != 7) {

Before and after 2014, you only do the year comparison, for 11 months of 2014 you only do the year and month comparisons, and only for the month of October 2014 do you have to compare all three. That's 31 days of doing three comparisons, 334 days of doing two, and infinity days of doing just one.

Your code first evaluates the day, which will equal 7 12 times each year, so you're doing two comparisons 11 times a year for all eternity, while on October 7th you're doing 3 comparisons, every single year. It doesn't stop there, though, because you're actually doing an extra operation to negate your compound comparison. In reality, your code is never doing a single comparison, it's always two, three, or four, and an infinite number of each, with the worst case being the second most common, rather than the rarest. We're talking about an embedded system, here, where this actually matters.

Anyway, just nitpicking... really just giving you a hard time, I'm sure you'd never put unoptimized code like that into production on a resource-limited embedded system. Right?

In reality, I'm going to be mentoring someone soon and I'm brushing up on my explanations of good and bad practices.

Re:In retrospect

[Garfong]

I think you need to review your boolean logic. !(a && b) is equivalent to !a || !b

Re:In retrospect

[BronsCon]

Step through that again, bud.

My code:
Is it 2014? No: Enter Loop. Yes:
Is it October? No: Enter Loop. Yes:
Is it the 7th? No: Enter Loop. Yes: Break Loop.

Your code:
Is it the 7th? No: Return False. Yes:
Is it October? No: Return False. Yes:
Is it 2014? No: Return False. Yes: Return True
Negate the return value of the compound logic.
Is the negated value True? Yes: Enter Loop. No: Break Loop.

It would be more clear in a flow chart, but your code does a lot more branching than mine, and branching uses CPU cycles, as well as registers.

Re:In retrospect

[OdinOdin_]

ah yes, de Morgan's law.

linksys

[junkgoof]

And they bought Linksys from Cisco. Deep sigh.

Re:One rule comes to mind...

[gstoddart]

ALWAYS be the master of your own network.

Dude, such a wasted opportunity to say "Master of your own domain ".

Belkin Router Owners Suffering Massive Outages

[FrostedWheat]

Asus Router Owners Suffering Massive Smugness

Why a fixed hostname?

[Orgasmatron]

Old cable modems sucked. Mine would often lock up, needing a power cycle to resume working. Very annoying when I was at work.

The quick and easy solution is to monitor the connection status and flip a relay to reboot the modem. But how to monitor the connection? Setting a single host or IP seemed like a bad idea because it would have added an extra, and totally unnecessary, single point of failure.

Instead, my home router (slackware box with 2 ethernet cards) collects the IPs that I connect to (by watching the conntrack stuff in /proc/ ), and if it can ping them, adds them to the ping list. It then pings random selections from that list to verify connectivity. IPs are removed if they are unreachable for a while (until it decides the connection is down; no point purging the whole list because of an outage).

Took me a couple of hours to set up and debug, back in like 2002 or 2005 or whenever I wrote it. I presume that there is some free software to do the same task by now.

Monitoring a single fixed hostname is foolish, at best. And this is like the 3rd or 4th big story (that I can think of) about home routers acting badly because of hardcoded values.

Re:Why a fixed hostname?

[ledow]

I wrote scripts to do ADSL load-balancing / failover using kernel patches that allowed all kinds of things that aren't in the base kernel. Problem was that our ADSL modems were sucky and wouldn't bring the connection back up when it could have done, so I stuck a £5 Velleman electronics kit and a couple of relays into a box, and with USB control we could reset them from the router itself.

It ran a school for 5+ years, even able to stick 3G sticks into the list and let it failover to them when a dead connection was spotted. And, handily, the 3G sticks worked as a perfect "text-to-fix" receptor and also sent out and received text messages on behalf of the school at the same time. Hell, in one emergency, we even just bought a shed-load of SIMs and every time we hit 1Gb data on a SIM, it turned it off, we changed the SIM for the next and threw the first one away (to get around stupid low data limits). We literally didn't have anyone know we'd done it, from inside the school, except myself and the bursar who bought the SIMs. Everything just worked seamlessly.

But, just watch number of packets incoming on connection. It's much easier. If your external DNS is down, you aren't going anywhere anyway, without manual intervention. If the root DNS is down, you're fucked. If traceroute can't trace to your ISP's gateway, you're probably dead anyway. All of these work, there's no need to get too complex and ping-out.

So if you aren't getting DNS packets coming back from simple queries, you might as well consider the connection dead and move onto the next. That's what we did. Then a few second later you'd hear a click, the lights would flash on one of the modems, and in a couple of minutes it's would be back up and pass traffic again.

The biggest problem? We had to put TWO IP's on the external VPN list because you were never quite sure what line was up and handling the route for the VPN. It could be either. Plug both in, let the VPN client try both. End of problem.

Was so sad when I realised that I'd left the hardware and scripts for that at my previous workplace.

Re:First!

[Anonymous Coward]

Looks like you were behind a Belkin router.

Re:Ummm - did we forget the obvious?

[dunkindave]

Many years ago I had a similar problem with Comcast. Their system's DHCP wasn't giving me an address, so I called the tech support number. The person on the phone told me that he couldn't help me with my problem since help with all DHCP issues was only handled through their new online text chat system. I pointed out that I couldn't get to their handy online text chat system because I COULDN'T GET AN IP ADDRESS. His only response was that maybe I could use a neighbor's computer. Sigh.

Re:One rule comes to mind...

[mythosaz]

This is true for all people who understand the code of OpenWRT in its entirety.

Else it's simply a choice of picking who to trust.

Re:One rule comes to mind...

[bobbied]

Yea, I know, but I'm married...

Arbitrarily breaking HTTP is a bad idea. Who knew?

[Behrooz]

Entertainingly enough, I've run into this issue before. You will encounter the same issue when trying to connect the affected Belkin routers through the Cisco Clean Access NAC here (AKA Campus Housing), because devices are quarantined in the VLAN ghetto until successfully authenticated and associated.

So, these terrible, terrible Belkin routers try to phone home, and when they are unsuccessful they redirect all HTTP requests to the router's administration page. Since sessions are required to authenticate via HTTPS, there is no way to login. Extensive investigation revealed no way to disable this behavior on the client side, SOP for anyone calling with connection problems involving a Belkin router became "Officially unsupported. Return it and get something else that isn't a Belkin."

I am beyond pleased that this incredibly foolish decision on Belkin's part has come back to bite them in general, and hilariously entertained to see that Belkin's temporary workaround was effectively "spoof DNS traffic to heartbeat.belkin.com to a server on your local network that will pingback to fix your ISP's broken clients"

Useful? Maybe if you could disable this 'feature'

[Behrooz]

It might be useful to have a way to disable this 'feature' on the client side.

The bad? There isn't.

The good? This 'feature' already broke connections for anything going through the campus NAC even before their heartbeat server crapped out. SOP for any Belkin-involved problems became "Belkins break RFC2616, they are officially unsupported, go return it and get something that doesn't suck." ...so there aren't any of them still in use to be broken today. Yay!

Re:Google Public DNS?

[slimjim8094]

No they won't.

From here:

In addition, Google Public DNS engineers have proposed a technical solution called EDNS Client Subnet. This proposal allows resolvers to pass in part of the client's IP address (the first 24/64 bits or less for IPv4/IPv6 respectively) as the source IP in the DNS message, so that nameservers can return optimized results based on the user's location rather than that of the resolver. To date, we have deployed an implementation of the proposal for many large CDNs (including Akamai) and Google properties. The majority of geo-sensitive domain names are already covered.

Step by Step Instructions

[CrashNBrn]

You know, cat5 ... maybe?

1) Unplug Cat5 from router.
2) Plug Cat5 into computer.
3) ...
4) Profit, er Internet.

I know, too complicated right?

Belkin? Unpossible!

[Mal-2]

This isn't the first time Belkin has implemented a hare-brained feature, only to have it cause backlash when it induces catastrophic failures across the world. I stopped buying anything with their name on it (except the occasional cable) over a decade ago, over this little feature.

Re:Useful? Maybe if you could disable this 'featur

[Phreakiture]

The bad? There isn't.

It seems as though installing DD-WRT/OpenWRT/Tomato/other-non-OEM-firmware will fix it on at least some routers made by Belkin.