Slashdot Mirror
Netgear Routers DoS UWisc Time Server
numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.
slashdot has hard coded a link to the UWisc CS server, sending a DoS to them too
oh, and fp.
And we would have gotten away too, if it weren't for those meddling kids!
---
WARNING:Slashdot karma not redeemable in the afterlife.
I did that to myself once. It was a piece of software that went to comp.sources.unix (or something similar) and was default-configured to send error mail to an alias that pointed to me. A patch was released very shortly afterwards.
I'd just send the wrong time back to netgear routers. I bet they wouldn't try that again.
Were this a Haxor attack, there would be criminal liability. I'm willing to believe that it was a simple mistake, with no criminal intent, but would NetGear be liable civilly?
It is foolish to code code dependencies on servers in firmware. There are two problems that result from this. The first is that specified in the article, the denial of service. The second is the high potential for broken network dependencies if, for example the hardcoded site goes offline or the ip address changes. Technically each site should be running their own ntpd to ease the load on the primary servers. ntp syncronization should not be the job of the router, but instead the job of the network administrator.
AntiRight, download now!
Not in this case- it's a public time server. If it wasn't, they'd be able to just block inbound UDP for the ntp port at the firewall.
Check out the NTPd man pages- I believe this server is a second echelon mirror.
What a strange bird is the pelican, his beak can hold more than his belly can.
Highlights how not to code embedded devices
I think this highlights a "how not to code" idea, period. In 1986, when I was taking a BASIC (boo, hiss) course in high school, I learned that values should be expressed as variables even if the coder does not expect them to change. So instead of using (32 feet/second^2), one should instead declare g once, using whatever units are appropriate, and thereafter refer to g instead of a hardcoded value. If g changes, the coder need only update one line.
Note: I am not a programmer/coder/developer in any sense of any of the words, so technical nits should remain unpicked; however, if I am completely out in left field, please feel free to point that out.
I want to drag this out as long as possible. Bring me my protractor.
IMHO, since this is blatantly a case of Netgear cocking up their appliance they should not only a)refund any monies spent by the university in this problem and b)send out patches, at their own cost, to all users of affected routers. For heavens sake, so many people don't have anti-virus software installed, don't patch, why would they with a router? They just think "I plug this in to my cable modem, plug my computer in and I dun got thar intarnet workun" why would they know that they need to upgrade the products firmware?
I am NaN
UWisc hard codes the date/time on their time time server to 2038-19-01 03:14:00.
After 6 seconds, the netgear will crash and burn as a result of the Y2K38 problem and the requests will be no more.
Unfortunately, the code droids seem to think that there's something magical about being at Stratum 2 instead of Stratum 3 or Stratum 4; also, they seem perfectly willing to take advantage of a nonprofit consortium (the owners/operators of public Strat 1 clocks) instead of spending the $500 or so on hardware to service their own customers, who presumably paid them for something.
Anyone else remember the Good Old Days when it was considered polite to ask first before using someone else's clock?
[Truechiming since 1987...]
...-.-
I can't get to the article, so in the meantime, here's the text of an email about this with some details that was sent to an ntp.org mailing list back in June:
David L. Mills wrote on 2003-06-26 10:55:
> Guys,
>
> I find myself on the review team for an incident taking place at U Wisconsin/Madison. Apparently, the Netgear folks have manufactured some 700,000 routers with embedded SNTP clients configured to use the public U Wisconsin NTP server. The server address is unchangeable and the client cannot be disabled. If that isn't bad enough, if the client gets no replies, it starts sending packets at one-second intervals until forever and without backoff.
>
> The U Wisconsin folks determined some 285,000 different IP addresses are now sending between 300 and 700 packets per second requiring between 150 and 400 megabits per second. Apparently, the principal eason for this flux is misconfiguration of the firewall component of the router. This is costing them $266 per day.
>
> The Netgear folks were slow to respond until U Wisconsin folks emailed the entire senior management and others known to be U Wisconsin alum. Netgear says they have no way to recall those routers and no way to insure the products are updated from the web site. The products cost between $20 and $40 depending on rebate.
>
> U Wisconsin have considered several ways to deflect the tide, the most promising may be noting the source port 23457 unique to these products and tossing them at the doorstep. The products do not use DNS and are not configurable. Another way considered is to configure a subnet visible to BGP and convince the ISPs to punch holes in the routing fabric. Send money.
>
> I never thought it could get as bad as that. My reasoned recommendation was to fire up the lawyers and sue the bastards for costs and punitive damages and to injoin the company from selling any products until proved safe. There is apparently some standards group that allegedly reviews and certifies new products for Internet use. The Netgear products were all certified, which surely says nothing about the standards group.
>
> Include me in any replies; I am not on any ntp.org list.
>
> Dave
First the time server
/.)
Then the e-mail server (from the helpdesk requests)
Then the webserver (from
What next?
want to see what the usage graph for a slashdotting looks like?
i ?target=%2Fweb-servers%2Fwww;ranges=d%3Aw;view=Acc ess
http://www.cs.wisc.edu/cgi-bin/cricket/grapher.cg
Yeah, I work at the CSL at UW Computer Sciences, and the tracking of this netgear issue was quite an interesting tale. Had us stumped for quite some time.
//FIXME: Bad
This didn't only generate trouble for U of Wisconsin, it also generated a lot of cost for some people using the router. Since the server was down, the Firmware has been trying to connect to the time server constantly, thereby keeping the connection from timing out. (Who wrote that algorithm?) For people whos connections are on metered internet access, this ment the connnection was never closed and they are stuck with the bill.
Aparently there are a lot of Netgear users in Germany who are stuck with horrendous bills now. I wonder if Netgear is going to pick those bills up?
Becasue it's not just a use of a public service, it's a complete abuse of a public service. It'd be like you damming up the colorado river for your own personal use and then telling LA to upgrade their water supply.
This was a big screwup - when an NTP query fails, you don't start retrying every second until it comes back. You don't hardcode a single server address for it. And you don't put this in 700,000 pieces of released hardware.
----
"I used to listen to Null Device before they sold out."
Someone on the coding team at Netgear needs to be taken outside and shot; they never seem to learn their lesson about abusing other people's services.
Story:
I used to work/volunteer for DynDNS.org. The Netgear firmware client for DynDNS tried to update regularly (I believe every 5 minutes) whether or not the IP address had actually changed AND whether or not it got a response. Once enough of these got out into the market, this became quite a problem for DynDNS, especially with users complaining that we "blocked" their hostnames updated with the Netgear client when their router advertised specifically that it worked with our service.
I believe after a year or so of nagging the Netgear people, they finally released a firmware update that actually fixed the problem.
Have you read the Moderation Guidelines Addendum?
First off i would like to disclaim that my views do not represent the company's views. With that said, I can say that I worked at Netgear for a short period of time in the area of support.
This specific issues was raised back in may... I can say within that same week they had already started testing firmware to fix the issue. The issue comes with the huge break between Netgear engineers and Netgear support. Umm often times the supports reps do not know of the release of the product until like 2 days or 3 days after its already hit the market. On top of that there is very little communication between the two on firmware and whats the latest version. Its been only in the past couple weeks have they really started to communicate.
Along with that Netgear did not have a device testing program until i would say about 3-4 months ago, before that it was just people there who had the time to test products... woudl test them. I know being one of those who has and still does test there products, that the communication is not very stable and that sometimes issues like these get short-cutted for other major issues such as security and hardware stability.
I am also sure anyone in the hardware market understands the rush that sometimes comes with products; in netgear this is not different. I can this was an issue that was not expected and was fixed as soon as it was reported. It should have never gone out as is and the products should have been tested throughly in the consumer enviorment. But, to Netgear's credit the company does sell pretty good products and there customer support although you may not always be able to get your answer to the issue or may not be able to sometimes understand the reps any and all issues do esclate to people who can fix them. If you issues are not getting fixed at that point the president of the company does read your mail and does forward them to the Head of the customer support. I can say that issues like these will become less of a problem now that Netgear has started a beta program and engineers are required to speak to support engineers on a regualr basis