Slashdot Mirror
Netgear Routers DoS UWisc Time Server
numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.
Simple mistake that should have easily been found and fixed during the testing phase. I hope whoever let this thing be released without following proper testing procedures got canned.
Yah right. Some hapless low level programmer probably got all the blame for putting test data in there in the first place.
Were this a Haxor attack, there would be criminal liability. I'm willing to believe that it was a simple mistake, with no criminal intent, but would NetGear be liable civilly?
to hardcode an address into thier systems? Do you need permission? There was a law a few years ago about 'deep-linking' and even linking... isn't getting the time somewhat the same thing?
"If you are on fire you can just stop, drop, and roll. If you fall into Lava you are just dead." - my 5yr old daughter
Wow, that list of Analysis Tools used for tracking this down had a bunch that I was not familiar with.
RRGrapher, FlowScan and Cflow being ones I have never messed with..
Cool.. new tools to play with!
anime+manga together at last.. in real time.
Routers tend to log activities such as access, configuration changes, firewall violation detection, etc. and it is often handy to know when that event occured.
Home centric routers do not tend to have their clocks set before shipping as there is no assurance that a battery keeping that clock powered will be doing so ver the entire span of time from manufacture to customer plugging it in. Even if it did the drift involved would give some inaccuracy as well.
There are two correct solutions. One is that Netgear should operate their own time server and hard code that server as a secondary or fallback time server. The primary time server should be aquired from the internet service provider when they get their network ip address via dhcp.
-Rusty
You never know...
want to see what the usage graph for a slashdotting looks like?
i ?target=%2Fweb-servers%2Fwww;ranges=d%3Aw;view=Acc ess
http://www.cs.wisc.edu/cgi-bin/cricket/grapher.cg
Yeah, I work at the CSL at UW Computer Sciences, and the tracking of this netgear issue was quite an interesting tale. Had us stumped for quite some time.
//FIXME: Bad
This didn't only generate trouble for U of Wisconsin, it also generated a lot of cost for some people using the router. Since the server was down, the Firmware has been trying to connect to the time server constantly, thereby keeping the connection from timing out. (Who wrote that algorithm?) For people whos connections are on metered internet access, this ment the connnection was never closed and they are stuck with the bill.
Aparently there are a lot of Netgear users in Germany who are stuck with horrendous bills now. I wonder if Netgear is going to pick those bills up?
Which is exactly what we did. We have a smallish IS shop: 200+ MS/Novell server, 100+ HP midrange servers, and bazillions of PCs. We put our own time server up which ALL of our corporate systems hit. That server then hits a service available via satellite. It is a lot cleaner and 'nicer' to do things in house than rely on some not-for-profit organizations generosity. I even have my PC at home hit my work time server (when I use the VLAN to connect).
Just my $0.02
"If you are on fire you can just stop, drop, and roll. If you fall into Lava you are just dead." - my 5yr old daughter
Just a suggestion.
This is funny - one of the head sysadmins for UW's network ops gave a firewall talk in one of my grad classes last semester. I remember him saying that they recently put a packet filter on their FW to block NTP requests because they started getting high numbers of them..
They thought that maybe somewhere someone had published a net time server in a document or whatever and that an IT department was deploying it on workstations or there was a document floating around telling people to set it up as their time server...
Looks like they finally got to the bottom of it!
THE MAGIC WORDS ARE SQUEAMISH OSSIFRAGE
I took a Unix course at the University of Colorado in Fall 2001, I think. We had a guest lecture from Evi Nemeth, who is a professor emeritus at CU.
She had done some work on a couple of the DNS root servers, G and H if memory serves. She showed a rate of query graphs for those servers. There was a huge jump in the middle of the graphs that corresponded neatly with the release of Windows 2000.
Turns out Win2000 had it hard-coded to consult the DNS root servers every time it wanted to run a nslookup!
Too late to be known as Bush the First, he's sure to be known as Bush the Worst.
We had customers complain that they couldn't connect to our streaming application. After much head scratching and wasted time, we discovered that the customers MR814 wireless router wasn't working properly.
After a lot of research on the internet, I discovered that this was a well known problem with the MR814, fixed with an update to its firmware. It was strange because I asked the user if he had updated his firmware, which he said he did.
It turns out that the firmware was only released on the Austrilian version of the NetGear website. Downloading and installing that version fixed the users problem.
I sent a polite note to NetGear technical support informing them of this on April 7th. I got back a note on 4/8 saying that it would be forwarded to the appropriate people. On April 17th I sent a more harshly worded note. On April 20th I got back a note saying again that my request would be forwarded to engineering.
I gave up. It wasn't worth it.
Just for fun on May 13th I checked their site again. They had finally updated the software.
This runaround was all to just make a solution to a problem that they had already fixed available. Imagine the hassle trying to get them to actually fix a problem?
Both Windows 2000 and XP have the "Windows Time Service" which once per day query an NTP server to set the system clock. By default, Windows 2000 does not have an NTP server set, and XP looks to time.windows.com -- every blasted installation of Windows XP phones home every day to set its clock and who-knows-what-else.
/setsntp:some.ntp.server and net time /querysntp, or in the Time and Date properties in XP there's the Internet Time tab.
One would expect millions of XP boxes phoning home daily would overload a time server. For myself, I've changed the NTP server to a different server (which I will not name) and had somewhat more reliable time syncing.
The commands are net time
Give me my freedom, and I'll take care of my own security, thank you.
I didn't notice it when I first installed my Netgear RP614 last fall, but several months ago I noticed that my dsl modem and RP614 activity lights were blinking once per second round the clock. Just in recent days it occurred to me that this activity had stopped. Having read the article (sorry I do that once in awhile, /. tradition notwithstanding) I see that UWisc's stopgap solution a was to begin servicing the sntp requests again and as such my Netgear device no longer feels compelled to query them every second
As a side note, one thing that frustrates me about the RP614, although I'm otherwise happy with it, is that even though I can choose an option to allow ping to function, it still wont allow other types icmp traffic through and renders traceroutes out from my workstation useless.