Slashdot Mirror
Netgear Routers DoS UWisc Time Server
numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.
slashdot has hard coded a link to the UWisc CS server, sending a DoS to them too
oh, and fp.
And we would have gotten away too, if it weren't for those meddling kids!
---
WARNING:Slashdot karma not redeemable in the afterlife.
BWAH-HA-HA-HA-HA!
It sure seemed like a good idea at the time tho...
Simple mistake that should have easily been found and fixed during the testing phase. I hope whoever let this thing be released without following proper testing procedures got canned.
Yah right. Some hapless low level programmer probably got all the blame for putting test data in there in the first place.
Now the
Or any other kind of software for that matter.
Roving Web-Teleoperated Robot
SCO claims that the offending code was copied from their kernel and most definitely MUST be paid for, including a $699 license fee for all people on planet earth owning any model netgear router.
Sig & Below
Yuck Fou
I did that to myself once. It was a piece of software that went to comp.sources.unix (or something similar) and was default-configured to send error mail to an alias that pointed to me. A patch was released very shortly afterwards.
I'd just send the wrong time back to netgear routers. I bet they wouldn't try that again.
I had gotten permission to sync my Linux boxen at home from a particular NTP server. I have since moved, and have not yet configured a closer server, sepite once again being online 24/7. The poor admin of my time source is probably wondering about the strange IP address requesting time. Gotta fix that.
You could've hired me.
It's not nice to kick someone when they're down.
// file: mice.h
#include "frickin_lasers.h"
"Quick! Block port 80!"
Were this a Haxor attack, there would be criminal liability. I'm willing to believe that it was a simple mistake, with no criminal intent, but would NetGear be liable civilly?
to hardcode an address into thier systems? Do you need permission? There was a law a few years ago about 'deep-linking' and even linking... isn't getting the time somewhat the same thing?
"If you are on fire you can just stop, drop, and roll. If you fall into Lava you are just dead." - my 5yr old daughter
Wow, that list of Analysis Tools used for tracking this down had a bunch that I was not familiar with.
RRGrapher, FlowScan and Cflow being ones I have never messed with..
Cool.. new tools to play with!
anime+manga together at last.. in real time.
I love the irony of trying to read an article about a DoS from a site that's experiencing one because of the article. Yummy.
why does a router need to sync time anyways ??
especially a home router....sounds like another port open for someone to hack at for no real gain....
errr....umm...*whooosh* *whoosh* Is this thing on ?
The C comments in the netgear code were a giveaway, they match those in SCOs code.
"/* Huge Bodge */"
"/* Kludge */"
"/* Magic numbers are cool */"
It took me a few seconds to figure out what was going on there. :)
Does anyone else think that Netgear owes the UW reparations? Bandwidth costs, time spent by the admins, loss of service, etc. seems like a good place to start... (trying in vain for a good Badger or "When you say Weh-scahn-sen, you said it all" joke, but it just ain't happening today)
Every year during my review, I just pray the words "slashdot.org" aren't mentioned.
It is foolish to code code dependencies on servers in firmware. There are two problems that result from this. The first is that specified in the article, the denial of service. The second is the high potential for broken network dependencies if, for example the hardcoded site goes offline or the ip address changes. Technically each site should be running their own ntpd to ease the load on the primary servers. ntp syncronization should not be the job of the router, but instead the job of the network administrator.
AntiRight, download now!
I hope they fired the guy that wrote the firmware for the routers... And I hope netgear reimbursed the university for its time and network usage.
Can't we just give these guys a break?
what doesn't kill you, will only make you stronger.
We're like rats, in some experiment! -- George Costanza
Maybe windowsupdate.com changed their DNS to point to the University of Wisconsin. :)
I'd hate to be working in Netgear's accounts payable dept. when the bandwidth usage bill arrives.
Beauty is in the eye of the beerholder.
Highlights how not to code embedded devices
I think this highlights a "how not to code" idea, period. In 1986, when I was taking a BASIC (boo, hiss) course in high school, I learned that values should be expressed as variables even if the coder does not expect them to change. So instead of using (32 feet/second^2), one should instead declare g once, using whatever units are appropriate, and thereafter refer to g instead of a hardcoded value. If g changes, the coder need only update one line.
Note: I am not a programmer/coder/developer in any sense of any of the words, so technical nits should remain unpicked; however, if I am completely out in left field, please feel free to point that out.
I want to drag this out as long as possible. Bring me my protractor.
IMHO, since this is blatantly a case of Netgear cocking up their appliance they should not only a)refund any monies spent by the university in this problem and b)send out patches, at their own cost, to all users of affected routers. For heavens sake, so many people don't have anti-virus software installed, don't patch, why would they with a router? They just think "I plug this in to my cable modem, plug my computer in and I dun got thar intarnet workun" why would they know that they need to upgrade the products firmware?
I am NaN
With the state of uni bugets out this way,
i think net gear should be thankfull that
it wasnt sued for the bandwidth costs and
the reduced levels of service for the uni..
You have 5 Moderator Points!
Which Helpless Linux zealot/MS basher do you want to mod down today?
What about that the University of Wisconsin-Madison has "... determined that at least the following code images explicitly contain our server's IP address: MR814_4_11.bin, MR814_v409.bin, RP614_4_0_0.bin, RP614_4_12.bin.".? Isn't this some kind of reverse engenerring or "theft" of copyrighted information / IP?
Will / can Netgear sue them under the DMCA or Copyright law?
NoSuchGuy
Grundgesetz * 23. Mai 1949 - 30. November 2007 - http://www.vorratsdatenspeicherung.de/
And then we got a ridiculous number of HTTP requests about the problem, which caused our server to explode and rain tiny bits of hazardous material into Lake Michigan. Fortunately, the indigenous wildlife was not affected, because nothing lives in Lake Michigan.
stuff |
UWisc hard codes the date/time on their time time server to 2038-19-01 03:14:00.
After 6 seconds, the netgear will crash and burn as a result of the Y2K38 problem and the requests will be no more.
Unfortunately, the code droids seem to think that there's something magical about being at Stratum 2 instead of Stratum 3 or Stratum 4; also, they seem perfectly willing to take advantage of a nonprofit consortium (the owners/operators of public Strat 1 clocks) instead of spending the $500 or so on hardware to service their own customers, who presumably paid them for something.
Anyone else remember the Good Old Days when it was considered polite to ask first before using someone else's clock?
[Truechiming since 1987...]
...-.-
Of course, UW-Madison isn't on Lake Michigan (it is in south-central Wisconsin). That must have been quite a server explosion (90+ miles)!!
The (official) reason "Alien Front Online" (a game with the word "Online" in the title!) went offline less than a year after its release is that SEGA developers hard coded the server's IP address, and did not provide any means of changing it. When the company hosting the server went under (gameloft?) it couldn't be moved to a different company since it wouldn't have the same address. Hence, buy a game advertised as "online", never be able to play it online.
It's not a new story, but I think it bears repeating as a showcase of stupidity.
I can't get to the article, so in the meantime, here's the text of an email about this with some details that was sent to an ntp.org mailing list back in June:
David L. Mills wrote on 2003-06-26 10:55:
> Guys,
>
> I find myself on the review team for an incident taking place at U Wisconsin/Madison. Apparently, the Netgear folks have manufactured some 700,000 routers with embedded SNTP clients configured to use the public U Wisconsin NTP server. The server address is unchangeable and the client cannot be disabled. If that isn't bad enough, if the client gets no replies, it starts sending packets at one-second intervals until forever and without backoff.
>
> The U Wisconsin folks determined some 285,000 different IP addresses are now sending between 300 and 700 packets per second requiring between 150 and 400 megabits per second. Apparently, the principal eason for this flux is misconfiguration of the firewall component of the router. This is costing them $266 per day.
>
> The Netgear folks were slow to respond until U Wisconsin folks emailed the entire senior management and others known to be U Wisconsin alum. Netgear says they have no way to recall those routers and no way to insure the products are updated from the web site. The products cost between $20 and $40 depending on rebate.
>
> U Wisconsin have considered several ways to deflect the tide, the most promising may be noting the source port 23457 unique to these products and tossing them at the doorstep. The products do not use DNS and are not configurable. Another way considered is to configure a subnet visible to BGP and convince the ISPs to punch holes in the routing fabric. Send money.
>
> I never thought it could get as bad as that. My reasoned recommendation was to fire up the lawyers and sue the bastards for costs and punitive damages and to injoin the company from selling any products until proved safe. There is apparently some standards group that allegedly reviews and certifies new products for Internet use. The Netgear products were all certified, which surely says nothing about the standards group.
>
> Include me in any replies; I am not on any ntp.org list.
>
> Dave
I wonder when someone is going to write a virus that delivers a payload that 1. detects the home router 2. connects to a remote server to obtain the proper files 3. upgrades the router with a custom built firmware that removes all normal function and just starts pinging a target. ...
First the time server
/.)
Then the e-mail server (from the helpdesk requests)
Then the webserver (from
What next?
One of the others was an IP address previously used by the "dyndns.org" dynamic DNS name service.
I really hope they did not include that IP while it was used by dyndns.org. If they did, I'd say they are the biggest assholes alive for generating tons of traffic to a free service. But then again they have already proved that now.
> what doesn't kill you, will only make you stronger.
:)
Yeah, or cripple you for life, or make you go broke, or, or, or... There are more than two outcomes...
I bet it would make for a super fast NTP server...
From excellent karma to terible karma with a single +5 funny post...
want to see what the usage graph for a slashdotting looks like?
i ?target=%2Fweb-servers%2Fwww;ranges=d%3Aw;view=Acc ess
http://www.cs.wisc.edu/cgi-bin/cricket/grapher.cg
Yeah, I work at the CSL at UW Computer Sciences, and the tracking of this netgear issue was quite an interesting tale. Had us stumped for quite some time.
//FIXME: Bad
This didn't only generate trouble for U of Wisconsin, it also generated a lot of cost for some people using the router. Since the server was down, the Firmware has been trying to connect to the time server constantly, thereby keeping the connection from timing out. (Who wrote that algorithm?) For people whos connections are on metered internet access, this ment the connnection was never closed and they are stuck with the bill.
Aparently there are a lot of Netgear users in Germany who are stuck with horrendous bills now. I wonder if Netgear is going to pick those bills up?
upgrades the router with a custom built firmware that removes all normal function and just starts pinging a target
1. Router upgrades are done through the admin control panel, which requires a password. I have changed the default password on my NETGEAR router, but others often haven't, so...
2. I'd imagine that router upgrades are digitally signed by the manufacturer.
Will I retire or break 10K?
When investigating time (mis)keeping on the D-Link DI614+, I found exactly the same thing there. Walking the strings of the firmware reveals a hardcoded list ntp servers and from observation it looks like they walk down the list, primary ntp servers first, to get the time.
The D-Link firmware is cobbled together from quite a few different libraries. It maybe the code exists in a library both systems use or the systems are re-badged from a common source.
How many others then???
I guess these guys should've set up a round-robin of stratum-2 ntp hosts themselves.
.sig
is write a worm that automatically patches the firmware. Maybe have it remove SoBig while it's at it...
It's "nitpick". It refers to the action of removing clusters of louse eggs (nits) from hair. Since louse eggs are so tiny, this requires meticulous precision. Thus, the word came to be associated with finding (often unnecessarily) the smallest defects in anything.
bytesmythe
Hypocrisy is the resin that holds the plywood of society together.
-- Scott Meyer
...for companies that do this? I don't know what can and should be done, but it seems like the 'net has become a sewer system more than an information system lately. Several large companies (one is an easy guess) have been responsible for several very significant hits on the 'net as of late...and they just get away with it, only incurring a bit of bad PR.
I'm not proposing legislation (that won't work internationally), but something should be done. How many large/small business and end users are getting dicked over by this kind of stupidity? As someone who sells a network-based shareware app (video conferencing) I have to say that I'm a bit concerned about making future utils.....how well are they going to work in this kind of environment? If that doesn't interest anyone: consider Everquest playability! (gasp)
When posting these stories could people please not say "University of Wisconsin" as Wisconsin has a system of Universities and not just one. Most often, and this case, the story is talking about the University of Wisconsin-Madsion. Up here at the University of Wisconsin-Oshkosh we haven't had any kind of DoS (:
Why aren't there any questions and answers here?
(rtfa)
-- This post (c) 2003, Knights who say Ni, LTD.
The article says they have over 700,000 affected routers out there! No wonder it brought down UofW! Given that NetGear is big enough to play with these kinds of numbers (and make that kind of money!) I think that implies big responsibility too.
At the very least, before releasing such a product to mass production, the damned thing should have been tested to within an inch of its life with a packet sniffer connected to it. ANY unknown or unexpected response from the box should have been tracked down and killed. Doesn't give ME much hope that the box actually does what it is supposed to do. I am not recommending Netgear anytime soon!
I also think that those numbers justify their own damned time-servers too! Even without the indavertant DoS on UofW, this smacks of leveraging profits at someone else's expense. UofW should sue their ass off!
..do as the romans do: change the NTP server's ip, hostname, or both!
This
This was an incredibly well-written article. Not only did it give me a good understanding of the issue at hand, but more information about NTP and some even some routing stuff I want to look into now.
As per the netgear issue, while the UWisc server may take a short beating from a slashdot link, I'm hoping that hopefully we'll at least get a good start on spreading the word about the bad Netgear routers and thus make it worth the slashdotting.
1 question though... who pays for uWisc's bandwidth, and wouldn't even ignoring/recrafting packets still count as incoming bandwidth. That's gotta be one hell of a bill, so hopefully netgear is pitching in a bit.
The problem is, if one reads the article (nudge, nudge), that 1) at least some of the routers do this with NO operator interface or settability, and 2) some older routers would keep hitting the hardcoded server address even when configured to use some other address. Plus 3) there were some fixes that weren't. The routers in question accept ANY response, even if it isn't an NTP packet! Sending the wrong time would have zero impact. (Why does a home-network router need a clock so badly, anyway? It's not like they do useful remote logging or anything...)
This is a case of ill-designed, badly written, poorly debugged, wretchedly tested code. The article details the testing of a code fix that still didn't fix things properly. On the bright side, Netgear is trying to Do The Right Thing now, and they deserve credit for that.
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
Yes it is a publicly accessible server.
However this isn't reasonable access, this is causing damage to uwisc.
IMO Netgear is responsible for fixing this problem, it is a defective product, it should be fixed.
To ease the load even further, routers should run their own time server for the local side so that all devices on the network could just request the sync'd time from the router. That would prevent further congestion from those same network devices performing the same request down the road.
A person generally tends to sync from the same time server every time. One request beats n-device requests.
can university of wisconsin put a fake ntp server which identifies netgear router queries and sends random times and throw them into confusion so much that each netgear router owner would call their support desk. i think this would be a good way to handle such menacing thing.
For example, redhat linux has clock syncronization daemon, and it syncronizes from clock.redhat.com. They should of setup their own server such as clock.netgear.com. That way the server is their resposiblity and they can fix it if it goes wrong.
Nero-burning ROM for Linux!
Now if NetGear had coded it to their own NTP server it might have been a nice method to estimate how many products you have deployed on the open internet. Of course, Slashdot might then have complained about the company spying on its users. :)
Nobody figured how to blame Microsoft yet? Come on you "M$" people - get cracking!
Has there been any further advance on IP broadcasting and service discovery protocols? It would be nice to just send out a polite broadcast asking if there's a timeserver nearby willing to talk to your app instead of having to hardcode names or IP addresses.
The bitter lessons of a veteran coder: http://bitterprogrammer.blogspot.com
Sweet! I have a Netgear router, does this mean I'm a hacker now?
---
Lousy rotten karmic retribution.
Plus, it makes for a lot more entertaining story.
--- Jason Olshefsky
Karma: Poser (mostly affected by adding this line long after everyone else did)
that I need to update the frimware on my router this weekend if I hadn't seen this article on /. How often do you check for new firmware for your home router?
;-)
I can also tell you that I haven't registered my router with Netgear as the only reason I would need to register it is to get support, and if you have ever talked to Netgear support, you are laughing right now
As the article states, netgear support doesn't answer.
I am finding this typical of most tech companies.
I buy fom the local shop, who provides me with the tech support. I know the manufacturer isn't going to do much, but my local guy will do some work for me, and stand behind what he sells.
Right now the Max is ~30 hits per second and the Current is ~29 hits per second, yet the site is rather sluggish and unresponsive. I would have expected the server to be able to manage a bit more than ~30 hits per second.
Not only have they been slashdotted, it's been done through a link that says, "denial of service."
mods?
Bought the firstgen netgears, rather crappy hardware. It did NAT well enough, but it couldnt handle too many machines, ipsec was broken, port mapping was not well implemented, access lists where poor and limited.
Got a dlink wireless router, same mistakes, dyndns was broken, pnat had restrictions.
Then I switched to FloppyFW and a P233 and pci nic cards, linux based. This is what funcationality a nat router should have, basically a linux router with all applications and good updated iptables.
Next step, just do everything manually on a mandrake linux box, with iptables bandwidth limiting.
I ran tcpdump, but nothing was coming across my net interface, so I figured it was just some strangeness with the router.
Gee. I thought I would never be a DoS zombie since I run Linux with a strong firewall. Sounds like a bad B movie: "Zombies In The Firmware"
In a band? Use WheresTheGig for free.
Actually, Netgear was using a stratum 2 time server, namely ntp1.cs.wisc.edu.
As for spending $500 on hardware to service their own customers, as the wisconsin people can tell you, it is costing them a little more than that. It's isn't just the hardware, it's the pipe to which it's attached.
I agree that Netgear should have been the ones to provide a time server if they were going to hard-code one. On the other hand, what if they weren't the ones who wrote the code? Maybe they just bought a "router kit" from some small company, slapped a "Netgear" logo on it, and shipped it out? That small company probably wouldn't know what NTP server NetGear provides. They may also have lots of other customers who each would need their own time server. Obviously though, the answer is not to hard-code the value.
As for the Good Old Days when it was considered polite to ask, the policy for UWisc's time server was "open access", not "open access; please send a message to notify". So... they didn't ask to be notified. Now I'm sure they're going to change that policy, and I'm also sure they would have wanted to know if their site was being set as the default on tens of thousands of routers.
Routers are standalone devices that are meant to operate without user input, so it doesn't make sense to require the user to manually configure the NTP server. On the other hand, there's currently no good way of providing a default NTP server, unless you provide it yourself. For commercial devices like a router, providing it yourself is reasonable. The bandwidth cost of providing a time server should be offset by the profits they make on the hardware. I suppose the other option is to provide a one-time service that will provide a random NTP server. Each time you hard-reset the router, and out of the box, it would check that service and then know what NTP server it should use.
(pardon the pun)
I was just looking at the traffic graphs on my home box this morning, and wondering "why is it never dropping below 3Kb/s?".
I took a look, and for some reason, my machine (debian linux) was chattering back and forth at a high rate with time.nrc.ca - 1.5Kb/s out, 3.2Kb/s in. Not massive, true, but certainly annoying.
I restarted xntpd, but I'm wary as heck now.
don't you know you're supposed to call us "insensitive clods"?
:D
honestly...
ed
Being a good net denizen, I've tried to download the update. Go the appropriate ftp server and you get "Permission Denied." Go Netgear!
Scratch that, all of the fixes are listed on this page.
a sp
http://kbserver.netgear.com/kb_web_files/n101176.
This is funny - one of the head sysadmins for UW's network ops gave a firewall talk in one of my grad classes last semester. I remember him saying that they recently put a packet filter on their FW to block NTP requests because they started getting high numbers of them..
They thought that maybe somewhere someone had published a net time server in a document or whatever and that an IT department was deploying it on workstations or there was a document floating around telling people to set it up as their time server...
Looks like they finally got to the bottom of it!
THE MAGIC WORDS ARE SQUEAMISH OSSIFRAGE
I've got some Sonicwall SOHO firewall boxes that apparently use some hardcoded address for NTS. Don't have a clue where the request goes. Probably thousands of those boxes calling someone with some regularity. Wonder who it is?
Someone on the coding team at Netgear needs to be taken outside and shot; they never seem to learn their lesson about abusing other people's services.
Story:
I used to work/volunteer for DynDNS.org. The Netgear firmware client for DynDNS tried to update regularly (I believe every 5 minutes) whether or not the IP address had actually changed AND whether or not it got a response. Once enough of these got out into the market, this became quite a problem for DynDNS, especially with users complaining that we "blocked" their hostnames updated with the Netgear client when their router advertised specifically that it worked with our service.
I believe after a year or so of nagging the Netgear people, they finally released a firmware update that actually fixed the problem.
Have you read the Moderation Guidelines Addendum?
You don't need a clock for those things - only a timer.
sPh
I'd been thinking aobut returning the #()*@& Netgear Wireless router for a week now. This swings it for me.
The blasted thing won't work with 128 bit encryption enabled, and now I find it's launching a DOS against a time server.
Great.
Screw Netgear. I'm returning the blasted hunk of junk.
There tech support is a pathetic joke anyway.
"Live Free or Die." Don't like it? Then keep out of the USA
Nobody seem to be mentioning it, but this is the schoolbook case of a DDoS.
Mirror #1 HTML PDF
Mirror #2 HTML PDF
Mirror #3 HTML PDF
I'm ignorant about GPS's.
When someone comes out with a GPS wristwatch, or every laptop/palm etc has one, could this happen?
I hope this doesn't hirt the stock price :(
"A synonym is a word you use when you can't spell the word you first thought of." - Burt Bacharach
Why not have their ISPs put in the static route for the /32 to null0 on their feed routers? That way the trafic never gets to UW in the first place. Yes, they would have to change their time server's IP address, but decent time clients would hande the DNS issues properly.
This would at least get it off their link and push the problem upstream in smaller (hopefully) more manageable chunks.
What exactly do you mean by "Don't touch this button?"
Well, the stupidity of the way their NTP client was written (hard-coded IP address, 1sec queries, etc.) is pretty obvious. And of course, the Netgear technical support response ("we don't have time to respond to your problem, but maybe it has gone away on its own already") is typical of the company.
But what seems particularly stupid is that many companies would jump at the opportunity of having an excuse for obtaining detailed usage information about their customers. They would point these devices at their own servers (probably with 1h rather than 1s query intervals) and know exactly how many of their devices are in operation and what software versions they are running.
That is almost certainly what Microsoft and Apple are using their NTP servers for: keeping track of number of installed systems by the day and by IP address blocks (home users, Fortune-500, etc.).
So, we have to conclude that not only Netgear's programming is iffy and that their customer support is pretty awful, they also fail to behave like a rational US corporation by giving up the opportunity of keeping tabs on their customers. Well, maybe that makes them a little less bad, actually, and at least their gear is cheap and works most of the time...
What does the University get out of this? They have expended hours upon hours of man-power to fix a flaw in a companies software becuase I negativly impacted their ability operate. At the very least there should be some donated hardware. If I were the University President, If the University doesn't get a big donation, I'd take them to court.
All your base are belong to us!
Shouldn't Netgear be liable for the costs they impose on the university? I mean, Netgear is a money-making enterprise and they chose to point an excessive number of clients at the service, they should pay for the IT staff, hardware, and bandwidth necessary to service their requests.
And, in addition to the monetary aspect, this would seem to fall under computer hacking statutes. I suspect that if this sort of thing had resulted from a student or small developer releasing some software, the FBI might be knocking on their door already, and protestations that it was an accident might not be listened to. I think Morris also claimed that his worm (the first Internet worm) escaped by accident, yet he got charged.
After tossing several routers, I bought a Netopia R910 router. It cost more than the cheap routers, but it actually works and doesn't need periodic resets. Netopia also appears to be a real company with an in-house engineering staff.
Of all the companies selling networking equipment, how many actually design and program their own equipment?
Mea navis aericumbens anguillis abundat
Some one said why has no one blaimed MS yet,
.
Check this out it is from the product upgrade info from Netgears support page;
Explanation:
FR114P Firmware version 1.3 (Release 16).
New Certifications:
* Adds logging and firewall enhancements. This release version has been certified against version 4.0 of the ICSA Labs Firewall Certification Criteria.
New Features Overview:
Basic Configuration
* Telstra BPA and Austria PPTP login support
Security
* Keyword blocking now applies to newsgroup names
Previously only checked URLs for keywords. Now checks newsgroup names, too.
* (Wireless only) Able to turn off bridging between the wireless and wired local networks
* Added Syslog for sending log entries to a syslog server
* Improved logging
Advanced
* Can turn off the print server
Modifications and Bug Fixes:
1. Fixed: dropping of TCP sessions.
2. Fixed: dropping of Windows Terminal Server sessions.
3. Fixed: traceroute passthrough.
4. Add idle time in PPPoE s basic page (also change help file)
5. Enable Dial on demand (in status page) && set idle=0 means keep alive.
6. Fixed: Add Reserved IP will show wrong MAC
7. Fixed: can Upgrade Firmware when the file path has
8. Fixed: Dyndns memory issue.
Known Issues:
* Dial-on-demand may not work with Austria PPTP service.
* If Print Server does not function after upgrade:
1. Go to LANIP menu
2. Disable Print Server
3. Click Apply
4. Enable Print Server
5. Click Apply
Upgrading to the New Software:
You can upgrade by using the web interface Router Upgrade menu.
1. Open the browser (Internet Explorer or Netscape)
2. Access the router (usually http://192.168.0.1)
3. Login to the router (User Name = admin Password = password unless you have changed it)
4. Under Maintenance, click Router Upgrade
5. Click Browse and locate the upgrade file
6. Click Upload
7. Wait for the router to reboot
Note:
NETGEAR strongly recommends that you clear the configuration after upgrading, using the default button on the rear panel, and re -enter your configuration information. Please write down all of your configuration information before upgrading.
Download:
Click on the following link to begin download
fr114p_1316.zip
FileName: fr114p_1316.zip
FileSize: 437 KB
Version: 1.3 (Release 16), ICSA certified
Date: April 20, 2003
With updates that are that simple you can bet sysadmins don't bother to look! It is all clear now upgrading firmware is sooooo difficult....NOT I just hope for the sake of ease of (use by poor stupid users) you guys don't screw up the most important side of the net. UNIVERSITIES and BUSINESS networks. I sure hope netgear has the sense to make the upgrade that fixes crap like this readily known to all the servers running this device! Even most pimple faced MS sysadmins could easily do this upgrade. Netgear should be forced do Tech TV, and journal adverts, and e-mail outs to help fix this shit. Same goes for MS who now gets the upgrade/patch publicity for free. They should be made to pay for the publicity for a change rather than having the news people do it for them! What am I saying TECH TV is in cahoots with MSNBC anyway!
endrant
OH THE SHAME I fell off the wagon and use sigs again!
As an undergraduate majoring in Computer Sciences at UW-Madison, I demand compensation from Netgear for damaging my education. I'm not exactly sure how, but I'm sure I've been irreparably damaged.
Netgear: Please send me some 10Gbit equipment, and I shall consider the damage repaired.
...there does seem to be a link. From the article:
Our investigation so far has shown that Windows systems such as 2000 and XP have an "Internet Time" feature which is usually configured to send SNTP requests to the Microsoft server "time.windows.com", but this server can be changed. I have yet to identify any SNTP client that regularly uses UDP port 23457 as its source port. (Note that that port number seems hand-picked, as the number subsequent to 23456.)
Any one find a SCO angle?
cheers- raga
I can't find any articles on it, but I do remember my college having this problem. They kept seeing similar-sized traffic heading to the same IP address every -- I don't exactly remember -- 30 minutes or so. At first they thought they had been infiltrated by a virus that was launching zombies against the IP in a DDoS attack. After sniffing the traffic, it turned out that they were basically ping packets all being sent to the same URL.
What had happened was the ingenious engineers at HP decided to hardcode some poor soul's URL into their new Internet-enabled keyboards -- you know, the ones with the hotkeys. The point was that every so often (which ended-up being very often) the keyboards would send this ping-esque packet to the URL and if it received a response it would know it's still connected to the Internet.
Unfortunately, there were some lapses in the plan. Number one, HP thought this was a good idea, but I guess not good enough of an idea to have them ping their own site. Secondly, with this keyboard a part of new HP systems, these systems turned into DDoS machines on this poor guy's domain. The tricky part was the domain they were sent to wasn't any other company's site, just some apparently random URL the HP team picked; that guy must of thought he was the luckiest person with all the traffic he received, and all the bandwidth he was charged. We are a small college, and even we saw a hit on our network traffic from these keyboards, imagine what he was seeing at the focal point!
The point is, sometimes lack of common sense can have drastic consequences.
Coda: We tracked the IPs of our computer systems pinging the site and told those who owned them to disable the Internet keyboard.
According to Netgear, only RP614, RP614v2, DG814, MR814 and HR314 NETGEAR routers are affected. Patched firmware can be downloaded from Netgear's support website.
See! I knew it was a massive DoS causing my grades for last semester to take a nose dive!
But noo, everyone said it was all the partying...
The latest firmware for the offending router includes "NTP Improvements". It was posted Aug 20.
a sp ?dnldID=427
http://www.netgear.com/support/support_details.
when I bought my router, it was in replacement for my netgear, and smc routers that seemed to always freeze because I had my symbol access point connected. Therefore I went to look for an interim all in one solution. In come the MR814 @ Fry's Electronics for 30 dollars plus a 30 dollar rebate! W00!! I only pay for california's 8.25% sales tax! Great, then one night I was trying to play StarCraft with my friends and I kept getting high lag bars in the game. I went downstairs to see if my cousin was uploading porn to his net-buddies to find that he was. I told him to cut it out and he did. However, I checked the stats on my netgear router and it still said there was an amount of upload bandwidth being used. I kept wondering what it was and if I had any sort of spyware that would cause it. But no, the computers at my home were just after I freshly reinstalled windows on each of them. Well anyway, I left that alone until I read this article on /.
Since then I've upgraded my router with the newest firmware and notice that instead of losing 1KB or 2kKB of my sbc dsl 128kb upload, I lose none of it. Yey!
So I read the article, and it appears a bunch of routers are programmed to hit a particular IP address as a time server -- but are set up incorrectly and are swamping the server at that IP address.
So...why doesn't the university just change the DNS record of their timeservers so they point to a different IP, and just have nothing hooked up to the hardcoded IP? I assume there's a reason this wouldn't work -- else they would have done it, right? Am I just not understanding the scale of the problem?
schof
Somehow, I suspect the computer science department of a Big-10 university can handle
I wouldn't be surprised if there were a lot of /. readers at the CS Departments of Big 10 universities.
Their local LANs sure as hell better be able to handle /. traffic from curious local on-lookers located on campus.
"Provided by the management for your protection."
Seriously. THANK YOU for not filing law suits, hiring the FBI, CIA, Marines, calling upon Patriot Act, etc.
To Netgear, THANK YOU for not calling upon the DMCA, filing NDA law suits, etc.
It was resolved in a diplomatic and professional manner...and the write up explaining the entire incident was educational and informative.
Now, if it had been SCO or Microsoft involved......
I took a Unix course at the University of Colorado in Fall 2001, I think. We had a guest lecture from Evi Nemeth, who is a professor emeritus at CU.
She had done some work on a couple of the DNS root servers, G and H if memory serves. She showed a rate of query graphs for those servers. There was a huge jump in the middle of the graphs that corresponded neatly with the release of Windows 2000.
Turns out Win2000 had it hard-coded to consult the DNS root servers every time it wanted to run a nslookup!
Too late to be known as Bush the First, he's sure to be known as Bush the Worst.
We had customers complain that they couldn't connect to our streaming application. After much head scratching and wasted time, we discovered that the customers MR814 wireless router wasn't working properly.
After a lot of research on the internet, I discovered that this was a well known problem with the MR814, fixed with an update to its firmware. It was strange because I asked the user if he had updated his firmware, which he said he did.
It turns out that the firmware was only released on the Austrilian version of the NetGear website. Downloading and installing that version fixed the users problem.
I sent a polite note to NetGear technical support informing them of this on April 7th. I got back a note on 4/8 saying that it would be forwarded to the appropriate people. On April 17th I sent a more harshly worded note. On April 20th I got back a note saying again that my request would be forwarded to engineering.
I gave up. It wasn't worth it.
Just for fun on May 13th I checked their site again. They had finally updated the software.
This runaround was all to just make a solution to a problem that they had already fixed available. Imagine the hassle trying to get them to actually fix a problem?
OK, I certainly am. But... why the hell does a little appliance router even need to know what time it is? Are they providing NTP to their connected machines? If not, why does it matter what time it is?
Maybe they have cute features like allowing access to certain things at certain times of the day? I'm guessing here.
Is your site maximizing it's revenue potential? Add AdCode and your are selling text ads on your site!
Grammatical errors on your company home page (in big bold blue text, no doubt) do not inspire confidence..
cpeterso
Maybe someone was playing too much Alpha Centauri
We use Netscreen and Cisco equipment here. If you want to use NTP, you need to configure it specifically. Otherwise, the boxes go off their internal reset clocks.
NTP is completely unnecessary on a home user's firewall/router.
sarchasm: The gulf between the author of sarcastic wit and the person who doesn't get it.
I attended a talk on this topic last night given by the author of the paper. While it's a very interesting problem, people are missing the real kicker: This had been going on for a long time. The network guys would see blips every once in a while (and wonder "who would try to DoS the ntp server?"), but it wasn't until May that they saw a bit spike.
/20 range to push the problem to the leaves of the network, where cable providers will only have to deal with ~10,000 rogue routers instead of the UW dealing with 700,000.
So they did the most reasonable thing, given that the traffic was easily identifiable (source port 23457): block it at the router. That's when the shit hit the fan. Each one of those 700,000 little boxes was configure, not just with a fixed IP address, but upon failing to receive a reply, to keep asking ONCE PER SECOND until it got a reply.
The end result is that the CSL *can't* block the traffic, or let the machine go down, etc etc. ntp1.cs.wisc.edu just became one of their most important servers, because it will cost them $60,000/mo in bandwidth if it goes down.
Options (at the end of the paper) include: doing clever things to keep the machine up, reliable, and spread bandwidth a bit, or sacrificing a
> const int SIXTY = 60.2;
The programmer would catch on pretty quick when it didn't compile. Now, if he declared it as a float, on the other hand...
Do daemons dream of electric sleep()?
slash back, dude.
All together now: "With a little help from my friends"
Karma: none (due to not believing in reincarnation)
bah... I just upgraded my router (netgear MR814 v1)'s firmware to the latest v4.13. Seems to be stabler and now it's not generating damn random packets... No wonder my Cable modem's traffic light was on 24x7x365!!! This new firmware seems stabler also, Netgear's firmware coders finally *FIXED* something... :)
Both Windows 2000 and XP have the "Windows Time Service" which once per day query an NTP server to set the system clock. By default, Windows 2000 does not have an NTP server set, and XP looks to time.windows.com -- every blasted installation of Windows XP phones home every day to set its clock and who-knows-what-else.
/setsntp:some.ntp.server and net time /querysntp, or in the Time and Date properties in XP there's the Internet Time tab.
One would expect millions of XP boxes phoning home daily would overload a time server. For myself, I've changed the NTP server to a different server (which I will not name) and had somewhat more reliable time syncing.
The commands are net time
Give me my freedom, and I'll take care of my own security, thank you.
If you want your machines at home to know what time it is, and all have the same time, then either you have one machine be the master and everybody else get their time settings from that, or else each machine has to do this on its own, or you have to coordinate which of your machines gets it from which of your other machines. Your router/firewall box is a good server for this kind of application, and it'll always be turned on, while you might turn off one or more of your other devices.
Also, if the public side of the firewall fetches the time from some real server and the private side of the firewall lets PCs fetch the time from the firewall, then the firewall doesn't need a bunch of rules about letting N different kinds of time protocols through it, so there's less configuration, less opportunity for exploits, and less opportunity for mistakes (though "mistakes were made" here anyway :-) It's much cleaner and simpler, and you only have to worry about exploits on the particular protocol that the router uses to fetch the time for itself, not all the possible exploits on other ports and protocols.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
In reading the article, it mentioned that the products in question will accept any response, even if it isn't valid. Could this be used to crash the problem routers by perhaps a buffer overrun or some such? This may cause people to return them, or in some other way take care of the problem at the users end. The unversity wouldn't deliberately do something like this (legal problems), but I just thought I'd throw it out there as another possible solution.
"Any outbound traffic on a connection will instruct people's auto-dialout routers and gateways to either dial out, or hold the connection - i.e. the phone call - open."
That's why I use a 3COM 3C886 (also available in a ISDN model). I can block any address that would try to hold a connection open.
If there's a legal cause of action here, (which as a non-lawyer I don't have a qualified opinion about,) it's not a "You Intellectual Property Thief! I want a Gigantic Punishing Damage Award!" or probably even a "You Dangerously Negligent Clod! I want a big punishment award so you don't do things this stupid again!" - it's at most a "Hey, you're at fault in this accident, you need to pay my costs and clean up your mess" kind of lawsuit, and in fact Netgear is working with them to clean up the mess.
UWisc published their time server in the lists of publicly available NTP servers at the standard NTP web site. Netgear obviously didn't ask for permission, because if they'd reached anybody with a clue, the answer would have been "[Expletive deleted]! No! Please don't do that!"; UWisc got Slashdotted with Extreme Prejudice here.
Now, hardcoding an IP address that isn't magic (like 127.0.0.1 or 192.168.0.1) is technically clueless, and hardcoding an address that doesn't belong to you is not only technically clueless, but clueless from a business standpoint, because your stuff will break if the other people move their server or change their IP address, which happens all the time, and because they might stop providing the service you're expecting, and of course because it's rude, and rudeness isn't good business practice.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Just have it send a password in the email body. If you want to be safer, have the password based on some sort of simple alghorythm that changes on a regular basis (if you don't trust unencrypted email from cellular).
That, or if you can get WAP service on your phone (and browse your own pages without paging a huge fee like my cell provider wants to charge) - then just do it via a WAP website.
I brought home one of the Netgear models in the article, fought it for 4 stinkin' days and then exchanged it for a Linksys that works. (Netgear couldn't handle my wife's need/want to use a webcam/audio with MSN Messenger)
While I did get an email response from Netgear support... the first response said it included a firmware s/w file as an attachment and it didn't. Turnaround on email from them was at least a day. From the email time stamp and the name used, it was obvious that Netgear outsources its customer service to India.
Phone call escalated to Level II support (after an hour on hold) confirmed that even the firmware that CS should have sent wouldn't fix the problem.
One has to wonder if Netgear also outsources its programming to India too....
sPh
I didn't notice it when I first installed my Netgear RP614 last fall, but several months ago I noticed that my dsl modem and RP614 activity lights were blinking once per second round the clock. Just in recent days it occurred to me that this activity had stopped. Having read the article (sorry I do that once in awhile, /. tradition notwithstanding) I see that UWisc's stopgap solution a was to begin servicing the sntp requests again and as such my Netgear device no longer feels compelled to query them every second
As a side note, one thing that frustrates me about the RP614, although I'm otherwise happy with it, is that even though I can choose an option to allow ping to function, it still wont allow other types icmp traffic through and renders traceroutes out from my workstation useless.
So now we finally have some hard evidence of what it takes to slashdot a machine. It seems to peek at just 30 hits/sec, and quickly degrades as it loses its top-of-the-page status.
Gee, my 256kb upstream cable modem could handle this load if each hit only took a couple kbytes to transmit! And I'm certain that my server could handle it, too.
aQazaQa
I assume that they worked with the owners of the Netgear routers or Netgear themselves to determine the routers that are infected.
in the article it mentions the review process with netgear. I imagine this is when they determined the firmware that was in use. The article also mentions contacting ISPs to determine what was on such and such IP.
With the time server at uwa.edu.au.
:> All in all, buying a hardware NTP server and sticking it in a colo would have worked out a lot better economically for them.
UWisc is big enough to look after themselves and presumably doesn't pay for traffic by the megabyte like we do.. 3.5c/mb might not sound like too much, but SMC's arrogance in hard coding the time server cost us thousands in network traffic.
Bastards. In vengence, we now don't buy anything from SMC
That's pretty nasty that Netgear would hardcode a NTP time server into their product, without even telling U-Wisc about it.
:)
When I configure my computers to use someone else's NTP server, I always send them an email to let them know (or whatever else they request that people do).
What's worse is that Netgear hardcoded the address, in a way that can't easily be changed without a firmware upgrade (something that very few of the intended Netgear firewall customers will do: these customers are looking for a plug-it-in-and-forget-it box, and are either unwilling or unable to learn how to set up a firewall box themselves). And then, on top of that, Netgear botches the implementation of the protocol, causing it to rapid-fire out requests in certain circumstances!
NTP is a very, very low-profile protocol. It uses UDP, so that connection state doesn't have to be maintained. It sends out packets very rarely, at most every few minutes while being set up, and then once time has been established and clocks are in sync, roughly one packet every few days. Netgear's botched programming caused a NTP flood of one packet per second! This is a ridiculous rate several orders of magnitude above what is normally seen in a functioning NTP implementation.
And Netgear sold hundreds of thousands of these things....
I'm amazed that U-Wisc put up with this effective DoS attack on their servers for so long. They showed great patience waiting several months for their request to crawl through Netgear's channels. Companies really need to have a quick method of access into their corporate structure for people who report major flaws like this! Because Netgear's traditional channels of customer feedback (tech support, etc.) weren't set up for this, U-Wisc's requests kept getting lost in Netgear's bureaucracy. Is Netgear so arrogant to believe that all of their products are and will always be 100% flawless?
There really needs to be a special method of access when people report security holes and such. Microsoft, surprisingly, is starting to come around with this, maintaining a special point of contact for people who have discovered security-related issues or major flaws like this. I hope that more companies do this in the future.
If Netgear would do these three things, I would be happy:
1) Set up their own NTP master servers (stratum 1, using a GPS receiver or atomic clock), at Netgear itself. They would use Netgear's own bandwidth, not U-Wisc or anyone else's. Netgear's future products would then default to using these servers, and they would put out a patch so that hopefully some fraction of older products would also use these servers. That way, if there is a flaw in the future, Netgear will eat their own dogfood! I am pleased to see that Netgear is already taking steps in this direction.
2) Change their corporate structure to be more receptive to outsiders who report serious design flaws or major issues caused by their products (such as this NTP flood), going beyond normal tech support, so that quick action can be taken to avert damage. Tech support is really only set up to handle questions about an individual device owned by the person calling in about it, and not set up to handle serious technical or security issues about all devices in an entire product line.
3) Reimburse U-Wisc for the cost of banwidth consumed by these buggy Netgear devices. If U-Wisc isn't blocking incoming NTP entirely by now, pay for robust NTP servers to handle the high volume of traffic. If Netgear had targeted pretty much any private company instead of U-Wisc, I'm sure they would have sued for damages by now!
And remember, ask first before using someone else's NTP server, especially if you plan to hardcode the address into your product
Dr. Demento On The 'Net!
First off i would like to disclaim that my views do not represent the company's views. With that said, I can say that I worked at Netgear for a short period of time in the area of support.
This specific issues was raised back in may... I can say within that same week they had already started testing firmware to fix the issue. The issue comes with the huge break between Netgear engineers and Netgear support. Umm often times the supports reps do not know of the release of the product until like 2 days or 3 days after its already hit the market. On top of that there is very little communication between the two on firmware and whats the latest version. Its been only in the past couple weeks have they really started to communicate.
Along with that Netgear did not have a device testing program until i would say about 3-4 months ago, before that it was just people there who had the time to test products... woudl test them. I know being one of those who has and still does test there products, that the communication is not very stable and that sometimes issues like these get short-cutted for other major issues such as security and hardware stability.
I am also sure anyone in the hardware market understands the rush that sometimes comes with products; in netgear this is not different. I can this was an issue that was not expected and was fixed as soon as it was reported. It should have never gone out as is and the products should have been tested throughly in the consumer enviorment. But, to Netgear's credit the company does sell pretty good products and there customer support although you may not always be able to get your answer to the issue or may not be able to sometimes understand the reps any and all issues do esclate to people who can fix them. If you issues are not getting fixed at that point the president of the company does read your mail and does forward them to the Head of the customer support. I can say that issues like these will become less of a problem now that Netgear has started a beta program and engineers are required to speak to support engineers on a regualr basis
-mse
Fiat Lux.
For the record, downloading the firmware & reinstalling is a piece of cake. I'm not enough of an admin to know how to check for the incessant SNTP packets, but following the Netgear reinstall directions was easy enough.
Why didn't they configure a time server to send out randomized results? That'd cause future network load to be spread out instead of spiking. It'd also tend to piss off the router owners, who would then upgrade their firmware to fix their "time" problem.
When they have 128,000 ip addresses?
This explains why there's a problem with IPV4.
I wish I was an uber-eleet h4x0ring packetmonkey just like netgear. I ph34r their ddos skillz!
LEARNING, n. The kind of ignorance distinguishing the studious. A. Bierce, The Devil's Dictionary
That would presumably break all the firmware and would remind the users in no uncertain terms that they need to upgraded to fixed software NOW.
Alternately, it could have told netgear "pick up our bandwidth charges or all that model of router will have to be fixed or replaced."
I see no particular reason to be nice about this, bandwidth costs and netgear is a commercial company that can be expected to pay for its mistakes.
As for anyone else using your server, there are lots of NTP servers out there and changing if yours go down isn't that difficult.
Tech Public Policy stuff
SMC shipped routers with the ntp server at the CSIRO in Sydney hardwired, starting around July 2002. So now you need a fixed IP and a prior arrangement to use their server.
that the US patent system is screwed up!
Either they get their WAN IP via DHCP or it's configured by the network admin staff. If DHCP, why not use the time server value specified in the DHCP lease? OTOH, if someone has to program the IP address, how difficult is it to require them to specify the NTP address as well?
There's simply no reason to require a default value, let alone a hard-coded and immutable one.
http://securityresponse.symantec.com/avcenter/venc /data/w32.sobig.f@mm.html
it reads :
"Sobig.F obtains the UTC time through the NTP protocol, by contacting one of several possible servers on port 123/udp (the NTP port)."
I then would expect the following screnario : Very likely most netgear router users have used its nifty builtin dhcp server. cisco also has such embedded dhcp servers and does SMC barricade. So far nothing special. However if the embedded dhcp server of the netgear router also has dhcp option "option ntp-servers ip-address" configured, using the UWisc Time Server ip-number then the following most probably has happened :
W32.Sobig.F@mm is released on the Internet and hits hard. It spreads through email. upon storing itself on the NT4, win2k, winxp PC, it asks for UTC time through the NTP protocol. If the dhcp client has through dhcp already a configured ntp-server, that one (The UWisc one) will be used. If no ntp-server is configured only then the virus will try to reach the ntpservers listed on the symantec security response description for W32.Sobig.F@mm.
Robert