Netgear Routers DoS UWisc Time Server

numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.

Comments

and now...

[Anonymous Coward]

slashdot has hard coded a link to the UWisc CS server, sending a DoS to them too

oh, and fp.

Re:and now...

[TenaciousPimple]

Apparently one good DoS deserves another...

Obligatory Scooby Doo reference

[OneIsNotPrime]

And we would have gotten away too, if it weren't for those meddling kids!

So who got fired?

[eln]

Simple mistake that should have easily been found and fixed during the testing phase. I hope whoever let this thing be released without following proper testing procedures got canned.

Yah right. Some hapless low level programmer probably got all the blame for putting test data in there in the first place.

Re:So who got fired?

[Trigun]

It would have never been picked up in the testing phase. It was only after having a huge install-base that this ever became an issue. It worked perfectly on the bench.

Re:So who got fired?

[(54)T-Dub]

100 MBits/second !?!?!?!?! Do you have any idea how much bandwidth that is?

About once a month a link to my company goes up on the MSN home page (about 3 links down in the top news section). It's like a firehouse and that peaks at an insane 14MBits/second.

Expecting a public service to handle 100 MBits is ridiculous. It was an erroneous mistake by netgear and there should be severe reprecusions.

Re:So who got fired?

[Cali Thalen]

Simple mistake, sure. Barely a trickle of wasted bandwidth, hard to even believe it matters...

Bah.

This is one 'simple mistake' by one company that namaged to send a constant "250,000 packets-per-second (and over 150 megabits-per-second)".

Now I know Netgear is a pretty big outfit, but there are LOTS of companies like that out there, and these little mistakes can add up. How much network traffic could be avoided with proper programming?

Also, this kind of makes me think about the useless network activity my XP box (bleh) tries to send out. Multiply that by millions and millions, and you get a number a whole lot bigger than the one above.

Who pays for all that wasted bandwidth?

Re:So who got fired?

[Trigun]

NO, but I did get to play with a few of these (although I'm not certain if they had the hardcoded NTP servers or not), and they all did work great on the bench.

Re:So who got fired?

[NulDevice]

Usually, there should be a code review.

Usually, someone should say "hey, are we following the RFC for the protocol here?"

Usually, someone should say "isn't hardcoding one single IP address for a service a bad design idea?"

None of these things apparently happened. It may not show up in "testing" (hey, everything worked fine) but in quality assurance, they should be checking their code for anomalies.

Re:So who got fired?

[MikeHunt69]

Code reviews are a whitebox testing technique. Im not a coder, Im a tester. I can't do a code review, because Im not familiar with the code.

Its up to the developer to follow the required standards and up to the architect to make sure bad design decisions are not made.

The grandparent was implying that it was the fault of a tester that the bug went undetected. My point is that in the absence of a spec, mistakes such as this can only be discovered and repaired by the developers.

(Im also not trying to shift blame, Im just saying it's almost impossible for a tester who is doing his job properly to find this)

Re:So who got fired?

[NulDevice]

Becasue it's not just a use of a public service, it's a complete abuse of a public service. It'd be like you damming up the colorado river for your own personal use and then telling LA to upgrade their water supply.

This was a big screwup - when an NTP query fails, you don't start retrying every second until it comes back. You don't hardcode a single server address for it. And you don't put this in 700,000 pieces of released hardware.

Re:So who got fired?

[pboulang]

Right, because when you analyse a security product, you don't look at every single packet to and from it when it is on the bench.

QA isn't just for spell checking.

Re:So who got fired?

[Dr. Blue]

In the full description, you'll notice that they include the "strings" output from the netgear software, which includes hardcoded IP addresses.
Netgear reported that the non-UW addresses were used for debugging by the developers.

Here's the interesting part: at least two of those are 12.* addresses --- cablemodems with attbi.com. So if you want to know who the developer responsible is, it might be a reasonable guess it's whoever lives at those IP addresses! :-)

Re:So who got fired?

[krist0]

bah, weaklings,

when i worked for a ISP over here in .nl, we hosted the live streaming of big brother...

300mbit/s out, on old cisco 7500s....had to get new gig cards for it...the original GEIPs started borking....

heh, everyone needed their vouyerism fix.

Re:So who got fired?

[NulDevice]

> Why would a code review catch this? The guy at the review may have said "I think that address is time.nist.gov" or something. There may be >100K lines of code in a product like this.

A code review would hopefully catch the "hey, we're only using *one single time server for all our hardware* and the *hey, there's no way of configuring this short of patching the firmware* parts. Maybe the address part was overlookable, but the other bits?

>> Usually, someone should say "hey, are we following the RFC for the protocol here?"

> According to the article the packets were well-formed.

Well-formed, yes. But sending retries every second on failure? I coulda sworn the RFC recommended a poll interval of at least 6sec...(but I could be wrong. might'n't've been the RFC - but somebody somewhere reccommends a much higher number for a retry interval, it even says so in the article). It may follow the letter of the law but not the spirit, if I may borrow a cliche.

> Isn't hardcoding a default address good design rather than leaving an uninitialized variable?

Lesser of two evils? Or possibly greater - if they'd left it unitialized, the damn thing wouldn'ta worked and it wouldn't make it to market before it got checked.

The worst part is the fact that they coded it *hard* - not just default-valued it, they coded it so you couldn't change it, and that's ludicrous for a system that's depending on resources it doesn't have control over.

Really, I think "quality assurance" in business-speak means different things to different orgs. I contracted once at a company that had a multipart QA system - some folks went over design specs, some went over code, some did blackbox testing of product. Granted it didn't work so well because they had idiots running the whole thing, but the point is, this was poor design that made it to market when it shouldn't have. Maybe it wasn't a "QA-department" issue, but it was some quality that wasn't assured.

Re:So who got fired?

[Malc]

Not their first simple mistake though. Ask the people behind dyndns.org what they think of the Netgear RT314's (and other products like the RT311????) implementation of the dyndns.org client. Trust me, they have nothing nice to say.

Re:So who got fired?

[orpheus2000]

Well, it's not that bad, most of what XP churns out are NetBIOS broadcasts and other non-routable packets. Makes your LAN a bit noisy, but if you've got your own private subnet (a must for broadband), turn off automatic update checking, it stays pretty quiet on the other side of the router.

Re:So who got fired?

[Stephen Samuel]

Usually, someone should say "isn't hardcoding one single IP address for a service a bad design idea?"

They didn't hardcode just one address. They hardcoded a bunch of them but, by the time UWisc figured out what was happening, they were the only one of the public servers left standing (at least, at the original IP address). BTW: {,X}NTPD doesn't support DNS names for all parts of it's config file, either.

In other words, NetGear managed to DOS a number of public NTP servers out of existence.

The problem here really isn't one of hardcoding a single IP address. It's a problem of taking a shortcut to RFCs and other protocol documentation and not seriously considering the long term consequences. And it's not likely to be caught in a normal code review because the problem looks like the result of a reasonably high-level design trade off. (hard-coded ping times, no DNS and fixed source port all smell of trying to delete "unnecessary" code from the PROM).

This is rather like a littering problem: "It's just one candy wrapper" seems harmless, until you multiply it by 300,000 people using the same road daily and the 2 year+ lifetime of some plasticised wrappers. Similarly, "It's just one packet a second" sounds harmless until you consider the effects of a 1 Million unit product run.

(BTW: I'm guessing that UWM's most recent NTP spike was when the power came back on in NewYork and Ontario last week).

Poor uWisc

[mobiGeek]

First the NTP flood.

Now the /. effect.

Re:Poor UWisc

[NulDevice]

You should see how the UW sysadmins drink. That explains a lot about the ranking.

Bad form in general

[Hayzeus]

Highlights how not to code embedded devices

Or any other kind of software for that matter.

Now...

[Scalli0n]

SCO claims that the offending code was copied from their kernel and most definitely MUST be paid for, including a $699 license fee for all people on planet earth owning any model netgear router.

I did that to myself once

[eschasi]

I did that to myself once. It was a piece of software that went to comp.sources.unix (or something similar) and was default-configured to send error mail to an alias that pointed to me. A patch was released very shortly afterwards.

If they did it to my NTP server...

[lightspawn]

I'd just send the wrong time back to netgear routers. I bet they wouldn't try that again.

Re:If they did it to my NTP server...

[gordon_schumway]

An impropperly formatted response, like "2/30/2003", would probably get people's attention.

From RFC 958: NTP timestamps are represented as a 64-bit fixed-point number, in seconds relative to 0000 UT on 1 January 1900.

Re:If they did it to my NTP server...

[charon_on_acheron]

Right. So just figure out what number represents how many seconds would add up to Febuary 30, 2003. Basically, it would be the same value as March 2, 2003, but you have to remember to set the evil bit. That'll do it every time.

Hasn't /. learned?

[ndogg]

It's not nice to kick someone when they're down.

In other news at the University...

[BMonger]

"Quick! Block port 80!"

I wonder what NetGear's liability is.

[Jammer@CMH]

Were this a Haxor attack, there would be criminal liability. I'm willing to believe that it was a simple mistake, with no criminal intent, but would NetGear be liable civilly?

Re:I wonder what NetGear's liability is.

[HBI]

Of course there is liability - liability means that 'is Party X responsible for the damage'. Netgear quite clearly was responsible for the damage. Even if they allege negligence on the part of their employee, it hardly matters: Netgear had a duty to assure that the software would not cause material harm to others. This is a classic product liability case, far as I can see.

As for the damages, those are somewhat vague. Sure, maybe they could be made to pay for the bandwidth used. The big hit would probably be punitive damages unrelated to the actual loss.

This would be a fun case and I would encourage them to sue. So many frivolous lawsuits floating around - this one would actually have some merit.

Re:I wonder what NetGear's liability is.

[seanadams.com]

They probably would be liable. What surprised me was that the article made no mention of the financial impact of the flood... are the guys who run the network so far removed from the guys who pay the bills that they have no idea, or do the universities get such sweet deals on bandwidth that it doesn't matter?

I mean, we're talking 150+ Mbps here, for months on end. That's $15K/mo in bandwidth, assuming they have a really good deal and pay only $100/Mbps/mo.

Re:I wonder what NetGear's liability is.

[ShortSpecialBus]

We are discussing several options with NetGear. I can't really go into them at the moment, but NetGear has been VERY cooperative throughout this whole thing.

Re:I wonder what NetGear's liability is.

[barfomar]

Rather than enrichen the lawyers, Netgear should just donate cash and appropriate equipment to the University.

It would probably be deductable, passing some of the cost on to we taxpayers; but would sit alot better with public perceptions of the company.

Set up a few CS scholarships or funding a chair at the University would help.

They could turn it into a publicity coup and end up paying out less in the long run (and screw the lawyers too). Some (not all) insurance companies have finally discovered that it's usually cheaper to negotiate with the plaintiff right away, avoiding all of the sabre rattling and lopping off a third (or more) of the total probable cost.

Litigation is rarely the best answer.

Re:I wonder what NetGear's liability is.

[HBI]

Dude I used to settle liability claims (PD and BI) - mostly bus accidents and truck mishaps for companies like JB Hunt and Schneider National.

Of course litigation isn't the best answer, but when one party tries to mitigate damages inappropriately, it is the _only_ solution. I doubt Netgear/Bay would offer sufficient compensation in this case. As for judging how much it was worth, i'd have to see a lot more information but it's probably worth more than a couple switches or 10 computers.

Re:I wonder what NetGear's liability is.

[Nucleon500]

This would be a fun case and I would encourage them to sue. So many frivolous lawsuits floating around - this one would actually have some merit.

I disagree. Netgear is obviously liable, but just because they could be sued doesn't mean they should be. There's a fine line between excercising your rights over others and being an ass, one that I think is crossed way too often. In this case, as you say, the actual damages (bandwidth) are vague. More importantly, Netgear and UWisc got together and are fixing the problem. Considering that this is (now) a very public story, Netgear won't want to further damage it's reputation, and I'm sure they'll donate and hardware and bandwidth necessary to fix the problem. If they had just ignored it, a suit would be justified, but at this point, litigation won't solve anything. It'll just make Netgear look bad, which will make them angry, and start a conflict that only lawyers will benefit from.

Re:I wonder what NetGear's liability is.

[zimage]

according to a post on an ntp.org mailing list, it's costing $266 per day.

Now did NetGear get permission

[eaddict]

to hardcode an address into thier systems? Do you need permission? There was a law a few years ago about 'deep-linking' and even linking... isn't getting the time somewhat the same thing?

Re:Now did NetGear get permission

[jenkin sear]

Not in this case- it's a public time server. If it wasn't, they'd be able to just block inbound UDP for the ntp port at the firewall.

Check out the NTPd man pages- I believe this server is a second echelon mirror.

Re:Now did NetGear get permission

[mahdi13]

Check out the NTPd man pages- I believe this server is a second echelon mirror.

Didn't you mean to say stratum?
Unless NTP is really a cover up to a top secret government information collection service =)
...now that I think about it...
Where's my tin foil hat?

Analysis Tools used in this article..

[joeldg]

Wow, that list of Analysis Tools used for tracking this down had a bunch that I was not familiar with.

RRGrapher, FlowScan and Cflow being ones I have never messed with..

Cool.. new tools to play with!

Delicious irony

[ryanvm]

I love the irony of trying to read an article about a DoS from a site that's experiencing one because of the article. Yummy.

Err why ?

[Archfeld]

why does a router need to sync time anyways ??
especially a home router....sounds like another port open for someone to hack at for no real gain....

Re:Err why ?

[NetJunkie]

Logging. You want your log files to have the right time. I've used my router log files many times.

Re:Err why ?

[rusty0101]

Routers tend to log activities such as access, configuration changes, firewall violation detection, etc. and it is often handy to know when that event occured.

Home centric routers do not tend to have their clocks set before shipping as there is no assurance that a battery keeping that clock powered will be doing so ver the entire span of time from manufacture to customer plugging it in. Even if it did the drift involved would give some inaccuracy as well.

There are two correct solutions. One is that Netgear should operate their own time server and hard code that server as a secondary or fallback time server. The primary time server should be aquired from the internet service provider when they get their network ip address via dhcp.

-Rusty

Re:Err why ?

[gordon_schumway]

There's no way Netgear is going to pull that stunt willingly with their own bandwidth.

RTWFA. This is exactly what Netgear did.

Re:Err why ?

[jmac880n]

In addition to needing accurate timestamps for logging, routers are very convenient NTP servers.

Rather than having your NTP packets pass through the router, have them stop AT the router, and have the router poll for accurate time. This is FAR less overhead for a large subnet (think hundreds of hosts).

Of course, the router SHOULD be responsibly configured to poll a willing timesource.

Indeed

[gilesjuk]

The C comments in the netgear code were a giveaway, they match those in SCOs code.

"/* Huge Bodge */"

"/* Kludge */"

"/* Magic numbers are cool */"

Re:Indeed

[crawling_chaos]

You forgot:

/* Too drunk -- debug later */

NTP should be responsibility of network server

[jefbed]

It is foolish to code code dependencies on servers in firmware. There are two problems that result from this. The first is that specified in the article, the denial of service. The second is the high potential for broken network dependencies if, for example the hardcoded site goes offline or the ip address changes. Technically each site should be running their own ntpd to ease the load on the primary servers. ntp syncronization should not be the job of the router, but instead the job of the network administrator.

blaster

[briancollins]

Maybe windowsupdate.com changed their DNS to point to the University of Wisconsin. :)

Ouch!

[MarkGriz]

I'd hate to be working in Netgear's accounts payable dept. when the bandwidth usage bill arrives.

It's not about just embedded devices...

[sczimme]

Highlights how not to code embedded devices

I think this highlights a "how not to code" idea, period. In 1986, when I was taking a BASIC (boo, hiss) course in high school, I learned that values should be expressed as variables even if the coder does not expect them to change. So instead of using (32 feet/second^2), one should instead declare g once, using whatever units are appropriate, and thereafter refer to g instead of a hardcoded value. If g changes, the coder need only update one line.

Note: I am not a programmer/coder/developer in any sense of any of the words, so technical nits should remain unpicked; however, if I am completely out in left field, please feel free to point that out.

Re:It's not about just embedded devices...

[Bryan Ischo]

Good point, but irrelevent. Even if you declare a global variable, you still have to hardcode its value. The fact that the IP address only showed up 1 time in their string search of the binary would indicate that they did exactly what you said.

So you're not in left field, it's just that the developer who wrote the software apparently did exactly what you said, which was not relevent to the mistake at hand, which was more about the faulty implementation of the NTP service, and the fact that it was hardcoded to a single IP address.

Re:It's not about just embedded devices...

[tommck]

Of course if the gravitational constant changes, we've got bigger problems than updating your high school programming assignments! :-)

Re:It's not about just embedded devices...

[jeffy124]

that is indeed still the case today. This past spring I was a TA for a freshman programming course, and was instructed to deduct points for those who didnt follow such practices -- pi, hours/day, minutes/hour, etc. On exams, the prof would write "-5 - use of magic numbers."

oh, and we laughed long and hard at the guy who put down:

const int SIXTY = 60;
const int TWENTY_FOUR = 24;

Netgear should bear the cost...

[Phil John]

IMHO, since this is blatantly a case of Netgear cocking up their appliance they should not only a)refund any monies spent by the university in this problem and b)send out patches, at their own cost, to all users of affected routers. For heavens sake, so many people don't have anti-virus software installed, don't patch, why would they with a router? They just think "I plug this in to my cable modem, plug my computer in and I dun got thar intarnet workun" why would they know that they need to upgrade the products firmware?

i know USA isnt .AU but..

[sjwt]

With the state of uni bugets out this way,
i think net gear should be thankfull that
it wasnt sued for the bandwidth costs and
the reduced levels of service for the uni..

And then, on friday august 22 2003..

[192939495969798999]

And then we got a ridiculous number of HTTP requests about the problem, which caused our server to explode and rain tiny bits of hazardous material into Lake Michigan. Fortunately, the indigenous wildlife was not affected, because nothing lives in Lake Michigan.

Re:And then, on friday august 22 2003..

[Ericfoos]

You mean Lake Monona and Lake Mendota, not Lake Michigan

Re:And then, on friday august 22 2003..

[Xenoproctologist]

Nothing organic, anyway. However, the hot microchip fragments could be the spark that triggers the genesis of a new race of chemo-silicon-based lifeforms.

Re:And then, on friday august 22 2003..

[h2oliu]

Nope, wrong lake. That would be Lake Mendota that UW is right next to.

Simple Fix

[Boss, Pointy Haired]

UWisc hard codes the date/time on their time time server to 2038-19-01 03:14:00.

After 6 seconds, the netgear will crash and burn as a result of the Y2K38 problem and the requests will be no more.

Think Strata

[n9fzx]

Dave Mill's original clock distribution architecture ala NTP was based loosely on the Bell System's inverted tree structure. Only the top level servers are locked to the national servers; the next level is locked to the top level, and so on. In theory, it's a perfectly scalable infrastructure, with terrific fan-out.

Unfortunately, the code droids seem to think that there's something magical about being at Stratum 2 instead of Stratum 3 or Stratum 4; also, they seem perfectly willing to take advantage of a nonprofit consortium (the owners/operators of public Strat 1 clocks) instead of spending the $500 or so on hardware to service their own customers, who presumably paid them for something.

Anyone else remember the Good Old Days when it was considered polite to ask first before using someone else's clock?

[Truechiming since 1987...]

Re:Think Strata

[seanadams.com]

Unfortunately, the code droids seem to think that there's something magical about being at Stratum 2 instead of Stratum 3 or Stratum 4;

If you're running a large network where clock synchronization is important, you are MUCH better off running your own time server than having you clients talk to someone else's, regardless of stratum. Otherwise the amount of jitter with all your NTP clients going longer distances to fetch the time will actually result in less consistent times overall.

Re:Think Strata

[eaddict]

Which is exactly what we did. We have a smallish IS shop: 200+ MS/Novell server, 100+ HP midrange servers, and bazillions of PCs. We put our own time server up which ALL of our corporate systems hit. That server then hits a service available via satellite. It is a lot cleaner and 'nicer' to do things in house than rely on some not-for-profit organizations generosity. I even have my PC at home hit my work time server (when I use the VLAN to connect).

Just my $0.02

(Geography)Re:And then, on friday august 22 2003..

[StuDude]

Of course, UW-Madison isn't on Lake Michigan (it is in south-central Wisconsin). That must have been quite a server explosion (90+ miles)!!

SEGA's online game servers

[lightspawn]

The (official) reason "Alien Front Online" (a game with the word "Online" in the title!) went offline less than a year after its release is that SEGA developers hard coded the server's IP address, and did not provide any means of changing it. When the company hosting the server went under (gameloft?) it couldn't be moved to a different company since it wouldn't have the same address. Hence, buy a game advertised as "online", never be able to play it online.

It's not a new story, but I think it bears repeating as a showcase of stupidity.

Re:SEGA's online game servers

[AEton]

Well, yeah, with Dreamcast games like Alien Front Online, or with more or less any game since the birth of the console, the read-only nature of the media is a problem. It's hard to issue a patch for a game cartridge or CD, and recalls would be expensive.

The idea a multiplayer game that only has one server to connect to should stir strong feelings of hatred and scorn in any sensible geek. The sheer idiocy of coding in an IP instead of a domain name should be obvious.

Mentioned on ntp.org mailing list a while ago..

[James_G]

I can't get to the article, so in the meantime, here's the text of an email about this with some details that was sent to an ntp.org mailing list back in June:

David L. Mills wrote on 2003-06-26 10:55:

> Guys,
>
> I find myself on the review team for an incident taking place at U Wisconsin/Madison. Apparently, the Netgear folks have manufactured some 700,000 routers with embedded SNTP clients configured to use the public U Wisconsin NTP server. The server address is unchangeable and the client cannot be disabled. If that isn't bad enough, if the client gets no replies, it starts sending packets at one-second intervals until forever and without backoff.
>
> The U Wisconsin folks determined some 285,000 different IP addresses are now sending between 300 and 700 packets per second requiring between 150 and 400 megabits per second. Apparently, the principal eason for this flux is misconfiguration of the firewall component of the router. This is costing them $266 per day.
>
> The Netgear folks were slow to respond until U Wisconsin folks emailed the entire senior management and others known to be U Wisconsin alum. Netgear says they have no way to recall those routers and no way to insure the products are updated from the web site. The products cost between $20 and $40 depending on rebate.
>
> U Wisconsin have considered several ways to deflect the tide, the most promising may be noting the source port 23457 unique to these products and tossing them at the doorstep. The products do not use DNS and are not configurable. Another way considered is to configure a subnet visible to BGP and convince the ISPs to punch holes in the routing fabric. Send money.
>
> I never thought it could get as bad as that. My reasoned recommendation was to fire up the lawyers and sue the bastards for costs and punitive damages and to injoin the company from selling any products until proved safe. There is apparently some standards group that allegedly reviews and certifies new products for Internet use. The Netgear products were all certified, which surely says nothing about the standards group.
>
> Include me in any replies; I am not on any ntp.org list.
>
> Dave

Poor UWisc

[EmagGeek]

First the time server

Then the e-mail server (from the helpdesk requests)

Then the webserver (from /.)

What next?

dyndns.org

[AchmedHabib]

One of the others was an IP address previously used by the "dyndns.org" dynamic DNS name service.
I really hope they did not include that IP while it was used by dyndns.org. If they did, I'd say they are the biggest assholes alive for generating tons of traffic to a free service. But then again they have already proved that now.

Re:How about a verb in that headline?

[leviramsey]

"DoS" is the verb...

Think, McFly, think.

Our usage graph...You Jerks!

[ShortSpecialBus]

want to see what the usage graph for a slashdotting looks like?

http://www.cs.wisc.edu/cgi-bin/cricket/grapher.cgi ?target=%2Fweb-servers%2Fwww;ranges=d%3Aw;view=Acc ess

Yeah, I work at the CSL at UW Computer Sciences, and the tracking of this netgear issue was quite an interesting tale. Had us stumped for quite some time.

Re:Our usage graph...You Jerks!

[ClippyHater]

Oh yeah?! Well, we just /.'d that one, too!

Go ahead, give us another, I dare ya! :)

Re:Our usage graph...You Jerks!

[Lizard_King]

Isn't this a tad bit irresponsible?

don't get me wrong, I love the irony, but your network admins are having enough troubles on a Friday already.

Re:Our usage graph...You Jerks!

[ShortSpecialBus]

I am a network admin, heh.

The load is fine. It's already subsiding. We can handle slashdottings, heh.

Look at the weekly graph, we had 2 this week already!

Just slows down for a while, but doesn't break anything.

Re:Our usage graph...You Jerks!

[Just Some Guy]

You really just linked to content that

1. is dynamic and has to be generated every time?
2. is graphic?

ShortSpecialBus, eh? ;-)

Re:Our usage graph...You Jerks!

[Just Some Guy]

Its cricket...

Think mrtg

Correct so far.

Its dynamic in the sense that it is generated at regular intervals, but it is static in the sense the webserver is serving pre-generated content.

So, yes, the page is static.... most of the time.

Not necessarily true. I run Cricket on my own network, and the images are generated by grapher.cgi; the HTML doesn't point to static images that get replaced on the server at regular intervals. Although grapher.cgi will return cached copies if one exists, you still have to pay the "CGI penalty" of launching a Perl program every single time you view an image. If they're using mod_perl, the overhead may not be so bad. It's still nonzero, though, and I'd hate to have all of Slashdot joyously reloading a Perl CGI on my already-overloaded server.

Re:Our usage graph...You Jerks!

[grozzie2]

I'm chuckling, cuz this has got to be the most informative /. thread in a while, but the useage graph kinda made me laff a bit. I work with code in tiny embedded devices all day long, so, I read the article with GREAT interest, and particularily the paths taken to resolution (which appears to be an ongoing thing).

My hat is certainly off to you folks, it's so refreshing to see somebody facing a serious problem, and actually go about the course of identify and deal with it, with no mention of 'sue them' etc etc. Instead, the problem was identifed, tracked, and eventually the root cause discovered. At that point, they stayed on the high road, and went thru the company to address it, even though initial contacts were 'problematic'. My expectation from most americans after that root cause was discovered, would be for them to get a bidding war going between various law firms as to who could garner the largest settlement, and only then make contact with Netgear, via whichever law firm was bidding highest.

I sympathise with the problem, and I can sure see how something like that slipped thru various pre-release testing cycles (or possibly the lack thereof). The article has definitely made me step back and think about how 'accidental' things like this can slip thru, and possibly consider a new set of release testing parameters to catch such accidents. The /. boys (and girl) are having fun screaming for the head of the folks that caused the problem, but I think there's a valuable lesson in this, made much more valuable by the paths taken towards resolution. It's so refreshing to see non confrontational co-operation in a case like this. That's the kind of spirit that makes the open source world thrive, and it can apply to more than just 'lines of code'.

It generated costs on the other side too

[Anonymous Coward]

This didn't only generate trouble for U of Wisconsin, it also generated a lot of cost for some people using the router. Since the server was down, the Firmware has been trying to connect to the time server constantly, thereby keeping the connection from timing out. (Who wrote that algorithm?) For people whos connections are on metered internet access, this ment the connnection was never closed and they are stuck with the bill.

Aparently there are a lot of Netgear users in Germany who are stuck with horrendous bills now. I wonder if Netgear is going to pick those bills up?

Not the only offender

[oneiric]

When investigating time (mis)keeping on the D-Link DI614+, I found exactly the same thing there. Walking the strings of the firmware reveals a hardcoded list ntp servers and from observation it looks like they walk down the list, primary ntp servers first, to get the time.

The D-Link firmware is cobbled together from quite a few different libraries. It maybe the code exists in a library both systems use or the systems are re-badged from a common source.

How many others then???

OT: answering side question in parent.

[bytesmythe]

It's "nitpick". It refers to the action of removing clusters of louse eggs (nits) from hair. Since louse eggs are so tiny, this requires meticulous precision. Thus, the word came to be associated with finding (often unnecessarily) the smallest defects in anything.

Alas, not true...

[OmniGeek]

The problem is, if one reads the article (nudge, nudge), that 1) at least some of the routers do this with NO operator interface or settability, and 2) some older routers would keep hitting the hardcoded server address even when configured to use some other address. Plus 3) there were some fixes that weren't. The routers in question accept ANY response, even if it isn't an NTP packet! Sending the wrong time would have zero impact. (Why does a home-network router need a clock so badly, anyway? It's not like they do useful remote logging or anything...)

This is a case of ill-designed, badly written, poorly debugged, wretchedly tested code. The article details the testing of a code fix that still didn't fix things properly. On the bright side, Netgear is trying to Do The Right Thing now, and they deserve credit for that.

Spytime

[aero6dof]

Now if NetGear had coded it to their own NTP server it might have been a nice method to estimate how many products you have deployed on the open internet. Of course, Slashdot might then have complained about the company spying on its users. :)

What

[Pvt_Waldo]

Nobody figured how to blame Microsoft yet? Come on you "M$" people - get cracking!

DoS

[smatt-man]

Sweet! I have a Netgear router, does this mean I'm a hacker now?

Re:How do you get the router fixed?

[stratjakt]

1) It's a stratum 1 server, which means it ultimately sets the clocks of millions of other machines, not netgear routers.

2) How many people with a home router (internet savvy or not) spend all that much time reading the logs, let alone making sure the time stamps are valid?

I know you probably do, but I dont. Because I'm just a simple caveman home networker, and your logs and timestamps frighten and confuse me.

Strata ain't the issue

[Merk]

Actually, Netgear was using a stratum 2 time server, namely ntp1.cs.wisc.edu.

As for spending $500 on hardware to service their own customers, as the wisconsin people can tell you, it is costing them a little more than that. It's isn't just the hardware, it's the pipe to which it's attached.

I agree that Netgear should have been the ones to provide a time server if they were going to hard-code one. On the other hand, what if they weren't the ones who wrote the code? Maybe they just bought a "router kit" from some small company, slapped a "Netgear" logo on it, and shipped it out? That small company probably wouldn't know what NTP server NetGear provides. They may also have lots of other customers who each would need their own time server. Obviously though, the answer is not to hard-code the value.

As for the Good Old Days when it was considered polite to ask, the policy for UWisc's time server was "open access", not "open access; please send a message to notify". So... they didn't ask to be notified. Now I'm sure they're going to change that policy, and I'm also sure they would have wanted to know if their site was being set as the default on tens of thousands of routers.

Routers are standalone devices that are meant to operate without user input, so it doesn't make sense to require the user to manually configure the NTP server. On the other hand, there's currently no good way of providing a default NTP server, unless you provide it yourself. For commercial devices like a router, providing it yourself is reasonable. The bandwidth cost of providing a time server should be offset by the profits they make on the hardware. I suppose the other option is to provide a one-time service that will provide a random NTP server. Each time you hard-reset the router, and out of the box, it would check that service and then know what NTP server it should use.

Re:Who pays?

[confused one]

It seems UofW is putting a "redundant fault tolerant server" at the border of their network to handle the traffic. Perhaps, Netgear should compensate them for the cost of the machine and the bandwidth...

Just a suggestion.

hey, now...

[ed.han]

don't you know you're supposed to call us "insensitive clods"?

honestly... :D

ed

Re:Netgear has fix

[Anonymous Coward]

Scratch that, all of the fixes are listed on this page.

http://kbserver.netgear.com/kb_web_files/n101176.a sp

They originally thought it was an IT Dept!

[altek]

This is funny - one of the head sysadmins for UW's network ops gave a firewall talk in one of my grad classes last semester. I remember him saying that they recently put a packet filter on their FW to block NTP requests because they started getting high numbers of them..

They thought that maybe somewhere someone had published a net time server in a document or whatever and that an IT department was deploying it on workstations or there was a document floating around telling people to set it up as their time server...

Looks like they finally got to the bottom of it!

This is par for the course for Netgear.

[Anitra]

Someone on the coding team at Netgear needs to be taken outside and shot; they never seem to learn their lesson about abusing other people's services.

Story:
I used to work/volunteer for DynDNS.org. The Netgear firmware client for DynDNS tried to update regularly (I believe every 5 minutes) whether or not the IP address had actually changed AND whether or not it got a response. Once enough of these got out into the market, this became quite a problem for DynDNS, especially with users complaining that we "blocked" their hostnames updated with the Netgear client when their router advertised specifically that it worked with our service.

I believe after a year or so of nagging the Netgear people, they finally released a firmware update that actually fixed the problem.

I love statements of this nature

[sphealey]

After receiving no response for days, I called Netgear's headquarters, leaving messages with two executives explaining the seriousness of the situation. I also emailed members of Netgear's executive team by guessing their email addresses, based upon their email naming convention. I included a "Return-Receipt-To" header, and their Mail-eXchanger notified me that all were delivered successfully. Here's a portion of that message:

Guys, there is this thing call the "US Postal Service", which has a wonderful product called "Registered Mail" with an optional "Return Receipt Requested" feature. When you have a serious problem of this nature, physically mail a paper letter to the senior executive of the organization, with a cc to the address where the organization accepts legal correspondence (determinable from State records) and also cc "Chief Legal Counsel at...". That will get to the right place faster than guessing random e-mail addresses.

sPh

Could this happen with GPS?

[vasqzr]

I'm ignorant about GPS's.

When someone comes out with a GPS wristwatch, or every laptop/palm etc has one, could this happen?

Re:Could this happen with GPS?

[bungeejumper]

GPS receivers are passive receiving devices, just like FM/AM radios.

Hewlett Packard did the same thing

[pascalb3]

I can't find any articles on it, but I do remember my college having this problem. They kept seeing similar-sized traffic heading to the same IP address every -- I don't exactly remember -- 30 minutes or so. At first they thought they had been infiltrated by a virus that was launching zombies against the IP in a DDoS attack. After sniffing the traffic, it turned out that they were basically ping packets all being sent to the same URL.

What had happened was the ingenious engineers at HP decided to hardcode some poor soul's URL into their new Internet-enabled keyboards -- you know, the ones with the hotkeys. The point was that every so often (which ended-up being very often) the keyboards would send this ping-esque packet to the URL and if it received a response it would know it's still connected to the Internet.

Unfortunately, there were some lapses in the plan. Number one, HP thought this was a good idea, but I guess not good enough of an idea to have them ping their own site. Secondly, with this keyboard a part of new HP systems, these systems turned into DDoS machines on this poor guy's domain. The tricky part was the domain they were sent to wasn't any other company's site, just some apparently random URL the HP team picked; that guy must of thought he was the luckiest person with all the traffic he received, and all the bandwidth he was charged. We are a small college, and even we saw a hit on our network traffic from these keyboards, imagine what he was seeing at the focal point!

The point is, sometimes lack of common sense can have drastic consequences.

Coda: We tracked the IPs of our computer systems pinging the site and told those who owned them to disable the Internet keyboard.

Download corrected firmware

[Luminous Coward]

According to Netgear, only RP614, RP614v2, DG814, MR814 and HR314 NETGEAR routers are affected. Patched firmware can be downloaded from Netgear's support website.

Thank you, UWisc and Netgear

[SamMichaels]

Seriously. THANK YOU for not filing law suits, hiring the FBI, CIA, Marines, calling upon Patriot Act, etc.

To Netgear, THANK YOU for not calling upon the DMCA, filing NDA law suits, etc.

It was resolved in a diplomatic and professional manner...and the write up explaining the entire incident was educational and informative.

Now, if it had been SCO or Microsoft involved......

They're not the only ones

[whterbt]

I took a Unix course at the University of Colorado in Fall 2001, I think. We had a guest lecture from Evi Nemeth, who is a professor emeritus at CU.

She had done some work on a couple of the DNS root servers, G and H if memory serves. She showed a rate of query graphs for those servers. There was a huge jump in the middle of the graphs that corresponded neatly with the release of Windows 2000.

Turns out Win2000 had it hard-coded to consult the DNS root servers every time it wanted to run a nslookup!

NetGear's Customer Support

[MojoRilla]

We had customers complain that they couldn't connect to our streaming application. After much head scratching and wasted time, we discovered that the customers MR814 wireless router wasn't working properly.

After a lot of research on the internet, I discovered that this was a well known problem with the MR814, fixed with an update to its firmware. It was strange because I asked the user if he had updated his firmware, which he said he did.

It turns out that the firmware was only released on the Austrilian version of the NetGear website. Downloading and installing that version fixed the users problem.

I sent a polite note to NetGear technical support informing them of this on April 7th. I got back a note on 4/8 saying that it would be forwarded to the appropriate people. On April 17th I sent a more harshly worded note. On April 20th I got back a note saying again that my request would be forwarded to engineering.

I gave up. It wasn't worth it.
Just for fun on May 13th I checked their site again. They had finally updated the software.

This runaround was all to just make a solution to a problem that they had already fixed available. Imagine the hassle trying to get them to actually fix a problem?

Nah, that's not a problem

[multipartmixed]

> const int SIXTY = 60.2;

The programmer would catch on pretty quick when it didn't compile. Now, if he declared it as a float, on the other hand...

Windows Time Service

[Webmoth]

Both Windows 2000 and XP have the "Windows Time Service" which once per day query an NTP server to set the system clock. By default, Windows 2000 does not have an NTP server set, and XP looks to time.windows.com -- every blasted installation of Windows XP phones home every day to set its clock and who-knows-what-else.

One would expect millions of XP boxes phoning home daily would overload a time server. For myself, I've changed the NTP server to a different server (which I will not name) and had somewhat more reliable time syncing.

The commands are net time /setsntp:some.ntp.server and net time /querysntp, or in the Time and Date properties in XP there's the Internet Time tab.

Re:Windows Time Service

[WoTG]

Sure it's a lot of traffic for some organizations. But for Microsoft to run time.windows.com, it's a drop in the bucket. Lets see... let's say 100M installations (probably high, since it's only XP, and boxes on a domain sync with the domain server) times 1kB per day (again, probably high) is about 100 GB per day and pretty evenly spread out over a 24 hour day. This amounts to less than two T1's. Not a bad deal, considering that in one "simple" move, a big portion of the wrong PC clocks that are out there are fixed. I wouldn't bother switching NTP servers on my XP workstations... why bother if MS is willing to pick up the dime...

Re:Windows Time Service

[Jman314]

And if you want to have even more fun with XP's ntp servers, check out the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\DateTime\Servers. Add a string value, give it a number, and set its data to the ntp server of your choice. Ta-da!

Standard disclaimer applies: back up the registy beforehand. I am not responsible if your computer crashes or blows up or something.

Re:Windows Time Service

[danielsfca2]

If you just need to change the NTP server, but don't need it to be added as one of the "given" servers in the list, you can just edit the combo box in the "Date/Time" Control Panel. You don't have to edit the registry in this case if you don't want to.

Not sure if you overlooked that, or if you were just pointing out the (useful to know) Registry location for adding default time servers. Probably the second one, but I just wanted to put that out there.

Wow, that solves my little blinkenlites mystery

[Jayfar]

I didn't notice it when I first installed my Netgear RP614 last fall, but several months ago I noticed that my dsl modem and RP614 activity lights were blinking once per second round the clock. Just in recent days it occurred to me that this activity had stopped. Having read the article (sorry I do that once in awhile, /. tradition notwithstanding) I see that UWisc's stopgap solution a was to begin servicing the sntp requests again and as such my Netgear device no longer feels compelled to query them every second

As a side note, one thing that frustrates me about the RP614, although I'm otherwise happy with it, is that even though I can choose an option to allow ping to function, it still wont allow other types icmp traffic through and renders traceroutes out from my workstation useless.

That's pretty nasty

[Krellan]

That's pretty nasty that Netgear would hardcode a NTP time server into their product, without even telling U-Wisc about it.

When I configure my computers to use someone else's NTP server, I always send them an email to let them know (or whatever else they request that people do).

What's worse is that Netgear hardcoded the address, in a way that can't easily be changed without a firmware upgrade (something that very few of the intended Netgear firewall customers will do: these customers are looking for a plug-it-in-and-forget-it box, and are either unwilling or unable to learn how to set up a firewall box themselves). And then, on top of that, Netgear botches the implementation of the protocol, causing it to rapid-fire out requests in certain circumstances!

NTP is a very, very low-profile protocol. It uses UDP, so that connection state doesn't have to be maintained. It sends out packets very rarely, at most every few minutes while being set up, and then once time has been established and clocks are in sync, roughly one packet every few days. Netgear's botched programming caused a NTP flood of one packet per second! This is a ridiculous rate several orders of magnitude above what is normally seen in a functioning NTP implementation.

And Netgear sold hundreds of thousands of these things....

I'm amazed that U-Wisc put up with this effective DoS attack on their servers for so long. They showed great patience waiting several months for their request to crawl through Netgear's channels. Companies really need to have a quick method of access into their corporate structure for people who report major flaws like this! Because Netgear's traditional channels of customer feedback (tech support, etc.) weren't set up for this, U-Wisc's requests kept getting lost in Netgear's bureaucracy. Is Netgear so arrogant to believe that all of their products are and will always be 100% flawless?

There really needs to be a special method of access when people report security holes and such. Microsoft, surprisingly, is starting to come around with this, maintaining a special point of contact for people who have discovered security-related issues or major flaws like this. I hope that more companies do this in the future.

If Netgear would do these three things, I would be happy:

1) Set up their own NTP master servers (stratum 1, using a GPS receiver or atomic clock), at Netgear itself. They would use Netgear's own bandwidth, not U-Wisc or anyone else's. Netgear's future products would then default to using these servers, and they would put out a patch so that hopefully some fraction of older products would also use these servers. That way, if there is a flaw in the future, Netgear will eat their own dogfood! I am pleased to see that Netgear is already taking steps in this direction.

2) Change their corporate structure to be more receptive to outsiders who report serious design flaws or major issues caused by their products (such as this NTP flood), going beyond normal tech support, so that quick action can be taken to avert damage. Tech support is really only set up to handle questions about an individual device owned by the person calling in about it, and not set up to handle serious technical or security issues about all devices in an entire product line.

3) Reimburse U-Wisc for the cost of banwidth consumed by these buggy Netgear devices. If U-Wisc isn't blocking incoming NTP entirely by now, pay for robust NTP servers to handle the high volume of traffic. If Netgear had targeted pretty much any private company instead of U-Wisc, I'm sure they would have sued for damages by now!

And remember, ask first before using someone else's NTP server, especially if you plan to hardcode the address into your product :)

To Netgears Credit...Okay maybe not..

[wacko-Netgear]

First off i would like to disclaim that my views do not represent the company's views. With that said, I can say that I worked at Netgear for a short period of time in the area of support.

This specific issues was raised back in may... I can say within that same week they had already started testing firmware to fix the issue. The issue comes with the huge break between Netgear engineers and Netgear support. Umm often times the supports reps do not know of the release of the product until like 2 days or 3 days after its already hit the market. On top of that there is very little communication between the two on firmware and whats the latest version. Its been only in the past couple weeks have they really started to communicate.

Along with that Netgear did not have a device testing program until i would say about 3-4 months ago, before that it was just people there who had the time to test products... woudl test them. I know being one of those who has and still does test there products, that the communication is not very stable and that sometimes issues like these get short-cutted for other major issues such as security and hardware stability.

I am also sure anyone in the hardware market understands the rush that sometimes comes with products; in netgear this is not different. I can this was an issue that was not expected and was fixed as soon as it was reported. It should have never gone out as is and the products should have been tested throughly in the consumer enviorment. But, to Netgear's credit the company does sell pretty good products and there customer support although you may not always be able to get your answer to the issue or may not be able to sometimes understand the reps any and all issues do esclate to people who can fix them. If you issues are not getting fixed at that point the president of the company does read your mail and does forward them to the Head of the customer support. I can say that issues like these will become less of a problem now that Netgear has started a beta program and engineers are required to speak to support engineers on a regualr basis