Slashdot Mirror
D-Link Firmware Abuses Open NTP Servers
DES writes "FreeBSD developer and NTP buff Poul-Henning Kamp runs a stratum-1 NTP server specifically for the benefit of networks directly connected to the Danish Internet Exchange (DIX). Some time last fall, however, D-Link started including his server in a hardcoded list in their router firmware. Poul-Henning now estimates that between 75% and 90% of NTP traffic at his server originates from D-Link gear. After five months of fruitless negotiation with a D-Link lawyer (who alternately tried to threaten and bribe him), he has written an open letter to D-Link, hoping the resulting publicity will force D-Link to acknowledge the issue. There are obvious parallels to a previous story, though Netgear behaved far more responsibly at the time than D-Link seem to be."
From TFA: "A number of D-Link products, so far I have at least identified DI-604, DI-614+, DI-624, DI-754, DI-764, DI-774, DI-784, VDI604 and VDI624, contain a list of NTP servers in their firmware and using some sort of algorithm, they pick one and send packets to it."
Give people an inch and they take a mile. I don't see why D-Link and Netgear couldn't just make their own stratum-1 NTP servers. I mean, if you trust the brandname enough for your routing, don't you trust them enough for your time as well?
pool.ntp.org?
NTP server use is tiered. So client PCs are not supposed to hit the tier 1s, they should hit 2nd tier or a local ntp server.
You don't use the root DNS servers for all your DNS requests, right?
Yes, you're confused. And, you didn't read the article. The author is pissed because he's running an NTP server intended to be accessed only by Danish networks, and for use by servers, not clients. D-Link products are only marketed to clients, and not just Danish clients.
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
He says that such a solution is hard to implement on Cisco, and would be too CPU intensive. FTFA: "Filtering the D-Link packets requires inspection of fields which are not simple to implement in Cisco routers, and in particular such filtering seems to send all packets on the interface through the CPU instead of fast switching, so ingress filtering the packets at the ingress of AS1835 is totally out of the question."
He followed standard protocol for NTP servers, which is to list the restrictions on the use of your server with its entry on the NTP server list. System administrators are supposed to check this to make sure they're not making an unauthorized connection. They're also supposed to contact the NTP server administrator to let him know they're using the server, unless the server admin states otherwise.
You can learn all this and check the list to be sure you comply within 10 minutes thanks to the power of Google. Any responsible company would know this and do so. D-Link made a big mistake (not in terms of the impact on them, sadly) and is evidently refusing to own up.
As others have pointed out, it's not easy to implement the restrictions that would enforce the access policy. It's also sad, though not surprising, that one would have to. It'd be one thing if the server was the target of script kiddie DOS attacks, but a legitimate company selling network products really ought to know better (and care).
send a private communication to the authentic users (not the robot moochers from D-Link) that on date X, the new IP service address will be unhacked.gps.dix.de or whatever suits him.
on date X, send bogus packets in response... not just wrong time, but seriously wrong time, like a packet with time of 9s in all fields, which would be most seriously wrong.
hopefully, it would lock up the offending junkpiles, and clear the problem right smartly.
the general idea in engineering an end to these things is to find a way to blow up the crooked machine by a seriously wrong entry that will screw up the internals. since they took an ugly and cheap shortcut by using firmware tables, they probably don't error-check their inputs from NTP and other services. so there should be a memory jump and a crash in those pirate boxes someplace.
and that puts the onus back where it belongs, on supercheap designers for obnoxious companies that don't give a shit about network etiquette. the market will punish them. that's how it should be for slap-happy outfits.
if this is supposed to be a new economy, how come they still want my old fashioned money?
So why didn't they just own up to the mistake, update the firmware and cut him a check for his expenses plus a 5% or so to apologize for the inconvenience? Bureaucrats and lawyers who cannot admit that they are wrong only end up creating more public disgust with their behavior. When you find yourself digging a hole, stop digging!
There are three conventions being violated:
* To keep the network working, the NTP system is tiered. Anything other than a time server used to redistribute time to other machines should probably access a Tier 3 system, or a Tier 2 if that is not possible. It should never hammer a Tier 1 -- this can screw up the rest of the NTP network.
* There are large lists of NTP servers, and they list access restrictions. As pointed out in the letter, this guy explicitly stated in his access rules that this server was not for client use.
* As pointed out in the letter, this guy explicitly stated in his access rules that this server was not for use outside of Denmark.
You may not be used to this sort of thing, because no such set of agreements exists for, say, webservers. However, in the NTP world, network administrators respect these, and it is why the time system continues to work.
What D-Link is doing hurts all Danish NTP users, and freeloads off a volunteer (D-Link is selling the product and profiting from it -- let *them* handle the traffic and factor any bandwidth costs into their product cost). It opens their product to potential abuse if the server becomes malicious (a properly-designed router would allow the user to specify an NTP server, or if the user is unable to configure a router, to do what the letter suggested and use a D-Link-controlled name.). It violates agreements that have been generally respected by the NTP-using administrator community for many years.
Any program relying on (nontrivial) preemptive multithreading will be buggy.
Let me clarify a number of details here.
1. My server has not replied to the packets sinde the CodeRed virus/worm abused NTP servers to coordinate attacks. That was a couple of years ago. I doubt D-Link ever even tried to test this.
2. NTP is a timing protocol. You do not want to do expensive and timeconsuming filtering on the packets because that disturbs your timing performance.
3. If I have to sue D-Link, it will be either in USA or Taiwan. Both their Danish marketing office and the UK european office will be able to deflect a lawsuit to their mothership.
4. If you download a firmware file from D-Link, it is often a ARJ archive. unpack that and run strings. If you see GPS.dix.dk in there, please use another version. If the firmware you run is older than about a month, please update it.
5. The list of products in my open letter is unlikely to be complete, those are the only ones I have been able to positively identify (using the method above). If you find out other products are affected, please email me.
6. We do have a number of very interesting sections of our penal code here in Denmark that are very likely to apply. Only problem is, they havn't been tried in a court yet. So I have to persuade an overworked criminal inspector to raise a criminal case against a foreigner over a, lets face it, quite small monetary amount. Then I have to spend a lot of time making sure that we convince a judge who have never heard of NTP that they are guilty and then if I win, I can see some D-link manager make a checkmark in their pocket book: "Remember to not visit Denmark under true name". I have better things to use my life for.
I can see a couple of hits from a C-class belonging to "D-Link Irwine": please escalate this guys, your bosses don't read slashdot.
Thanks for all the supportive email.
Poul-Henning
Poul-Henning Kamp -- FreeBSD since before it was called that...
1. Buy the domain name off this poor guy / arrange for alternate hosting if it can't be sold.
/. community to set up an alternate server.
2. Take a collection from the
3. Wait a month for all the legitimate users to switch to a new URL.
4. Fire up a server at the old URL reporting Midnight, Jan 1, 1900
5. Let D-Link deal with users accusing D-Link of failing to sell a Y2K compliant product in 2006.
"Live Free or Die." Don't like it? Then keep out of the USA
He discovered a problem.
He contacted the company causing the problem.
He explained the problem, and simply asked them to fix it.
They didn't.
They put him off.
They threw a lawyer at him to threaten him.
They offered 'compensation' that didn't come close to covering his costs.
He was trying to do it all quietly and nicely, not crusading, and they wouldn't have it.
So instead of going through the often extremely troublesome and lengthy legal procedings (which are even worse than normal since this is an international case), he was hoping to publically embarrass the company into fixing the problem they caused. Seems like a reasonable attempt at a speedy solution, not a crusade.
Right, because lawyers are cheap... right.
I like how he doesn't mention any numbers.
He already has dedicated hosting, do they charge him $1 per megabyte or something?
If you'd bother to RTFA, once again, he answers how much the hosting is costing him. He talks about numbers all over the place.
" because I offer this service free of charge and NTP is a low bandwidth protocol, the organization behind the DIX has graciously waived the normal DKR 27.000,00 (approx USD 4,400) connection fee."
" the current theory is that I will have to close the GPS.DIX.dk server or pay a connection-fee of DKR 54.000,00 (approx USD 8,800) a year as long as the traffic is a significant fraction of total traffic to the server."
" I owe $5000 to an external consultant who helped me track down where these packets came from."
" I have already spent close to 120 non-billable hours (I'm an independent contractor) negotiating with D-Link's laywers and mitigating the effect of the packets on the services provided to the legitimate users of GPS.dix.dk."
" Finally I have spent approx DKR 15.000,00 (USD 2,500) on lawyers fees trying to get D-Link to negotiate in good faith."
" If I closed the GPS.dix.dk server right now, wrote off all the time I have spent myself, then my expenses would amount to between DKR 45.000,00 and DKR 99.000,00 (USD 7,300 to 16,000) and several hundered administrators throughout Denmark would have to spend time reconfiguring their servers.
If on the other hand we assume I leave the service running and that the unauthorized packets from D-Link products continue for the next five years, the total cost for me will be around DKR 115.000,00 + 54.000,00 per year (approx USD 18,500 + USD 8,800 per year) or DKR 385.000,00 over the next five years (USD 62,000). " block the NTP traffic from anything outside his network if it is sooooo expensive for him. You can do that at the ISP level in most cases.
He also mentions how blocking traffic is not feasible, and why, IF YOU'D BOTHER TO READ THE FUCKING ARTICLE. Learn how to read or STFU about him being an asshole.