Slashdot Mirror
D-Link Firmware Abuses Open NTP Servers
DES writes "FreeBSD developer and NTP buff Poul-Henning Kamp runs a stratum-1 NTP server specifically for the benefit of networks directly connected to the Danish Internet Exchange (DIX). Some time last fall, however, D-Link started including his server in a hardcoded list in their router firmware. Poul-Henning now estimates that between 75% and 90% of NTP traffic at his server originates from D-Link gear. After five months of fruitless negotiation with a D-Link lawyer (who alternately tried to threaten and bribe him), he has written an open letter to D-Link, hoping the resulting publicity will force D-Link to acknowledge the issue. There are obvious parallels to a previous story, though Netgear behaved far more responsibly at the time than D-Link seem to be."
From TFA: "A number of D-Link products, so far I have at least identified DI-604, DI-614+, DI-624, DI-754, DI-764, DI-774, DI-784, VDI604 and VDI624, contain a list of NTP servers in their firmware and using some sort of algorithm, they pick one and send packets to it."
NTP server use is tiered. So client PCs are not supposed to hit the tier 1s, they should hit 2nd tier or a local ntp server.
You don't use the root DNS servers for all your DNS requests, right?
Yes, you're confused. And, you didn't read the article. The author is pissed because he's running an NTP server intended to be accessed only by Danish networks, and for use by servers, not clients. D-Link products are only marketed to clients, and not just Danish clients.
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
If there's one thing I hate more than incompetence, it's people who don't care that they are incompetent and carry on churning out crap regardless of the problems it causes others.
According to this page, D-Link have an office operating in Denmark. This makes them subject to Danish law whether they like it or not. I don't know whether Denmark's computer crime laws cover this, but it wouldn't surprise me.
Bogtha Bogtha Bogtha
He says that such a solution is hard to implement on Cisco, and would be too CPU intensive. FTFA: "Filtering the D-Link packets requires inspection of fields which are not simple to implement in Cisco routers, and in particular such filtering seems to send all packets on the interface through the CPU instead of fast switching, so ingress filtering the packets at the ingress of AS1835 is totally out of the question."
The pool guys have been talking of implementing a $company_name.vendor.ntp.pool.org setup. Having the $company_name specificity would allow them some leeway if an individual vendor does something silly. I don't know if any vendors have bought into this though.
He hosts a NTP server with the intention of it being used by a certain audience. He's not pissed people outside of that audience are using the server, he's pissed that D-Link decided to abuse the service he's providing and now the overwhelming majority of the people using his service are outside the intended audience.
Sorta like how server admins get pissed when an article posted on their site causes them to be Slashdotted.
And honestly, the fact that D-Link is acting in the way it is while he trys to get them to resolve the issue probably isn't helping matters.
Then again, as a former owner of a D-Link product which rebooted itself anytime I went over 50 simultaneous connections (think P2P), I don't doubt they'd be too cheap to actually just run their own.
The DI-624+ is not on the list and it is possible to manually change the NTP server which the router uses.
A good idea, but not easily doable, since the allowed networks include most of Denmark. He would have to filter traffic based on the AS of the sender; this would require a full BGP feed and probably also a continuously updated mirror of the RIPE database.
I own a D-Link Ethernet ADSL modem and guess what, the local IP adress is fixed to 192.168.0.1. Nope, no changing that thing. If I had known beforehand... I had to completely renumber my network. I only had 8 NICs and two LANs but was pissed off nevertheless.
I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
Sorry to correct your rant, but he does say in TFA that the offer was so low that it didn't even cover his costs. That would be a good enough reason to say no wouldn't it?
I already have done a complete 180 on recommending D-Link, since much of the D-Link equipment I use and work with has failed spontaneously.
And that was BEFORE this.
.... Well, if you read the article....
It's not just about money, it's also about client routers using bandwidth meant for BGP routers used by ISP's. It's a public network, but one intended for ISP's to transfer Data, not for Client use.
He is asking for some reimbursement for the troubles he's endured, but D-Link is saying he is extorting them.
IMHO, it is a problem D-Link did cause by their incompetence, and what is being asked is reasonable. The problem won't go away totally, because it relies on the average joe customer to actually update firmware, and now he has to deal with the situation for a long time to come. To be able to continue his "free" service, he may now have to pay for bandwidth that was free to him before D-Link wrongly implemented a protocol feature in some of their routers.
just a correction, I sorta got stratum 0 confused in there, it should be lowered by a stratum, but honestly many recommend you connect to stratum 2 servers to lighten the load on the stratum 1 who's main purpose should be time distribution. (or high presision for those in need)
FreeBSD uses pf (well, it can use pf if you want to) as a packet filter. It has the wonderful option to filter traffic according to the OS fingerprint, as in you can block traffic originating from specific operating systems. I'd advice this guy to block all traffic from these dlink devices.
If there's no fingerprint on record yet you could generate it yourself, it's not that difficult to generate one.
Reality has a notoriously liberal bias -- Stephen Colbert
RTFA. He discusses this.
1. He's already out a bunch of money trying to figure out what happened.
2. He could change the DNS name, but then every legitimate user would have to change their configuration, and there's no guarantee D-Link wouldn't just update the firmware with the new name.
Actually there are some pretty good alternatives out there.
I have been using and recomending both SMC and Asante products. They work flawlessly and the price is good too.
That was Belkin.
Because there are ~2000 legit users of his ntp server. But in the end that is probably the solution he'll have to do... rename his ntp server, allow legit users to update their config and then point gps.dix.dk at a collection of boxes on D-Link's network.
If the govt becomes a lawbreaker, it breeds contempt for law, it invites man to become his own law, it invites anarchy
Let me clarify a number of details here.
1. My server has not replied to the packets sinde the CodeRed virus/worm abused NTP servers to coordinate attacks. That was a couple of years ago. I doubt D-Link ever even tried to test this.
2. NTP is a timing protocol. You do not want to do expensive and timeconsuming filtering on the packets because that disturbs your timing performance.
3. If I have to sue D-Link, it will be either in USA or Taiwan. Both their Danish marketing office and the UK european office will be able to deflect a lawsuit to their mothership.
4. If you download a firmware file from D-Link, it is often a ARJ archive. unpack that and run strings. If you see GPS.dix.dk in there, please use another version. If the firmware you run is older than about a month, please update it.
5. The list of products in my open letter is unlikely to be complete, those are the only ones I have been able to positively identify (using the method above). If you find out other products are affected, please email me.
6. We do have a number of very interesting sections of our penal code here in Denmark that are very likely to apply. Only problem is, they havn't been tried in a court yet. So I have to persuade an overworked criminal inspector to raise a criminal case against a foreigner over a, lets face it, quite small monetary amount. Then I have to spend a lot of time making sure that we convince a judge who have never heard of NTP that they are guilty and then if I win, I can see some D-link manager make a checkmark in their pocket book: "Remember to not visit Denmark under true name". I have better things to use my life for.
I can see a couple of hits from a C-class belonging to "D-Link Irwine": please escalate this guys, your bosses don't read slashdot.
Thanks for all the supportive email.
Poul-Henning
Poul-Henning Kamp -- FreeBSD since before it was called that...
Startum 1 servers aren't "expensive" nor are they a limited resource; any time server that pulls its timebase from GPS, for example, is stratum 1.
Only on slashdot can a posting be rated "Score -1, Insightful".
No, he can't "just firewall the server" and "tell the few people that would affect". There are thousands of legitimate users distributed across thousands of ASes covering thousands of IP ranges which may change from day to day or even hour to hour. His server is directly connected to the core switch at the Danish Internet Exchange, where all major Danish networks exchange BGP routing information and domestic IP traffic, and its purpose is to provide a stratum-1 reference for NTP servers on these networks. To determine which IP ranges may legitimately access his server, he would need a full BGP feed and a continuously updated copy of all as-block and aut-num records in the RIPE database.
Now that you look at your ethernet sniffs (I assume you just went running off and ran ethereal) look at the source ethernet address... Hmmmmm - doesn't that look familiar, like maybe it looks kinda like your first hop routers MAC address.
Nice try -
Thank you, Come Again
And please read either Stevens or Comer before posting on networking topics again
I have mod points and I am not afraid to use them
http://www.pool.ntp.org/
These people may disagree. This doesn't change the fact that D-Link makes shitty firmware.
Dear Sir or Madam,
I have learned of your company's persistent unwillingness to deal with a serious design flaw in a growing range of your products. This flaw is disrupting internet services for a large number of users. You have been informed in detail of the problems you are causing, and you have done nothing of substance to resolve the issue and compensate those involved.
The issue I refer to is described in the "open letter to D-Link", available at http://people.freebsd.org/~phk/dlink/.
Until this problem has been resolved in a professional and universally satisfactory manner, I will not purchase any D-Link products and will act in my capacity as an I.T. professional to discourage others from doing so.
Sincerely,
Writing Style Nazi
(I'm not a spelling nazi, so please check this again)
PHK have (of course!) considered moving his box to a new DNS name, the problem lies in the way it is used:
By moving it, he'll require every single BGP router in Denmark to be reconfigured, if you read his Open Letter you'll notice that he has considered and rejected this option as unworkable.
Terje
(Who's been hosting windows ntp binaries for several years now, at http://norloff.org/ntp/)
"almost all programming can be viewed as an exercise in caching"
customerservice@dlink.com
webmaster@dlink.com
analysts@dlink.com
sale@dlink.com
broadband@dlink.com
bdm@dlink.com
oem@dlink.com
productinfo@dlink.com
hr@dlink.com
edusales@dlink.com
si@dlink.com
Someone else replied, but let me actually EXPLAIN.
pool.ntp.org is a collection of volunteer NTP servers, served up via DNS. You should not expect to get meaningful results from pointing a Web browser at such a host name, but because it is random, you could end up hitting Amazon.com (assuming they volunteered) or some guy that just set up an Apache server.
http://www.pool.ntp.org/ is what you meant, as a simple google search for "pool ntp" would have told you.
How long do you think it would take most people to even notice? I bet most people have never heard of NTP.
How many people do you think are likely to upgrade their firmware? The ones they've already shipped are doing this.
Hint: If this is a default setting that people are unaware of, they will never cause a suppport call to happen, but they will continue to affect this guys bandwidth bill.
As he pointed out, had D-Link done this differently, they could have redirected the NTP from within their own organization. As it is now, it's a burned in value that isn't likely to change.
Lost at C:>. Found at C.
Startum 1 servers aren't "expensive" nor are they a limited resource; any time server that pulls its timebase from GPS, for example, is stratum 1.
The problems come where you have embedded devices which have a small number of (S)NTP servers hardcoded. This can easily create a distributed denial of service, especially since a coder likely do this is also likely to make other mistakes in their implimentation.
If the idea is for the device to autoconfigure it needs to be picking randomly from a large list or able to discover which server(s) it should be using. e.g. DHCP, SLP, etc.
It's a stratum-1 NTP server. Stratum-1 NTP servers are *ONLY* supposed to be used by other stratum-1 NTP servers and by stratum-2 NTP servers, *not* by any random device on the internet. A LAN router should *NEVER* be using a stratum-1 NTP server; it should be using a stratum-3 NTP server if possible, or *maybe* a stratum-2 server, with special permission, under unusual circumstances, if there is no stratum-3 server available. If D-Link won't do anything, this guy's going to have to notify everyone who runs a stratum-1 or stratum-2 server in Denmark, give them time to reconfigure, and then shut down the service.
Cut that out, or I will ship you to Norilsk in a box.
Right, because lawyers are cheap... right.
I like how he doesn't mention any numbers.
He already has dedicated hosting, do they charge him $1 per megabyte or something?
If you'd bother to RTFA, once again, he answers how much the hosting is costing him. He talks about numbers all over the place.
" because I offer this service free of charge and NTP is a low bandwidth protocol, the organization behind the DIX has graciously waived the normal DKR 27.000,00 (approx USD 4,400) connection fee."
" the current theory is that I will have to close the GPS.DIX.dk server or pay a connection-fee of DKR 54.000,00 (approx USD 8,800) a year as long as the traffic is a significant fraction of total traffic to the server."
" I owe $5000 to an external consultant who helped me track down where these packets came from."
" I have already spent close to 120 non-billable hours (I'm an independent contractor) negotiating with D-Link's laywers and mitigating the effect of the packets on the services provided to the legitimate users of GPS.dix.dk."
" Finally I have spent approx DKR 15.000,00 (USD 2,500) on lawyers fees trying to get D-Link to negotiate in good faith."
" If I closed the GPS.dix.dk server right now, wrote off all the time I have spent myself, then my expenses would amount to between DKR 45.000,00 and DKR 99.000,00 (USD 7,300 to 16,000) and several hundered administrators throughout Denmark would have to spend time reconfiguring their servers.
If on the other hand we assume I leave the service running and that the unauthorized packets from D-Link products continue for the next five years, the total cost for me will be around DKR 115.000,00 + 54.000,00 per year (approx USD 18,500 + USD 8,800 per year) or DKR 385.000,00 over the next five years (USD 62,000). " block the NTP traffic from anything outside his network if it is sooooo expensive for him. You can do that at the ISP level in most cases.
He also mentions how blocking traffic is not feasible, and why, IF YOU'D BOTHER TO READ THE FUCKING ARTICLE. Learn how to read or STFU about him being an asshole.
I ended up writing a simple perl script to handle the updates instead.
Here's a ready-made Perl-scripted daemon for this kind of stuff: http://ddclient.sourceforge.net/
USE HOT GRITS WITH STATUE OF NATALIE PORTMAN (NAKED AND PETRIFIED)
Dear Zardo,
I never use anonomity to hide behind, I have no opinions of which I am ashamed.
You seem to be missing a very fundamental point in this: I live in Denmark.
Danish lawyers are not allowed to work on contingency. You get your bill first, then the verdict.
Therefore, $2500 in lawyers fees is actually not very much over here. If I tried to get this case in front of a judge, I would have to pay something like ten times that.
Furthermore, you seem to question a lot of things you could have determined for yourself by reading the actual letter I wrote.
Finally, I have probably done more for the internet and open source than you will ever be able to imagine so if you want to paint me as a simple extortionist, you may have a bit of trouble making people belive you.
In all likelyhood, I wrote the function which protects your password.
Poul-Henning
Poul-Henning Kamp -- FreeBSD since before it was called that...
1. D-Link update with a USER_AGENT of 'client/1.0' (how original). This violates all published dynamic DNS specifications, be it DynDNS, TZO.COM, no-ip etc
2. DynDNS blacklists these D-Link routers (block all agents using 'client/1.0')
3. D-Link responds by changing USER_AGENT to be '$username/1.0' (where $username is your ddns username).
I'm NOT kidding you. They took the time to do a string change to circumvent blocking, but not solve the problem! Fuck, why not set the USER_AGENT to 'Mozilla' while you're at it. Jerks.
(earth to D-Link... send at LEAST 'dlink_piece_of_shit/1.0'... or better yet send 'dlink [router:$routerver/firmware:$fwver]' so maybe only SOME of your routers get blacklisted. )
DynDNS blocks D-link routers. TZO, and no-ip currently do not.
Who pays for the customer's phone angst? Not D-Link... they've already set Support expectations SO LOW no professional will talk to them.
I even put one of their fucking routers WAN ports under a packet sniffer, and SENT THEM A HOW-TO on fixing their router! My request was last seen in Mumbai-istan-dia by a script reader named 'Steve'. These people follow RFCs as well as Myspace or GoDaddy. Outsourced Customer service is not going to be proactive about protecting a reputation of their employer's employer.
D-Link have 6 "OEM developers" who are outside contracters. When they have to fix a bug in one OEM's product, there is NO CODE SHARING with the other development teams. It's the customer's fault for not reporting the bug in every affected model, you see...
Why should D-Link care about stealing anyone's bandwidth from their own firmware bugs?
From their perspective, these things still fly off the shelf at Best Buy.
You can enable dynamic DNS in a D-link, and if you do NOT set the username and password (meaning the DDNS will fail), they HAMMER on the update server. Oh gee, a failed update means RETRY right?
The motherfucking OEM coders in Taiwan skip reading the specs because they are only written in English.
If QA doesn't complain, ship it.
Disclaimer: I work for one of these dynamic DNS companies. Avoid D-Link... go with Linksys or SMC or Buffalo or US Robotics. For the love of god stay away from D-Link PLEASE!
The guy had help in finding out who it was who abused his service, by Richard Clayton, he writes in his blog about this: "on a typical day he'd receive 3.2 million bad packets (that's 37 a second!). "
Here he explains how he traced down who was behind, what he calls a DDoS attack: His blog
If Google really cared they would fix Android Chrome to reflow text, instead of discriminating