Slashdot Mirror
Crowdsourcing Confirms: Websites Inaccessible on Comcast
My first clue came when a friend of mine set up the website http://www.helpmatt.org/ and asked her friends to donate. I said the website appeared to be down; they replied back that it was working fine for other people — and I narrowed it down to Comcast DNS servers not resolving the hostname www.helpmatt.org correctly. When I accessed the same website over my Frontier DSL connection, it worked. (I had recently signed up for Comcast cable Internet to save money over DSL, but I kept my DSL connection "just in case" something went wrong. At the time, I thought maybe I was being paranoid -- how hard could it be for a cable company to just run a straight Internet connection to my house and not screw anything up? Hollow laugh.)
I put out an informal survey to my Comcast-using friends, and a few of them said they couldn't access the website either. Still, I thought, this wasn't enough evidence that it was Comcast's fault; maybe the hostname was only resolving intermittently, and just by sheer coincidence it happened to be up when all of my non-Comcast-using friends tried it? I was about to do a more formal experiment, and recruit a larger sample of testers through Amazon Mechanical Turk to test whether the site was inaccessible to other Comcast users, when the problem spontaneously fixed itself and suddenly the website became accessible 100% of the time to everyone.
But, my curiosity had been piqued. Was there something wrong with Comcast's DNS servers -- whether deliberate or not -- that was causing other websites not to resolve correctly? I wrote a perl script to take a sample of websites -- part of the same list that I had used to find websites that were mis-blocked as 'pornography' by Smartfilter — and attempt to resolve them using both Comcast's main DNS server (75.75.75.75) and one of Google's public DNS servers (8.8.8.8). (You won't be able to do this experiment yourself unless you have a Comcast Internet connection, because while Google's DNS servers accept queries from anywhere, Comcast's DNS servers will refuse queries from any IP address not assigned to one of their customers.)
The script ran through a few hundred hostnames and flagged anything that failed to resolve on Comcast but resolved correctly on Google, although most of these were false positives caused by Comcast's DNS servers being temporarily unresponsive. But after running through the list of false-positives repeatedly, I found the first website that consistently failed to resolve on my Comcast Internet connection while resolving on Google: http://www.021yy.org/.
The website is for a second-hand furniture store in Shanghai; I have no idea what the domain "021yy.org" has to do with the business. (Perhaps the IP address that the domain name resolves to used to be occupied by a different website, and that IP address was inherited by the furniture store but the old hostname still points to it.) The hostname www.021yy.org resolves to the IP address 116.251.210.33 (for *ahem* non-Comcast users, that is), which according to the Asia Pacific Network Information Centre is part of a block of IP addresses assigned to a hosting company in Singapore. I'm not blocked from accessing the IP address of the website over Comcast; I can ping and send web requests to the IP address 116.251.210.33 with no problem. Only the hostname fails to resolve. (I can still access the site by using a VPN or a proxy server.)
So, I created a survey on Amazon Mechanical Turk, asking people three questions:
- Can you access the website http://www.021yy.org/?
- If you can't access the site, what error message does your browser give you?
- What provider are you using?
and offered 25 cents to every user who filled out the survey, up to a maximum of 50 people. Amazon Mechanical Turk, if you've never used it before, lets you create low-payment tasks and outsource them to a crowd of workers. Like any simple and powerful tool, it can be used for purposes that the original creators probably never imagined (presumably including this experiment), and someday I'd like to look into the most creative and bizarre things people have done with it. (Although, in this case, it seems like the site may not have done a great job of matching this task with available workers. Only 20 people filled out my survey in the 24 hours after I created it -- surely, out of all the available Mechanical Turk workers, there were more than 20 people who would have been interested in doing a simple website accessiblity check for 25 cents?)
20 unique users filled out the survey and reported:
- Out of the 14 non-Comcast users, 100% of them were able to access the site.
- Out of 6 Comcast users, 4 of them were blocked from accessing the site, and reported errors symptomatic of DNS failures ("Oops! Google Chrome could not find www.021yy.org" or "Server not found. Firefox can't find the server at www.021yy.org").
Even with such a small sample, that's enough to conclude that it's not a coincidence. (The real question is how two out of those six Comcast users were able to access the site at all. Maybe they're in a region of the country that's assigned different DNS servers. If I did the survey again, I'd ask people to include where they were living.)
So Comcast users -- at least some of them, probably most of them -- are blocked from accessing certain websites, which are perfectly accessible to users on other providers. I "only" had to test a few hundred domain names before finding one that would consistently fail to resolve on Comcast while resolving successfully on other companies' nameservers. With hundreds of millions of distinct websites "out there," if the same proportion holds, that would suggest that there about a million or more websites similarly affected. And that's not even counting all the other sites — like helpmatt.org, and also including some of the sites in my sample — which apparently resolve 100% of the time on other providers while sometimes failing to resolve on Comcast, but where the failure was not consistent enough to use them as a test case for the Mechanical Turk survey.
Unlike, say, the kerfuffle over Comcast threatening to de-prioritize content delivery from websites that don't pay them a fee, it's unlikely that Comcast is meddling with traffic intentionally here (especially since the sites' IP addresses are not blocked). It's more of a demonstration that if a company is sufficiently big and if it's sufficiently hard to prove that a problem is being caused on their end, the problem can exist for a long time without being solved. I called Comcast tech support after I discovered that sites were blocked on their network but not on other providers, and said that the problem really needed to be brought to the attention of the higher-ups, but tech support was adamant that it was impossible for a member of the public to reach anybody higher up than the call center.
Even if the number of affected sites is huge, at least it's only a small percentage of websites — I did have to run my script on a few hundred sites before I found one that appeared to be resolving on other DNS servers but not on Comcast. But that likely would have provided scant comfort to my friends who set up the helpmatt.org site, when they were urging people to visit the site and donate, and 25% of potential visitors were unable to reach the page. When it's your website, it's kind of a big deal.
Stop using your ISP's DNS
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
I stopped using comcast DNS servers years ago, and have avoided many an "outage".
I remember several large DNS outages on comcast that I was completely unaware of for hours or days, until some mention came up.
I have been using OpenDNS mostly, but I fall back to the google DNS servers if something there flubs up
208.67.222.222
208.67.220.220
Remember these numbers
Gasp! I can't access it through comcast? How ever will I buy office chairs in china without 021yy.org?!?! It's SO much better than those humps over at 022yy.org.
(In case the link gets slashdotted, it's a website for office furniture in Chinese. At least according to google translate.)
With hundreds of millions of distinct websites "out there," if the same proportion holds, that would suggest that there about a million or more websites similarly affected.
Why are you assuming that this scales linearly? Are you suggesting that this is a technical glitch? If the websites are blocked due to the nature of their content it most certainly won't scale in a linear fashion.
It's not just Comcast. No ISP I have used has ever run a reliable DNS service. 8.8.8.8, 8.8.4.4 is your friend.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
That's not a solution, that's a workaround. The author is clearly trying to define the actual problem and make a supposition as to the cause, not just find a way to make the symptoms stop happening.
"Tell me doctor, with all of your defenses, are there any provisions for an attack by killer bees?"
My ISP, who is not Comcast but another major American ISP, also blocks certain websites via DNS failures. Simply switching DNS to Google's DNS servers or FreeDNS resolved the problem.
Do not use comcast DNS... just use googles.
https://developers.google.com/...
Good idea -- otherwise, Google might miss out on some of your browsing activity if you're using another browser, use their DNS to make sure they can capture all of your activity.
So, if you do the DNS query from another provider's DNS, can you get to the website over Comcast? Seems like a basic troubleshooting step that was missed. At least not mentioned in the extended summary.
Learn to love Alaska
Interesting. I don't always want to be messing with my DNS setting every time I get a 404 not found.
What is needed is a quick way to temporarily try using a different DNS, to see whether that's the problem.
http://www.geoffreylandis.com
if you do a compare between two DNS servers then you are bound to also come up with differences that show how outdated one server is compared to the other... There has to be many new domains registered / re-registered and associated / re-accociated with a new IP every minute, if you run the script for long enough between two different snapshots you are bound to find one of these...
So my appropriately verbose question in response to your post is: how often do you think google and comcast update their DNS servers, and do you think they update at exactly the same time... I know ISPs like to filter stuff... just wondering if your method is sound.
DNS is a theoretically good system and one that we obviously all rely on every day. However, so many DNS implementations from the registrar level down to your cheap little wifi-router-all-in-one box that connects to your ISP are so totally broken. I think the way this is written is pretty trollish and should instead have focused on the wider question of how we can advance to where so many devices and programs that have to deal with name resolution will act more to-spec and consistently. Comcast should take some heat here for a partially broken DNS implementation, but without better evidence, I see no intentional evil in this particular story.
Last time I had to talk to anyone in the company I had to explain to the tech how DOCSIS modems worked. You will never get an individual from that company on the phone who knows enough to give you a real answer. Turnover is too high in call centers, and people who know the answer are not on support phone detail.
You can set any DNS you want on your computer. You don't have to use the one handed out by the ISP's modem or router.
"Tell me doctor, with all of your defenses, are there any provisions for an attack by killer bees?"
OpenDNS hijackes NXDOMAIN failures, which is one of the big reasons to drop many ISP's DNS in the first place. I don't want to get into evaluation of motivation and such, but the effect is the same.
That is interesting. When I read the article.... and I am ready to hate on comcast at any time, they are my provider for various reasons (including me being lazy yes) but I am not a huge fan of them.
That said, I couldn't help but think... that is an odd domain name, and its not like it makes any sense that it would be blocked. It looks like the kind of randomly named domain a phisher might use, which makes me wonder... maybe this domain was blocked due to being part of some botnet or equivalent and then later became owned by the current owners? (not cleaning up things like that is hardly a new or unique issue)
Now I see your post and I think.... you may be on to something. I think that unless someone can find rhyme or reason for bans, then we should probably assume incompetence rather than malice. I mean, its not like there is a pattern of blocking based on content or ownership, they are not even competitors of comcast unless they have some diversification plans that I wouldn't have ever expected.
"I opened my eyes, and everything went dark again"
Let me understand this correctly. You found Comcast's DNS isn't perfect and doesn't resolve some names. It does not appear to be malicious in any way, as the two domains you find affected are a foreign furniture store, and your friend's brand new website. It's fairly obviously a bug.
So: you call Comcast Tech support, demand to talk to the Boss of Comcast, and then write a 10,000 word article (I didn't count) about it on Slashdot where you know 90% of the readers will take "Websites inaccessible on Comcast" as meaning "OUT OF CONTROL MEGACORP MONOPOLIST COMCAST IS CENSORING WEBSITES!!!"
This makes sense to you? This is what you do? Really? Really?
Just curious, but that time you got a duff cable modem and had to send it back, did you write a 60,000 article on how Comcast has banned you from the Internet, and did you demand to speak to the PRESIDENT OF THE INTERNET? When it rained that one time and you attempted to tune in the cable TV, only to find many of your channels were inaccessible, did you write a 75,000 word article on how COMCAST IS DROPPING CHANNELS and did you call tech support demanding to talk to THE LORD HIGH RULER OF TV?
I think I've found an article where the discussion would be likely improved for once if the Betoddlers spammed it with anti-Beta comments.
You are not alone. This is not normal. None of this is normal.
I started Comcast service about a year ago, and supplied my own modem and router. They have not done anything like forcing me to use their internet hardware.
not akamai
there was an issue with itunes and google dns years ago. apple uses akamai for their CDN and people using google dns when they rented movies on apple tv would stream from 3000 miles away instead of a local copy because google's DNS IP's are virtual IP's and the true IP passed to who ever you are trying to access may be any server around the world
I don't know if this is an issues with Comcast, but there are ISPs who force all DNS traffic to use their servers. It was a constant frustration when I was stuck with Excede (a US satellite internet provider).
Failure to follow this advice may result in non-deterministic behavior.
How does Comcast's DNS look like when tested by namebench?
Does it find the same problem?
Actually, there are a few major GTM (Global Traffic Management) schemes that do use the IP address of your DNS server, rather than your actual IP. They basically abuse the DNS system with super-short TTLs and give a different response to the DNS query based on the IP of the downstream DNS server. So, if you use a DNS server located on the east coast of the US when you're on the west coast, you'll get an east coast server even if that service has a west coast datacenter available.
This is done primarily to free companies from the burden of having to design proper geolocation into their app/service, turning it into a more plug-n-play solution while breaking several of the finer points of DNS (like proper caching). This type of traffic management could easily be contributing to Comcast's DNS troubles, as it drastically increases load on the entire DNS infrastructure. Paul Vixie did a good detailed write-up about this type of traffic management a few years back. Unfortunately it's probably here to stay, and is used by some very major corporations and online services.
If you want the most reliable DNS service, and want to be directed to the closest servers for the services you use, your only real option is to run your own recursive name server. A simple caching name server isn't enough, and will curse you with many of the same problems you see from your upstream. Fortunately, recursive name servers are pretty simple to set up, in both the *nix and Windows worlds.
The DNS for 021yy.org is rather fishy looking. The .org servers have NS records pointing to ns1.booen.com and ns2.booen.com, which have a 20 second time to live (vs. a normal 1 day TTL), which is common in botnet command & control networks. Also, the ns1/2.booen.com servers give answers to 021yy.org A lookups, but return NXDOMAIN for NS lookups (which is completely bogus; NXDOMAIN means that 021yy.org does not exist, not that it doesn't have NS records, which would still be bogus).
The NXDOMAIN for NS records would cause many caching servers to cache NXDOMAIN for all records (not just NS), which would cause the domain to not resolve (depending on the order things were looked up). Basically, I don't see this as a Comcast problem, but rather a problem with the DNS servers for 021yy.org. This may be accidental (although AFAIK no normal DNS server would reply with A records but return NXDOMAIN for NS records), but looks possibly like it is intentional and possibly part of a botnet C&C. There's a lot of that going on lately.
don't use the fast ISP? like you have a CHOICE??
I can pick dsl (dog slow link; that's what DSL means) or I can pick comcast.
what makes you think people in the US can actually choose an isp? they are all based on where you live. you'd have to MOVE to be able to choose an alternate.
not sure why you posted this BS but its not helpful in the least...
--
"It is now safe to switch off your computer."
You can use downforeveryoneorjustme.com, though it will use its own DNS and routing so it will still require you to figure out which of those is the problem.
Say, that's a nice site. Wish I had mod points, I'd moderate you "informative".
http://www.geoffreylandis.com
Wish I had mod points, I'd moderate you "informative".
You would if you made more interesting remarks than this.
If someone asked me to "go check a website" and the site URL looked like some random malware host, I'd probably not choose his 25 cent task either. What is this guy smoking?
Wish I had mod points, I'd moderate you "informative".
You would if you made more interesting remarks than this.
Wish I had mod points, I'd moderate you "insightful".
Turns out that for some reason, their DNS servers were making a query for the name of my nameservers as listed in the registrar database. When those failed, it dropped any caching of the address like a hot potato, thus resulting in very spotty name resolution. Using Google's DNS worked just fine, if a bit slower due to the lack of multi-hosting.
So basically, if the registrar has example.com's nameservers listed as foo.example.com = 10.1.1.1 and bar.example.com = 10.1.1.2, AT&T's DNS will query 10.1.1.1 to look for foo.example.com. If that DNS server lists itself as ns1.example.com, but does not resolve foo.example.com, AT&T's nameserver will think something is fishy and decide you don't exist at all.
This was a pain in the ass to figure out, but everything has been fine since I fixed that. I would still like to find a place where this behavior is documented, because I was only able to discover it by turning debug logging on for my nameserver. I also found out that someone in Germany had been using it as their primary DNS for who knows how long, so I shut off recursive searches from outside my LAN.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
This was probably just a negative cache entry. Someone on Comcast (possibly you) probably tried to look up helpmatt.org before it was propogated to all the root servers, and 75.75.75.75 got a lookup failure and cached it. Negative caching is part of proper DNS operation and it can last a while. DNS is full of delays like this.
FYI... It's working just fine now.
root@atomrouter:~# host helpmatt.org 75.75.75.75
Using domain server:
Name: 75.75.75.75
Address: 75.75.75.75#53
Aliases:
helpmatt.org has address 192.155.89.14
helpmatt.org mail is handled by 20 alt1.aspmx.l.google.com.
helpmatt.org mail is handled by 30 aspmx3.googlemail.com.
helpmatt.org mail is handled by 30 aspmx2.googlemail.com.
helpmatt.org mail is handled by 20 alt2.aspmx.l.google.com.
helpmatt.org mail is handled by 10 aspmx.l.google.com.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
Wish I had mod points, I'd moderate you "informative".
You would if you made more interesting remarks than this.
Wish I had mod points, I'd moderate you "insightful".
I wish I had mod points, I'd moderate you "Underrated". Your comment has a je ne sais quoi.
So you invite everyone in the world to submit their domain name and IP address on postcards?
Yes. HOSTS files. Exchange HOSTS files. Manually merge and edit them.
TBH, I thought DNS was going to be a fad.
(Yes, I'm capitalizing HOSTS because that's what it was called on the pre-historic TOPS-20 system I was using. I also thought that commie-pinko "unix" thing was also going to be a fad.)
Welcome to the Panopticon. Used to be a prison, now it's your home.
Hi - Jason from Comcast's DNS team here. First off, we have a nifty website @ http://dns.comcast.net/ where you can check our cache and find a form to contact us directly. Let's breakdown the issues with www.021yy.org. 1 - Sub-optimal TTL: The DNS admin is not doing themselves any favors; the TTL for www.021yy.org seems to be set to 60 seconds. That will cause recursion every 60 seconds or less from US-based DNS servers to authoritative servers in China. I recommend a more industry standard TTL to enhance cacheability of these records and minimize global recursions at this frequency. I would suggest no less that 5 minutes (300 seconds in the DNS record) or even as much as 1 hour which is usually fine (3600). 2 - Auth servers seem to be in China? If you expect many users of www.021yy.org in the US, you may want to add at least one authoritative name server in the US so that when recursion does need to occur that it is faster than US-to-China transit time. 3 - Are the auth servers responsive? I get NXDOMAIN responses when asking several recursive servers, such as Google's. Macintosh-3:~ jason$ dig @8.8.8.8 021yy.org ns ; > DiG 9.8.3-P1 > @8.8.8.8 021yy.org ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER> DiG 9.8.3-P1 > @8.8.8.8 slashdot.org ns
; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER- opcode: QUERY, status: NOERROR, id: 26387 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;slashdot.org. IN NS ;; ANSWER SECTION:
slashdot.org. 19088 IN NS ns2.p03.dynect.net.
slashdot.org. 19088 IN NS ns4.p03.dynect.net.
slashdot.org. 19088 IN NS ns1.p03.dynect.net.
slashdot.org. 19088 IN NS ns3.p03.dynect.net. ;; Query time: 17 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Tue Mar 11 17:42:38 2014 ;; MSG SIZE rcvd: 116
In any case, we're flushing our cache right now just in case but I am not sure that will solve a deeper DNS issue with the authoritative DNS service for this domain.
The short TTLs aren't really needed for doing geolocation stuff (it's not like a downstream dns server is going to physically move which keeping it's cache), the main reason for using short ttls is so you can quickly move traffic to another datacenter in the event of a failure or overloading.
The alternative is to move the traffic around using routing protocols, but that has costs of it's own.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Bump.
Seems like this is a flaky domain with some messed up settings. There's a very good chance comcast cached an NXDOMAIN. Wouldn't be too surprised if something similar had happened with his little personal site. Many DNS servers serving large volumes of users ignore low TTL's and cache longer than normal. It only hurts edge cases they don't care much about since large established sites do not rely on fast DNS updates for things like load balancing or failover.
Use another DNS server is still a good suggestion.
Without a more extensive test (1 in a few hundred random sites is not a statistically good sample size... could have hit the same random one out of a million, for example), this doesn't really say much.
The poster put enough time into this that it shouldn't be difficult / much more time intensive to expand the test and provide a larger list of good/bad domains. Those could also be weeded out to find those that are generally flaky or configured poorly. If any remain, then test those.
His buddies personal site didn't work for an hour or so, and some random chinese site doesn't reliably resolve... that's not enough to start the scare tactics (...that there about a million or more websites similarly affected").
Over the years I have been both a Comcast and Time Warner Internet customer in way more than 3 different cities. Avoid them if possible. No fun paying $10 more for more bandwidth and not seeing your bandwidth increase. Thanks to DD-WRT you see your actual bandwidth in real time.
Everyone should learn how to access websites they deem critical via that websites IP address alone. Its simple enough if you know the IP address, which can be discovered via the commands nslookup, dig and traceroute (tracert for you windows users). To learn more, google any of those commands and learn. Once you know the IP address using this in your browser's Location Bar (some browser installations turn off the browser's location bar, but you can turn it back on...if not use a better browser):
http://xxx.xxx.xxx.xxx/ where xxx.xxx.xxx.xxx is the IP address of the website you want to reach.
If they (cable company) will not provide you with only a cable modem, go with another provider. You want it to be nothing but a simple modem. No Wifi, no firewall, no router. Then add your own firewall/router that you control 100%.
Since they are going to throttle your cable connection anyway, see if DSL is available. It will be cheaper per month also. Go with DSL if you can not get FTTH. Funny how cable companies only offer you more when they are forced too.
If you must use cable:
Cable Modem (no Wifi, no DNS) + DD-WRT enabled firewall/router ~ is your best option.
The FCC use to define broadband as sustained bandwidth speeds above 768Kbps, that page has since been removed, wonder why? If a cable provider throttles service to below 768Kbps, at any time, should it be allowed to be called Broadband? I think NOT.
If your broadband is symmetrical, not an up to bandwidth lie, there is no business incentive to restrict, limit, throttle and reduce a customer's bandwidth perpetuating the scarcity myth lie related to Internet access. There are less than 30 FTTH communities in the USA where a residential customer can purchase symmetrical Internet bandwidth today. Thankfully more are being planned. Except in the 14 states where the Cable companies have gotten politicians to enact laws preventing competition and FTTH.
To learn more about nslookup, dig and configuring your DNSsee this Google Developer's web page. There are command examples on that page, enjoy.