Slashdot Mirror

Long-time Slashdot reader syn3rg quotes the DNS Flag Day page: The current DNS is unnecessarily slow and suffers from inability to deploy new features. To remediate these problems, vendors of DNS software and also big public DNS providers are going to remove certain workarounds on February 1st, 2019.

This change affects only sites which operate software which is not following published standards. Are you affected?
The site includes a form where site owners can test their domain -- it supplies a helpful technical report about any issues encountered -- as well as suggestions for operators of DNS servers and DNS resolvers, researchers, and DNS software developers. The Internet Systems Consortium blog also has a list of the event's supporters, which include Google, Facebook, Cisco, and Cloudflare, along with some history. "Extension Mechanisms for DNS were specified in 1999, with a minor update in 2013, establishing the 'rules of the road' for responding to queries with EDNS options or flags. Despite this, some implementations continue to violate the rules.

"DNS software developers have tried to solve the problems with the interoperability of the DNS protocol and especially its EDNS extension by various workarounds for non-standard behaviors... These workarounds excessively complicate DNS software and are now also negatively impacting the DNS as a whole. The most obvious problems caused by these workarounds are slower responses to DNS queries and the difficulty of deploying new DNS protocol features. Some of these new features (e.g. DNS Cookies) would help reduce DDoS attacks based on DNS protocol abuse....

"Our goal is a reliable and properly functioning DNS that cannot be easily attacked."

alphadogg writes: Attackers could exploit a new vulnerability in BIND, the most popular Domain Name System (DNS) server software, to disrupt the Internet for many users. The vulnerability affects all versions of BIND 9, from BIND 9.1.0 to BIND 9.10.2-P2, and can be exploited to crash DNS servers that are powered by the software. The vulnerability announced and patched by the Internet Systems Consortium is critical because it can be used to crash both authoritative and recursive DNS servers with a single packet.

You asked Carl Malamud about his experiences and hopes in the gargantuan project he's undertaken to prod the U.S. government into scanning archived documents, and to make public access (rather than availability only through special dispensation) the default for newly created, timely government data. (Malamud points out that if you have comments on what the government should be focusing on preserving, and how they should go about it, the National Archives would like to read them.) Below find answers with a mix of heartening and disheartening information about how the vast project is progressing.

LoC?
by an Anonymous Reader

So how many GB/TB is a Library of Congress? :)

Or, more seriously, how big are you estimating? Are you using raw scans or some sort of compression (JPG, PNG, etc)? What resolution are you using? Do you vary the resolution depending on the document?

What sort of meta data are you putting in?

CM: The reason John Podesta and I suggested a Federal Scanning Commission in our letter at YesWeScan.Org is we really don't know how big the holdings of the government are. I can tell you that the Library of Congress is about 32 million cataloged books (a significant increase from the 6,487 books Thomas Jefferson donated to get them started). But, this is about more than books, it is about paper records, microfilmed technical papers, video, audio, photographs, and much more.

The scale is fairly vast. The Smithsonian has 137 million objects, including about 13 million images. David Ferriero, the Archivist of the United States estimates he has over 10 billion pages of text documents, 7.2 million maps, and 40 million photographs including everything from past census records to presidential dinner menus, and that includes about 7.5 million motion pictures and sound recordings. The Government Printing Office distributes their documents to the Federal Depository Library Program, and that includes over 60 million pages of collections including the Official Journals of Government such as the Federal Register. That's just scratching the surface, and we recommended a Federal Scanning Commission to begin the process of understanding what we have (and what is worth digitizing).

As to standards? There are lots of pretty good standards on how to digitize. NARA, Library of Congress, GPO all spec out document scans at 400 dpi, for example. For photographs, moving images, and other objects, there are some pretty good and pretty detailed standards at www.digitizationguidelines.gov. I know Brewster Kahle's operation and my own tend to work off those specifications (in fact Brewster does quite a bit of scanning for the government).

As to compression? Well, I've found people tend to overcompress things. That said, sometimes the initial quality isn't that great, so a 600 dpi uncompressed scan would be silly in some cases. But, for photographs I try very hard to keep the TIFF images around and not rely on JPEG. Likewise, for audio it is really nice to keep a nice 48 khz version of your file around if you can simply because if you screw up the compression maybe somebody else can do a better job in a few years. Disk space is relatively cheap, so that isn't the barrier it used to be. For video, I rip MPEG2 at whatever it is on a DVD, when I'm actually digitizing I try to get the video bitrate up to 8-10 mbps when ripping a Betacam or Umatic. Some people think that is overkill, but I'd rather be safe than sorry.

Metadata? Well, you got to have it or you're not going to get very far when it comes to access. Many librarians have made perfect the enemy of the good when it comes to metadata and have resisted any attempt at digitization because we don't have the very best metadata we might have. I'm more in the camp of scan what you have and get as much of the metadata as you can into it. For example, we have 3,200 1000-page volumes of briefs from the 9th Circuit of the U.S. Court of Appeals. We didn't have good metadata, but we had the Internet Archive scan them anyway. Then, after we got our PDF files, I shipped those off to a double-key team in India and they broke the briefs up into individual documents and typed the metadata into a spreadsheet for me, which we hope to release soon.

My point is that sometimes you can shoehorn the metadata in after the fact or you can use a variety of techniques to pull the metadata out of the documents (e.g., smart OCR). In theory, you can use crowdsourcing to get the metadata, but so far I've not had a lot of luck persuading thousands of people to spend their time doing that kind of work. A captcha is a quick thing to do and is between you and something you want, whereas entering metadata in for videos or documents is one of those civic duty things that everybody thinks everybody else should be doing.

Total size? Brewster says a book is about 400 Mbytes (though he's very quick to point out that you could put the words in all the books in the library into a terabyte and if you're distributing PDFs, you can easily throw 130,000 full-color, searchable PDFs onto a 4 TB drive). But, you were probably asking about raw data. Here's some raw numbers:

32 million books at 400 Mbytes each is 12.8 petabytes 50 million photos at 150 Mbytes each is 7.5 petabytes 10 billion pieces of paper ("records") at 100 Kbytes each is 1 petabyte 20 years of video at 8 mbps is only 630 Tbytes.

(Somebody check my math?)

If you're talking a decade-long federal digitization initiative, we're looking at well south of 50 petabytes, which seems pretty doable in this day and age!



Can the rare books collections be digitized?
by autophile

Three closely related questions about the rare books collections at the Library of Congress:

1. I know there is some kind of effort going on to digitize the rare books collections, but can it be sped up? There are many high-quality low-cost archival book scanners out there (such as the ones developed at diybookscanner.org).

2. It gets really annoying to have to receive paper copies of books when copies are requested. Why not DVDs of high-quality images?

3. Why is there no outreach by the LoC to smaller, cheaper book scanning efforts? The Internet Archive, DIYBookscanner.org, and Decapod all come to mind.

CM: In reverse order. I don't know why we aren't distributing and decentralizing our scanning efforts. The Internet Archive is a heavy-duty production shop and they do an amazing job, as do folks like Google Books and the folks digitizing things the Mormon Church. But, there are a bunch of DIY solutions and it would be really nice if we could get more people pitching in. The biggest problem on distributing the digitization efforts is quality control. I know when it comes to ripping video, I can easily teach other people how to grab an MPEG2 off a DVD, but when it comes to things like digitizing a Betacam, that takes some training. But, we're all trainable and I wish we could all do more.

Getting back paper copies of books and papers when they're doing a copy anyway is just plain dumb. Likewise with things like FOIA results. John Podesta testified before the Senate about FOIA and said if an agency answers a FOIA request, they should also post their result online so others can see it. That seems pretty obvious.

As far as digitizing rare book collections, there are some amazing pockets throughout the government but there is no real coordination and there certainly is no effort to scan at scale or to come up with a realistic national digitization strategy. That is why we called on the White House to lead the effort. Within the Library of Congress there are some amazing collections, but if you look around to places like the National Agricultural Library or the National Library of Medicine or the libraries in the service academies you'll find lots more. Some have argued that digitizing rare books is silly because the audience is just a few academics, but I can tell you from my own experience helping host the network site for the Archimedes Palimpsest that when you make this kind of information available, there is an amazing long tail.

If you scan it, they will come. And, to answer your question, if we all scan it, they will come much sooner.



Real time legislation drafting
by kerskine

Would it be possible to implement a system that would allow real-time and continuous review of legislation while it's being drafted? Much has been made over the past three years about legislation being available for review before voting by the House or Senate. The final draft for review usually is huge PDF that makes it near impossible for citizens, interest groups, and the media to thoroughly analysis in time.

CM: You want to see the sausage being made not just buy the hot dog! I'll comment on the U.S. Congress since that's the system I know best. Thomas is a pretty good system if you happen to be stuck in 1994. It does have all the amendments and the actions and the various stages that legislation go through. But, it isn't real time, more like "pretty quick." As Van Jacobson once quipped, "Same day service in a nanosecond world." And, Thomas isn't really machine processable, it is final form, usually formatted ASCII text (shades of NROFF!). People like Josh Tauberer who built GovTrack.US have spent considerable time crawling those systems and trying to get the data into regularized formats and make it available to others to reuse via APIs, but that isn't the same as exposing the inner working of the sausage factory.

Majority Leader Cantor's staff has been pushing a system to make the raw data all available in XML from the Clerk's office and I think that is a very promising initiative which hopefully will bear fruit. (They're having a February 2 conference to discuss their plans if you are interested. I have no idea if it will be streamed for those of who aren't Inside the Beltway and I don't know their schedule for moving past conferences and into production.)

Congress is a pretty complicated beast. I know some folks like Sean McGrath have had better luck with some of the state legislatures. The problem is you need to dig deep into the inner working of a legislature. In the Congress, that means you're changing things like authoring tools that are used in the Clerk's office and by all the staff members, so you have to be careful or you get a bunch of really angry Congressman yelling at you because their staff can't crank out the flavor-of-the-week in the form of a bill or amendment.

There's also a bit of an issue of will. My work with the Congress to put hearings on-line showed that you could take the official transcripts of a hearing and use those to generate closed captions on the video. All you need is the official transcript of the hearing, but in order to get those I had to execute a special Memorandum of Understanding with the House Oversight Committee. Other committees guard their transcripts jealously and won't let them out for several when. When I started processing a bunch of historical videos we purchased from C-SPAN, I went to the Government Printing Office and found that many committees never deliver their transcripts, even a decade after the fact!



How to keep track of legislative activity about open access?
by oneiros27

Recently in the federal register, there were two calls for comments about access to data and research from federally funded research:

http://federalregister.gov/a/2011-28623 [federalregister.gov] http://federalregister.gov/a/2011-28621 [federalregister.gov]

I didn't hear about these until ~4 weeks after the original announcement, and with the holidays, it was too late to try to get the societies I'm involved with to prepare and vote on official statements. Are there any places where people can get/post notices of these sorts of things so that we can stay informed and try to help influence policies?

CM: The Federal Register is getting a lot better now that it is a much more open system. The idea of "Federal Register 2.0" was a paper I wrote for the Obama transition, so it is an issue I've tracked pretty closely and frankly, I've been amazed at how much better it is now. What they did is instead of selling the raw data feed for the Federal Register for $17,000/year, they went from SGML to XML and then released the data in bulk for free. A few guys out in San Francisco were looking for something to do to enter a contest and they took that bulk data and dreamed up GovPulse.US. That was such a better version of the Federal Register that the Office of the Federal Register switched the official site over to their open source platform. My point is the tools are there to do better notification mechanisms, and I'm sure the government would welcome somebody grabbing the GovPulse.US code out of Github and making it even better.

That's the technical answer. But, the substantive answer is that there is a huge boatload of stuff in the Federal Register and it is pretty hard to figure out what to pay attention to. I also missed that particular call for comment, and I've even missed several Requests for Information coming out of places I try and pay attention to, like the White House's Office of Science and Technology Policy. And, I do this stuff full-time! Perhaps better targeted notification mechanisms are the answer. Maybe it is a social media solution, where you pay attention to things your friends are paying attention to. I hope the answer is not that the only way to pay attention is to be employed with a beltway bandit which can afford hundreds of minions that do nothing but pay attention to Washington. Indeed, there are some very fancy for-pay services from folks like Congressional Quarterly and Bloomberg that cost an arm and a leg, but I can't help but think there has to be a better way that is also open.



What do you think of corporate partnerships?
by mhh5

I'd like to know what you think about corporate partnerships in the process to get public data released. (I'm not sure if Google Patents existed before the USPTO released its databases.) Do corporations that get involved in the process tend to make the process better without question, or are there tradeoffs in some areas because the corporations always want to help but then try to retain a proprietary version of the data for themselves?

CM: The theory is that the government gets some kind of valuable service (like digitization) that the government wouldn't get otherwise so it is a "win-win." But, the reality is all too often the government gets snookered and what we do is give some corporation exclusive access to some pot of data and the government doesn't get much of anything. The deal between Amazon and the National Archives was a good example of that kind of a private fence around the public domain. With a help from Boing Boing, I started systematically purchasing those public domain videos and re-releasing them in the wild. I have no problem with Amazon selling public domain video, I just hate it when they get a de facto or a contractual exclusive. (My testimony before Congress on this subject is here.)

There are lots of other examples of government getting snookered. For example, the Government Accountability Office let Thomson West get access to 60 million or so pages of federal legislative histories. At great cost to the government, they were all packed up and dispatched to West which digitized them all and then sent them back to the government. West now sells access to his amazing database. What did the government get for it's trouble? A few logins for GAO staffers. Even members of Congress need to pay to access the database! (We have an interesting paper trail on this issue.)

I'm glad you brought up the Google Patent system because I was personally involved in making that happen and I can tell you that this one is totally legit. Jon Orwant is the lead developer on this for Google and I played a small part in helping convince the White House and the Patent Office they ought to give Jon access to their data (the heavy lifting on that deal was by Beth Noveck who was the Deputy CTO at the time). Google makes all the data they got from the Patent Office available for bulk access with no strings attached. I can vouch for that because I did a mirror of their system. Last I heard Google was sending out anywhere from 1 to 10 terabytes of data PER DAY to external sources and even normally very critical folks who work in this arena have been really happy.

The big problem in the Patent Office is their computing infrastructure is a real catastrophe. Their power plant is over 95% capacity (e.g., plug in a computer, bring the building down!) and even though the Under Secretary knew that selling DVD subscriptions was silly, he wasn't able to switch over to an FTP service. He cut the deal with Google Patent and it worked out well for the government, for Google, and for everybody else.

What's the difference between the Google deal and the Amazon deal? In the case of the Amazon and GAO/West deals, the government lawyers did all the negotiating and they were totally outsmarted by some sharks in industry. But, when government has people like Under Secretary Kappos and Beth Noveck doing the negotiating, these things can work out just fine. The key is government should partner with people who want to do public service, not people who want to service the public.

Encouraging Governments?
by theNAM666

In a city such as Nashville, things as basic as business ownership and property records are not available online. In states such as New Jersey, public records such as basic corporate filings (officers, operating address/address for service of process) are accessible only for a fee.

What concrete actions can citizens confronting such situations, take to encourage accessibility and accountability?

CM: I find you need a carrot and a stick to make this stuff happen, especially at the local level. Folks like Everyblock.Com and CodeForAmerica.Org have done great working prying some of these databases loose, but there is still lots to do.

The first thing you should do is pick up the phone (or pick up your email client) and write/call the people who run the system. Ask them if you can have access to the data. Sometimes, it is as simple as that.

Other times, though, it isn't quite as simple since they want the money (or they want the control or they think this should be done by "private industry" by which they mean some buddy who is a contractor). The nice thing about any government system is somebody usually has oversight responsibilities. So, the next step is to find a city council member of state legislator who has oversight on the agency in question and ask them.

Again, life isn't usually that simple, but sometimes you win! If you can't get anywhere that way, what I usually end up doing is basically competing with the government system. Build a proxy system like RECAPtheLaw.Org did to recycle paid documents. Or, get a sponsor and buy a reasonable number of docs and build a web site that looks like it is going to be a real production system.

Then, go back again and ask. Maybe if you have eyeballs or at least have a nice web site, that is enough to get the government moving. But, if that doesn't do the job, you may have no choice but to compete with them for real, which of course requires a big commitment in time and energy and not everybody can do that. I know in the case of the Patent Office, I started pestering them in 1993, including several times when I spent 6-figure sums purchasing their data, and it still took until 2011 to crack that nut.

The real trick is focus/obsession. Pick one thing you really care about and just keep pestering them until you crack it open. If you're surfing from one opengov problem to another, showing up for a 1-day hackathon then moving on to something else, you're not going to get anywhere. Pick something real and make it your thing.



Privately Owned, Copyrighted Law
by AdamnSelene

I think I have read that the law itself cannot be copyrighted and it should be possible to make it available available to everyone. But as a techie who drafts standards and specifications, I was wondering about how far this goes--especially since Congress recently proposed enacting some of our standards into law. (They decided not to, but they read some parts into the committee records as they debated.) Can you still accomplish your project if a governmental body adopts (or considers adopting) a privately owned, copyrighted technical reference manual or set of safety standards as administrative law (or regulations that carry the force of law)? Or would such obstacles keep you from being able to digitize all of the government's laws (and archives of proposed laws)?

CM: The idea that the law has no copyright is a fundamental part of the American system of government. That applies to states and municipalities as well. The basic decision is Wheaton v. Peters from 1834 but that decision has been reaffirmed over and over. The law is sacred in the American system. You can't have equal protection under the law or due process under the law if there is a poll tax on access to justice.

When we get to a privately developed standards however, it turns into a very interesting issue. The basic mechanism is called Incorporation by Reference. The government will take some external document (such as a model building code) and incorporate the entire text to make it the law of the land. A guy named Peter Veeck was responsible for a landmark decision in 2002 when he published the Texas Building Code which was an incorporation of a privately-developed and very expensive model code. The court ruled that while the model code had copyright, the law of the land did not.

Based on the Veeck decision, my group went and posted many of the public safety codes enacted by the states. We started by purchasing model codes, finding the incorporating legislation, and concatenating the two pieces together and posting the resulting PDFs. More recently, we've done some extensive reworking of the California public safety codes, known as Title 24, converting the entire text into valid XHTML, recoding the graphics as SVG graphics, the formulas as MathML, and regenerating the PDF documents as nicely typeset documents instead of low-quality scans. You can see this work on the web but it is also available as Google Code project.

The federal government also uses this mechanism intensively, with over 2,000 standards incorporated into the Code of Federal Regulations. This is non-trivial stuff, things like all the OSHA safety regulations. The issue was recently considered by a federal group called the Administrative Conference of the U.S. which basically rolled over and endorsed the idea that it is ok for important parts of the law to cost money. (Read EFF's protest letter if you want a good critique of what they did.)

I'm not necessarily saying that government should be able to appropriate any privately-developed standard and make it available. And, I'm not necessarily saying you want OSHA bureaucrats drafting the standards. But, I do think the big standards establishment and the government regulators have cut a deal that results in the law not being available and the costs forked off on private citizens and small business with extortionate monopoly prices. I just paid $847 for a 48-page safety standard from Underwriters Labs and $60 for 2-page safety standard from the Society of Automotive Engineers, both of which are mandated by law in the CFR. They do need money to run their operations, but let me just point out that in 2009 the 501(c)(3) nonprofit Underwriters Labs paid their CEO $2,138,984 and the nonprofit SAE paid their CEO $412,578.

Ancestry.com
by An Anonymous Reader

What is your opinion about websites like Ancestry.com which make use of public records and charge a subscription fee for access? What is the incentive for the government to migrate old documents into digital form when services like these exist? Do you think Ancestry.com should be a 501(c)(3)?

CM: I'm not a big fan of for-profit corporations that have a business model of monetizing the public domain. I'm fine if they exist and fine if they make billions of dollars, but if they are the only game in town they've taken something that belongs to all of us and and turned it into their private property.

The government got snookered on the Ancestry.Com deal. They could have insisted that the raw data be available in bulk for anybody else to use. The folks that approach the government to cut these sweetheart deals argue that is unreasonable because they need a "return on investment" and the argue that if they don't get the return on investment they won't do the deal (and by extension nobody else will do the deal).

But, government can argue much harder! For example, instead of negotiating some exclusive thing with Ancestry.Com, how come they didn't ask the Internet Archive to grab the data? Or put together something creative with a couple of foundations that would pay for the digitization in return for the kind of payback the foundations like to see (e.g., good press, photo opportunity with the President, or other tools of the trade)?

You asked if Ancestry.Com should be a 501(c)(3)? Not all nonprofits do something that I think which should be an essential part of their mission, which is allow others to compete with them. I believe providing open access to all data ought to be a precondition to getting nonprofit status (an idea that Gil Elbaz has been pushing for quite some time). A good example of a nonprofit that builds walls is Guidestar which wants to be the place where you go for all your nonprofit information. The IRS should be making all Form 990 returns of nonprofits available in bulk for anybody to use, which would knock the bottom out of Guidestar's attempts to build walls and force them to stay innovative and provide value.

Pacer Problems
by onyxruby

How much difficulty do you anticipate in getting and publishing records in Pacer? If there's one system that should be free it the decisions that our courts make and yet you are charged by the page just to view the results. Are you concerned about a court taking an unkind view on your archiving what is in Pacer?

CM: PACER is an abomination. Do they take a dim view of our efforts? Well, the Administrative Office of the U.S. Courts reacted so strongly to our efforts to make their data available that they called the FBI on Aaron Swartz and cancelled the only meaningful public access system they had, which consisted of one terminal in each of 17 public libraries around the country. In this era of rapidly decreasing costs, they just boosted their access charges from 8 cents a page to 10 cents a page, arguing that this is a bargain compared to 25 cents a page for a copy machine.

What I find so disturbing about PACER is that when we did get 20 million pages of docs, we were able to conduct a comprehensive analysis of privacy violations in the courts, an analysis that led to a nice thank-you letter from the Judicial Conference and changes in their privacy rules. In other words, only when public interest groups got access to the data did we begin to address privacy issues. Public access is not just about pro se prisoners defending themselves from a jail cell, which is the view of many in the Administrative Office of the Courts. Public access is about attempts like ours (and many other folks) to make our system of justice function better. When we say we are "an empire of laws not a nation of men" that means we write down what we are doing in our courts so that it is no longer the arbitrary decisions of individuals. The paper trail is there so we can make sure the system is functioning properly. When you limit that access to those that only have a Gold Card, you pervert democracy and you pervert justice.

This principle that access to justice shouldn't hide behind a cash register goes back to the Greeks. Theseus in Euripedes' Suppliants said "when there are no public laws, one man holds power by keeping the law all for himself, and there is no more equality. But when the laws are written, the weak man and the rich man have equal justice." The PACER system is justice for the rich man.

Steve Schultze and the team at Princeton did a lot of the heavy lifting on this issue, including the very nice RECAPtheLaw.Org system they built. They've also done a lot of financial analysis that shows that the courts are not only recovering their costs for operating the expensive PACER system, they're making a huge profit (to the tune of $100 million/year) and using their excess profits to do things like buy big-screen TVs in direct violation of the E-Government act.

The basic problem on PACER is the Judicial Conference has delegated the issue to a few techie judges who think what they've built is something great. But, PACER is a hairball of bad PERL code and the result has not served the judges, the bar, or the American people very well. My only hope is that eventually, the Judicial Conference will see that their information technology is 30 years behind the rest of the Internet and feel ashamed at the travesty they have wrought. Until then, we have RECAP.

If you're interested in the issue, a couple of resources to look at are the PACER paper trail and a bit of a rant that I delivered at the Gov 2.0 summit.



How to visualize opened data?
by hardwarejunkie9

The amount of information you're trying to free is entirely staggering and consists, largely, of tables of numbers. These numbers are incredibly significant, but people generally can't see them.

After you free all of this information and make it available to the public (as it should be), then what? What do you expect for the public to do with these numbers? Tables of information are not nearly as useful as graphs. This data needs to be seen, but, more importantly, it needs to be understood.

Do you have any ideas for how to disseminate this information? Perhaps a team-up with someone like gapminder.org's Hans Rosling might be particularly valuable for all of us.

CM: Actually, most of the data I'm looking at is not tables of numbers, it is video, images, textual documents, technical papers, maps, and books.

But, I definitely get what you're saying and there are a lot of numbers. For example, the IRS Form 990s should be structured data instead of PDF documents, so extracting the data from the mass of paper is the initial challenge. There are lots of other examples of this kind of initial extraction, getting what were printed paper docs into structured data. There are some interesting tools, such as OCRopus which does layout analysis, but there needs to be much more. One of the reason we called for a Federal Scanning Commission is that we think there is a lot of directed R&D that could not only scale up mass digitization but could also work on the important value-added of extraction of structured data and handling some of the tricky issues like detecting the presence of Social Security Numbers.

Once you have the data, as you say, then what? I'm a big fan of the idea that the government starts by providing bulk data, then they provide an API, and then maybe they also build web sites and apps and other things along with everybody else out there. That's a 3-part hierarchy that Ed Felten and some of his students developed and it should be a law that applies to all government information systems that are externally facing.

The issue here is that all too often people look at a problem like "digitize all government information" and they want to see the whole stack of the solution from one place. But, I think you can do a layered approach and count on the fact that there is always somebody smarter out there and our job is to reduce the barriers to entry. So, how would I visualize the data? I have no idea, but I'd make damned sure that folks like Martin Wattenberg at Many Eyes and Hans Rosling at Gapminder knew the data was out there and then I'd sit back and be amazed at whatever they come up with. How's that for pushing the problem downstream?



Why is data access so hard?
by CanHasDIY

Can you provide any explanation as to why it is so difficult and cost-prohibitive to obtain records from the government, especially considering the abundance of laws requiring government compliance with requests for information (AKA "Sunshine Laws")?

Is it simply a matter of government employee ineptitude, or have you found evidence of a more nefarious rationale?

CM: I get that question a lot. Why would a member of Congress take deliberate steps to stop public hearings from being available? Why would a court administrator deliberately restrict access to public court documents? Usually the answer is, as Heinlein said, "you have attributed conditions to villainy that simply result from stupidity." When I'm explaining why something is so broken on a big government system, my usual answer is that there are a lot of people still stuck in the 1970s and 1980s, when information dissemination was really, really hard and it took men in white lab coats and computers the size of freight trains to process data. In other words, the problem with a lot of folks who are government gatekeepers is they just don't get the Internet and they don't get computers. In fact, usually when some senior bureaucrat is throwing stones at me, you can find younger staffers working for them rolling their eyes.

That's an optimistic view, and if I'm right things will get better. But, I'm often wrong on my predictions of the future. (I was the guy who saw TimBL demo the web in 1992 and thought to myself "interesting, but it won't scale.")

But, there is also some more nefarious stuff happening, often the accumulation of power by being able to cut exclusive deals with contractor buddies. If your life in government consists of receiving emissaries from Lockheed Martin, maybe you think you're making everybody happy by letting them build you a $1 billion computer system. Often, you think your problems are so unique that the $1 billion solution is the only answer.

And, in some cases, as we've seen from numerous GAO reports, Inspector General reports, Congressional hearings, and newspaper articles, there are some really evil people out there who think the public domain and the government is their personal business opportunity. Looting the federal government is the kind of civic crime that ranks right up there in my book with stealing cookies from Girl Scouts and selling fake medicines to sick people.



Who is the worst?
by TheBrez

Which government agency is the worst to get information from?

CM: I don't know who the worst are (there's a lot of competition for that slot), but the ones that piss me off the most are the ones that should know better.

Public.Resource.Org is a really small operation. I'm the only staff member. My part-time sysadmin is @mdkail who is pretty busy with his day job as CIO at NetFlix. My ISP is Jim Martin and his team at ISC who are kind of busy running the F-Root. My office net is supported by the amazing systems team at O'Reilly which rents me office space at below-market rates.

I'll grant you government would have a tough time getting that kind of help. But, I'm a one-man shop and we run the 4th most popular U.S. government video channel on YouTube, we're the source for a lot of the on-line presence of the U.S. Court of Appeals, and we've supported efforts for the U.S. Congress, the White House, and the National Archives. If we can do this out of Northern California, couldn't the vast resources of the federal government in Washington, D.C. do a whole lot better than they're doing now?

For me, my current bete noir is the U.S. Congress. We got half-way through processing their archives of video from congressional hearings, publishing about 31 terabytes of data. Then, a couple of staffers decided this was a bad idea and pulled the rug out from under us. They actually decided it was a bad idea to publish video from public congressional hearings.

Like any agency, Congress is a mixed bag. We had tons of support from Darrell Issa, for example, and ran a very successful pilot project for him for a year. We talked to all sorts of people on committees and in the various agencies that support the Congress. But, at the end of the day, a couple of staff members were able to decide that the public archive shouldn't be public and they terminated our project. (If you have some time, you might like to read our rather surreal paper trail.)

So, rather than the worst, I think we need to look for the most shameful, the ones that have the privilege and the power and could easily do better. I know it is in vogue to throw stones at government in general and Washington in particular, but there are times when government can be so useful and so awe inspiring it takes your breath away. Government can be that shining city on the hill but we all have to take an active part in our government to keep those lights shining bright.

joabj writes "The ISC is seeking some open source magic for the next version of the widely used BIND. Although the BIND is already open source, most of the work thus far done on the DNS server software has come from contractors, the government and Unix vendors. 'The goal is to move away from having BIND a heavily sponsored corporate product,' said BIND 10 manager Shane Kerr. Kerr is hoping that more eyes will equal fewer bugs, and that more users will go ahead and implement the features they've been requesting themselves. BIND 10, due by the end of the year, features a new modular architecture, one designed to circumvent many of the security woes that have bedeviled BIND 9."

Morty writes "BIND, the popular DNS server software, has been crashing all over the Internet. The root cause is believed to be a 0-day vulnerability in BIND's resolver. The ISC has issued an alert. Quoting: 'An as-yet unidentified network event caused BIND 9 resolvers to cache an invalid record, subsequent queries for which could crash the resolvers with an assertion failure. ISC is working on determining the ultimate cause by which a record with this particular inconsistency is cached. At this time we are making available a patch which makes named recover gracefully from the inconsistency, preventing the abnormal exit.'"

Morty writes "BIND, the popular DNS server software, has been crashing all over the Internet. The root cause is believed to be a 0-day vulnerability in BIND's resolver. The ISC has issued an alert. Quoting: 'An as-yet unidentified network event caused BIND 9 resolvers to cache an invalid record, subsequent queries for which could crash the resolvers with an assertion failure. ISC is working on determining the ultimate cause by which a record with this particular inconsistency is cached. At this time we are making available a patch which makes named recover gracefully from the inconsistency, preventing the abnormal exit.'"

epee1221 writes "Several security professionals released a paper raising objections to the DNS filtering(PDF) mandated by the proposed PROTECT-IP Act. The measure allows courts to require Internet service providers to redirect or block queries for a domain deemed to be infringing on IP laws. ISPs will not be able to improve DNS security using DNSSEC, a system for cryptographically signing DNS records to ensure their authenticity, as the sort of manipulation mandated by PROTECT-IP is the type of interference DNSSEC is meant to prevent. The paper notes that a DNS server which has been compromised by a cracker would be indistinguishable from one operating under a court order to alter its DNS responses. The measure also points to a possible fragmenting of the DNS system, effectively making domain names non-universal, and the DNS manipulation may lead to collateral damage (i.e. filtering an infringing domain may block access to non-infringing content). It is also pointed out that DNS filtering does not actually keep determined users from accessing content, as they can still access non-filtered DNS servers or directly enter the blocked site's IP address if it is known. A statement by the MPAA disputes these claims, arguing that typical users lack the expertise to select a different DNS server and that the Internet must not be allowed to 'decay into a lawless Wild West.' Paul Vixie, a coauthor of the paper, elaborates in his blog."

penciling_in writes "ISC has made the announcement that they have developed a technology that will allow 'cooperating good guys' to provide and consume reputation information about domain names. The release of the technology, called Response Policy Zones (DNS RPZ), was announced at DEFCON. Paul Vixie explains: 'Every day lots of new names are added to the global DNS, and most of them belong to scammers, spammers, e-criminals, and speculators. The DNS industry has a lot of highly capable and competitive registrars and registries who have made it possible to reserve or create a new name in just seconds, and to create millions of them per day. ... If your recursive DNS server has a policy rule which forbids certain domain names from being resolvable, then they will not resolve. And, it's possible to either create and maintain these rules locally, or, import them from a reputation provider. ISC is not in the business of identifying good domains or bad domains. We will not be publishing any reputation data. But, we do publish technical information about protocols and formats, and we do publish source code. So our role in DNS RPZ will be to define 'the spec' whereby cooperating producers and consumers can exchange reputation data, and to publish a version of BIND that can subscribe to such reputation data feeds. This means we will create a market for DNS reputation but we will not participate directly in that market.'"

r00tyroot writes with news that slipped by yesterday, quoting from the Internet Systems Consortium's release: "ISC joined other key participants of the Internet technical community in celebrating the achievement of a significant milestone for the Domain Name System today as the root zone was digitally signed for the first time. This marked the deployment of the DNS Security Extensions (DNSSEC) at the top level of the DNS hierarchy and ushers the way forward for further roll-out of DNSSEC in the top level domains and DNS Service Providers."

Ethanol writes "Internet Systems Consortium, producers of BIND 9 (the most popular DNS implementation on the internet), have spent the past year working on a successor, BIND 10. It's entirely new code, redesigned and rewritten from the ground up, and now the first glimpse of what it will eventually look like has been released. 'This code is not intended for general use, and is known to be inefficient, difficult to work with, and riddled with bugs. These problems will all be fixed over the next couple of years, as functionality is added and refined, and the software matures. However, the codebase has a good framework for moving forward, and the software is capable of serving as a DNS server with significant functionality.' (Full disclosure: I work for ISC and I'm one of the engineers on the project.)"

Ethanol writes "Internet Systems Consortium, producers of BIND 9 (the most popular DNS implementation on the internet), have spent the past year working on a successor, BIND 10. It's entirely new code, redesigned and rewritten from the ground up, and now the first glimpse of what it will eventually look like has been released. 'This code is not intended for general use, and is known to be inefficient, difficult to work with, and riddled with bugs. These problems will all be fixed over the next couple of years, as functionality is added and refined, and the software matures. However, the codebase has a good framework for moving forward, and the software is capable of serving as a DNS server with significant functionality.' (Full disclosure: I work for ISC and I'm one of the engineers on the project.)"

Ethanol writes "Internet Systems Consortium, producers of BIND 9 (the most popular DNS implementation on the internet), have spent the past year working on a successor, BIND 10. It's entirely new code, redesigned and rewritten from the ground up, and now the first glimpse of what it will eventually look like has been released. 'This code is not intended for general use, and is known to be inefficient, difficult to work with, and riddled with bugs. These problems will all be fixed over the next couple of years, as functionality is added and refined, and the software matures. However, the codebase has a good framework for moving forward, and the software is capable of serving as a DNS server with significant functionality.' (Full disclosure: I work for ISC and I'm one of the engineers on the project.)"

Ethanol writes "Internet Systems Consortium, producers of BIND 9 (the most popular DNS implementation on the internet), have spent the past year working on a successor, BIND 10. It's entirely new code, redesigned and rewritten from the ground up, and now the first glimpse of what it will eventually look like has been released. 'This code is not intended for general use, and is known to be inefficient, difficult to work with, and riddled with bugs. These problems will all be fixed over the next couple of years, as functionality is added and refined, and the software matures. However, the codebase has a good framework for moving forward, and the software is capable of serving as a DNS server with significant functionality.' (Full disclosure: I work for ISC and I'm one of the engineers on the project.)"

Icemaann writes "ISC is reporting that a new, remotely exploitable vulnerability has been found in all versions of BIND 9. A specially crafted dynamic update packet will make BIND die with an assertion error. There is an exploit in the wild and there are no access control workarounds. Red Hat claims that the exploit does not affect BIND servers that do not allow dynamic updates, but the ISC post refutes that. This is a high-priority vulnerability and DNS operators will want to upgrade BIND to the latest patch level."

sawky puck writes "Researchers at Secunia have flagged a 'highly critical' vulnerability in Samba, the widely deployed open-source software for networked file sharing and printing. Successful exploitation allows execution of arbitrary code by tricking a user into connecting to a malicious server (e.g. by clicking an 'smb://' link) or by sending specially crafted packets to an 'nmbd' server configured as a local or domain master browser. This issue affects both Samba client and server installations."

Slashdot contributor Bennett Haselton writes "A recurring theme in editorials about Net Neutrality -- broadly defined as the principle that ISPs may not block or degrade access to sites based on their content or ownership (with exceptions for clearly delineated services like parental controls) -- is that it is a "solution in search of a problem", that ISPs in the free world have never actually blocked legal content on purpose. True, the movement is mostly motivated by statements by some ISPs about what they might do in the future, such as slow down customers' access to sites if the sites haven't paid a fast-lane "toll". But there was also an oft-forgotten episode in 2000 when it was revealed that two backbone providers, AboveNet and TeleGlobe, had been blocking users' access to certain Web sites for over a year -- not due to a configuration error, but by the choice of management within those companies. Maybe I'm biased, since one of the Web sites being blocked was mine. But I think this incident is more relevant than ever now -- not just because it shows that prolonged violations of Net Neutrality can happen, but because some of the people who organized or supported AboveNet's Web filtering, are people in fairly influential positions today, including the head of the Internet Systems Consortium, the head of the IRTF's Anti-Spam Research Group, and the operator of Spamhaus. Which begs the question: If they really believe that backbone companies have the right to silently block Web sites, are some of them headed for a rift with Net Neutrality supporters?" Read on for the rest of his story.

In the aforementioned instance, AboveNet and TeleGlobe were not selling "parental filters" or other common types of filtered Internet access; the users being blocked from our Web sites were adults paying for what they thought were unfiltered Internet connections. What had happened was that AboveNet and TeleGlobe signed up to block Web sites on the Realtime Blackhole List, a list which was widely (but inaccurately) thought to be a list of "spammers", put out by a group called the Mail Abuse Prevention System. (MAPS and the RBL still exist, but under new management and in a form that bears little resemblance to their late-90's forerunners.) Most ISPs that used the RBL used it to filter only incoming e-mail, but AboveNet went all-out and blocked users from even viewing RBL'ed web sites, presumably because two of MAPS's founders, Paul Vixie and Dave Rand, were on the AboveNet board of directors. And it turned out that the RBL not only included spammers, but also Web sites that were not sending mail at all but were blocked because of their content -- in our case, our ISP got blocked because some other customers were selling mailing list software that MAPS believed could be too easily abused by spammers.

These two distinctions -- (1) the distinction between blocking incoming e-mail from spammers, versus blocking Web sites; and (2) the distinction between blocking traffic due to spam activity, versus blocking sites because of their content -- both go to the heart of what Net Neutrality is, and isn't, about. Net Neutrality is about user preferences -- not meaning that as a buzzword, but as an actual guiding principle to figure out what is and is not covered by the cause. If an ISP filters incoming mail from known spammers, that generally improves the user experience, and is something many users would expect an ISP to do anyway. But if an ISP blocks users from reaching Web sites (even, for the sake of argument, the Web sites of actual spammers), then that's generally counteracting the user's wishes -- if the user didn't want to go there, they wouldn't have typed it in. (After all, I visit spammers' Web sites all the time, usually right before I sue them.) Similarly, if an ISP blocks traffic from sites because of spam or other network abuse, that serves to protect their own users. But if an ISP blocks users from viewing sites because of their content, that's generally not expected by users, unless they've specifically signed up for something like parental controls. The Snowe Net Neutrality amendment proposed last year recognized both of these distinctions, and stated that nothing in the amendment would be interpreted to prohibit spam filtering, parental control services, or measures to protect network security.

The MAPS incident thus shaped most of my opinions about Net Neutrality 6 years before the debate even had a name. When I first found out in August 2000 that our ISP was blacklisted, like most people I believed that the RBL really was a list of spammers; after all the MAPS web page said that the RBL was a list of networks that "originate or relay spam". So I called my ISP screaming at them for being incompetent spam-enablers (the culmination of many frustrating issues with them), and saying that if they really were letting customers send spam, or running an insecure server that spammers were hijacking, I would leave on principle, if the cretins managing our server didn't drop it in the lake first. The ISP owner then told me what happened: that the ISP was not blacklisted for spamming customers, but because of the content of the other sites. (Buried in the list of RBL criteria on MAPS's site was the statement that sites could be blacklisted for providing "spam software", although the criteria did not define how they distinguished between spam software and regular mailing list software, which is how our ISP got caught in the net. And the criteria did not disclose anywhere the most controversial feature of the RBL, which is that if an ISP didn't comply, MAPS would start blacklisting other unrelated sites at the same ISP to put more pressure on them.) I agreed that this seemed to be absurd, and said I wouldn't leave the ISP if they were being blackballed just because of the content of hosted pages.

I don't know exactly what the mail software in question did or where MAPS thought the line should be drawn, but I am a purist about content -- it's a long-standing principle among the Internet security community that if a tool exists which exploits a security hole, you don't try to make the software disappear, you fix the hole. And besides, since MAPS and their supporters wanted to blackball ISPs that hosted spamming software (however you defined that), but the same people had never advocated blackballing ISPs that hosted network break-in tools and other cracking programs, for example, then what were they really saying? That spamming someone more unethical than breaking into their network?

But by far the most common objection to my complaint about AboveNet blocking Web sites was, "Hey, if a private company blocks things, as long as they're being honest to their users about it, who cares?" Well, true, but the fact that AboveNet blocked Web sites was not widely known even within the company; when I once called AboveNet feigning ignorance and asking them if they blocked RBL'ed Web sites, the technician who spoke to me said, "No, that wouldn't make any sense." (Well, half right.) Their AUP mentioned "protecting users from spam" but said nothing about blocking Web sites. In fact, other than "family-filtered" ISPs and similar services, I've never heard of any company blocking Web sites that actually did try to make their users aware of it. (On the other hand, even if AboveNet had fully disclosed their filtering, they were still a backbone company selling connectivity mainly to ISPs -- and I think if you sell something wholesale that can only be re-sold to the public by fraudulent means, then you're at least partly complicit in that fraud as well.)

If you're tempted to argue that backbone providers should be allowed to block whatever they want as long as they bury it in their AUP (although AboveNet and TeleGlobe didn't even do that much), just consider: When you access Google from your home computer, have you read the AUP of every network that the packets pass through, to check whether they reserve the right to block or even modify your traffic? Without doing a traceroute, could you even name all the networks that the traffic passes through? Do you really want the burden to be on you to check with all of them every time there's a problem reaching a Web site? Or do you feel like there's an understanding that as long as you pay your bill, they should let you go wherever you want?

Some have argued that if an ISP blocks the user from reaching a Web site, then even if the ISP is defrauding the user, that's still strictly an issue between the user and the ISP. But if a user is trying to reach your Web site, the user is trying to give you something of value: their attention, their eyeballs on your advertisements, sometimes even their money (with the expectation that you will provide them with something in return, of course, like some content worth reading). If the ISP steps in and blocks that, then the ISP has taken something of value that the user was attempting to give to you, and diverted it to serve their own interests. To me that doesn't seem ethically much different from the FedEx driver swiping the chocolates that someone tried to send you for Valentine's Day. Is that just between the sender and FedEx? Or do you have a beef because you didn't get the present that was intended for you, and you had to eat last week's chocolates to cheer up?

The modern-day threats to Net Neutrality are different: slowing access to Web sites unless the site owners pay a "toll", instead of blocking access to sites because of the content of other sites hosted at the same ISP. But they both boil down to the same thing: not giving end users what they have already paid for. If a user buys Internet access, they almost always buy it with the understanding that if they access a site, the content will download as quickly as their connection allows.

Thus the most common misconception about Net Neutrality is that the proponents are fighting against "capitalism" -- ISPs just charging more for different delivery speeds. But ISPs are already charging users for those delivery lines -- including different tiers for different prices. That's capitalism, and it works, with prices falling all the time in a fairly competitive market. But charging publishers for those higher delivery speeds to the user's house, is really more like double-billing, because the user has already been charged once for the lines that the content is coming over, so the ISP is trying to charge the content publisher again for the same service. Of course, if you charge party A for doing X, and then you try to charge party B for the same instance of doing X, and party B doesn't pay up so you don't do X, you're also breaking your deal with A. Brad Templeton of the EFF stated as much on his blog in 2006:

The pipes start off belonging to the ISPs but they sell them to their customers. The customers are buying their line to the middle, where they meet the line from the other user or site they want to talk to. The problem is generated because the carriers all price the lines at lower than they might have to charge if they were all fully saturated, since most users only make limited, partial use of the lines. When new apps increase the amount a typical user needs, it alters the economics of the ISP. They could deal with that by raising prices and really delivering the service they only pretend to sell, or by charging the other end, and breaking the cost contract. They've rattled sabres about doing the latter.

And I think the same is clearly true if, instead of trying to extract money from the content publisher, the ISP tries to extract something else, like an agreement to shut down certain Web sites before the ISP will let their users view other sites hosted at the same company. You can talk all day about how evil those Web sites are, but the ISP has already sold the user a connection with the implied ability to access them.

Anyway, this all came out in 2000 when a Slashdot article revealed that AboveNet had been blocking Web sites, and AboveNet stopped doing it two hours after the article came out. (TeleGlobe stuck with it for a few more months.) But from the hostility of the reaction, you'd think that we had published cartoons in a Danish newspaper showing Paul Vixie with a bomb in his turban. I got more e-mails than I could count arguing that AboveNet had the right to block whatever Web sites they felt like, regardless of whether the end users knew it was happening. To those people, I'd be sincerely interested in their answer to this question: Does that mean they've have no problem if they found out their ISP was silently blocking sites for political reasons? There is a clear line between following user preferences by blocking spam, and countermanding user preferences by blocking sites because of their content -- and once you've crossed that line, where's the logical stopping point? Seriously, I would have liked to have known how they would answer that, if I could have gotten any meaningful dialog going with them, which most of the time I couldn't. At the time, I'd just spent four years telling people that kids looking at porn was a non-issue, and that by the way if their kids came to my Web site I'd even help them get around their blocking software, and I still got more angry e-mails for disclosing the fact that AboveNet blocked Web sites based on their content, than I'd gotten in all the previous four years combined. (A few even accused us of moving into a blacklisted address block on purpose. This was because the actual move happened after the blacklisting was in place, even though I told them all that our ISP had announced the coming move two months before -- repeat, before -- they ever heard from MAPS. Some people were so in love with that "smoking gun" that they didn't believe me; that's their prerogative. But don't take my word for it -- when one supporter wrote to MAPS to ask about un-blocking our site, MAPS officer Kelly Thompson replied:

>Would it be possible to
>selectively unblock peacefire.org (209.211.253.169)?
Technically? Yes, it is. It's a violation of our policy, though, so I can't do so.

I would be willing to help you find other free or reduced cost hosting, however.

It was MAPS's decision, not ours or our ISP's, to have our site blocked. That should settle that once and for all, just as soon as there is peace in the Middle East and a black lesbian in the White House.)

But what do all these people think about Net Neutrality, 6 years later? I tried to track down the influential people who had spoken out supporting AboveNet's blocking of Web sites, or at least their right to block Web sites. My position was, we can agree to disagree on that, but if they really feel that way, why haven't they been speaking out against Net Neutrality? The proposed Snowe amendment was pretty clear:

SEC. 12. INTERNET NEUTRALITY
(a) Duty of Broadband Service Providers- With respect to any broadband service offered to the public, each broadband service provider shall--
(1) not block, interfere with, discriminate against, impair, or degrade the ability of any person to use a broadband service to access, use, send, post, receive, or offer any lawful content, application, or service made available via the Internet.

John Levine, webmaster of Abuse.Net, head of the IRTF's Anti-Spam Research Group, and one of the most vocal critics of Peacefire's campaign against AboveNet's Web filtering, said that he would have opposed the bill but didn't bother because it didn't have much chance of passing. Well, it didn't, but the bill was significant not because of its likelihood of passage, but because it articulated the principles that the Net Neutrality coalition had rallied around, and with the momentum behind the movement, it's likely to achieve at least some of its goals, by legislation or otherwise.

Paul Vixie, Dave Rand, and Steve Linford did not respond to requests for comment on Net Neutrality. But Paul Vixie wrote something very interesting in a May 2006 blog post:

Second, there's network neutrality. In telephone service, the government mandates that all companies providing voice-grade telephony interconnect with eachother at preset rates, thus ensuring that any phone can call any other phone and that new phone companies can enter the field to help ensure competition. In Internet service, the government mandates nothing. Recently SBC (I mean AT&T, I think, is it Wednesday?) rattled its sabre and said that Google and other content supplying companies should be paying for the use of SBC's backbone to reach SBC's eyeballs. Most of us said, uh, what? "Aren't SBC's own customers paying SBC to carry that traffic?" Some of us even said "I am not an eyeball, I am a person!" But anyway, from time to time these Internet companies shut down interconnects in hopes of creating new cash flows among eachother, and until the government regulates this, we're all at risk of higher prices or lower service with zero notice. Some well meaning democrats are trying to challenge this with "network neutrality" legislation, but this probably isn't their year. Or their decade.

San Francisco has a government, though. And if San Francisco owned and operated its own wireless Internet plant, we could mandate that any Internet company wishing to do business in this city interconnect at fair and reasonable cost to all other Internet companies wishing to do business in this city.

"Until the government regulates this"? "Government mandates"? "Fair and reasonable cost"? Quick, call the anti-socialist intervention squad! How long does it take those San Francisco hippies to suck the new arrivals' brains out anyway? Of course, I agree with everything he said. It's just that if you replace "create new cash flows" with "try to get ISPs to remove content from their servers", this describes exactly what Vixie and AboveNet were doing a few years earlier. He's a smart guy, and I'm sure this didn't escape his sense of irony, so perhaps this confirms something I'd suspected all along, which is that Vixie understood the subtleties of the issue better than most of his cheerleaders, and may be having second thoughts about AboveNet's Web-blocking misadventure. From the beginning, in a 1997 interview with Sun World, he sounded like someone trying to at least keep an open mind:

Concentration of power into a single individual: It's very true that power has corrupted every individual in whom it has ever been concentrated in the history of mankind. I do not feel that I am necessarily above whatever elements of human nature give rise to that. I worry about it. Probably other people worry about it more than I do.

Although, he didn't get to making any such frank statements during the controversy over AboveNet's Web site blocking. (Perhaps MAPS's lawyers were worried that he was a little too unfiltered and advised him not to comment; at the time, the MAPS Web site had a "How to sue MAPS" link on the front page.)

Speaking of which, Anne Mitchell, Director of Legal and Public Affairs for MAPS during the time when AboveNet was blocking Web sites, was the only MAPS adherent from the era that I could find who has since clearly and publicly come out against Net Neutrality. In May 2006 she wrote:

Here's the thing that the 3Ns (Net Neutrality Nuts) don't get: bandwidth costs money. And if you can't charge those who use the majority of it accordingly, then you are going to have to amortize it across everybody.

So, if a net neutrality law passes, don't be surprised when your costs to have an Internet account skyrocket.

Because somebody has to pay those bills, and if the law says that the ISPs can't charge the big guys - the big users - differently, it means that they have to charge them the same rate that they charge everyone else. And that means not that their rate will go down, but that everybody else's rate will go up.

And then again in February 2007 in another blog post titled "Towards A Nanny Internet", she wrote, "Network neutrality is the idea that ISPs should be forced to charge everybody the same for their Internet use", grouping it together with proposed anti-bullying and anti-anonymity laws.

Well, points to Anne for being consistent, and for publicly declaring her views in no uncertain terms, which is all I'm asking of the other supporters of AboveNet's website blocking policy. (Although she's coming at it from a different angle this time, "How do we work out who pays for the traffic" rather than "ISPs should be allowed to block whatever they want without telling anybody".) But this is also a textbook example of what I think are the three major fallacies of opposition to Net Neutrality:

First, lumping it together with other examples of unpopular regulation and calling it one more example of Big Government -- an argument also tried in other editorials ("Politicians and public figures alike should realize the absurdity of advocating more red tape to keep the Internet free"). This meme has never really caught on, possibly because groups like the ACLU and the EFF that have traditionally opposed true Internet censorship, have lined up in favor of Net Neutrality. All the proposed "red tape" and "regulation" really says is that if a user attempts to access a Web site over a connection that they've paid for, the ISP may not block or slow down their access, a law which most people would hardly consider tyrannical.

Second, asserting that "Network neutrality is the idea that ISPs should be forced to charge everybody the same for their Internet use." I've never actually heard anyone advocate anything close to that, but a common question among skeptics is why different "tiers" for Internet traffic are really any different from different-tiered pricing for dial-up vs. DSL, or for different levels of Web hosting. The difference is that when users and Web site owners pay for those connections, they are paying for their respective connections to the rest of the Internet. But an ISP charging a Web site owner to carry their traffic the last mile to the user's house, is not charging for a product or service, but really charging a fee not to break a service that they've already agreed to provide to the user.

Which leads to the third misconception: "Here's the thing that the 3Ns (Net Neutrality Nuts) don't get: bandwidth costs money... So, if a net neutrality law passes, don't be surprised when your costs to have an Internet account skyrocket." But it's not about how much a service costs, but about the ethics of double-billing for it. We know that ISP pricing models can already support the total traffic that people consume today, and ISPs do already follow net neutrality principles most of the time, so nobody's costs will "skyrocket" just because a neutrality law passes. If vastly more people start trying to stream CNN over the Internet 24/7, and fully using the services that ISPs have "only been pretending to sell" as Brad Templeton put it, then ISPs may have to charge more for users who consume too much bandwidth, encouraging people to stay at today's average levels by rationing themselves and perhaps watching 24 on their $5,000 TV sets sometimes instead of downloading it off of BitTorrent to their laptop every week because it makes them feel like a haX0r. Much as we all love our unmetered connections, it wouldn't be a violation of Net Neutrality for ISPs to charge users for bandwidth hogging, to keep everyone from going too far above today's levels. What ISPs should not do is charge users for implied full-throttle connections, and then turn around to charge publishers for moving bits over those same lines, or block the connection for any other reason.

So, yes, Virginia, blocking of Web sites does happen -- and by "Virginia", I mean FTC Chairman Deborah Platt Majoras, who said in a speech in August 2006: "I have to say, thus far, proponents of net neutrality regulation have not come to us to explain where the market is failing or what anticompetitive conduct we should challenge; we are open to hearing from them." This was echoed in an editorial later that month from Sonia Arrison of the Pacific Research Institute:

Internet service providers have voluntarily upheld content-neutral practices without the need for government intervention, and consumers would never stand for blocked Web sites... If the loss of net neutrality principles was really a problem, advocates wouldn't need to scare Americans in order to win their support. Using government regulation preemptively to shortchange business partners is a reckless abuse of the public policy process. New laws should be based on facts and reality, not fear and hypothetical situations.

I guess both of those ladies' ISPs must be blocking access to the SaveTheInternet.com Web site, so I e-mailed both of them the coalition's list of examples, and added a note about the AboveNet/TeleGlobe incident as well. No personal response from either of them yet, but I'm sure they just got lost in the shuffle while they were so busy sending out corrections. (On the other hand, I did get a courteous response from Randolph J. May of the Free State Foundation, when I wrote to him about an editorial he penned which also argued that violations have not happened: "It is generally agreed that except for a few isolated and quickly remedied incidents, neither the cable operators nor the telephone companies providing broadband Internet services have blocked, impaired or otherwise restricted subscriber access to the content of unaffiliated entities." He said he hadn't known about the AboveNet/TeleGlobe incident either.)

Another theme in some anti-Net-Neutrality editorials is that existing laws are enough to deal with the problem. In Majoras's speech, she said, "We should not forget that we already have in place an existing law enforcement and regulatory structure." Arrison's echoed that "Numerous federal agencies already have set a basic legal framework in place to preserve fair competition and business practices on the Internet". Well, as Yogi Berra says, in theory, there is no difference between theory and practice, but in practice, there is. After I found out AboveNet and TeleGlobe were blocking my Web site, I called about twenty lawyers in the Bellevue phone book, figuring: I wasn't greedy, but surely there would be financial damages for deceiving users and blocking our site, enough to pay a lawyer in return for handling the case? I think about two lawyers called me back, and they both said that even though what the backbone companies were doing clearly looked like fraud, it would take tens of thousands of dollars just to get started, and even if we ever got to court, the judge could call it however they wanted. Whatever laws exist now, they may help the slightly smaller big guy against the bigger big guy, but are not much use to the little or medium-sized guy.

So, any informed debate about Net Neutrality has to include the fact that, yes, some providers have blocked Web sites on purpose, for long periods of time, and no, the free market didn't fix it by itself. Even if something on that scale never happens again, if the free market and the anti-trust laws didn't automatically correct a case where Web sites were being blocked outright, then it's wishful thinking to think that those forces will prevent ISPs from merely slowing down Web access to sites that haven't paid a "toll", as they have made noises about doing. One AboveNet customer, Sam Knutson, said when he found out about the Web site blocking, "This type of behavior on the part of an ISP is reprehensible. I pay for a pipe and don't expect this type of monkey business." Well, I agree that it's reprehensible; whether we should "expect" more of it or not, depends on how much the Net Neutrality movement achieves its goals.

MaestroRC asks: "I've been looking around online, and I have found several people wanting to do the same thing, but no one seems to have figured it out yet. What I am wanting to do (and before you go further, understand this is for work, i.e.: no innocent people will be harmed in the implementation) is to set up a name server that selectively forwards queries. For example, I would like to create a list of acceptable domains (less than 20) using wildcards such as *.google.com, that the name server will forward a query on to and reply to normally. For anything not in the list, I want it to reply NXDOMAIN or some such. I've looked at BIND, and there doesn't appear to be a way to do what I'm wanting; it can either have recursion on or off, and any specific zones of type forward still do not forward if it is off. The solution doesn't have to be pretty, and it can just be a simple DNS proxy, but I'm not adept at coding, so it needs to be installable by a regular sysadmin on Linux. Has anyone heard of something like this?"

miller60 writes "Netcraft is reporting that 'domain registrar Joker.com says its nameservers have been hit with a massive DDoS attack, causing outages for customers. More than 550,000 domains are registered with Joker, meaning the outages could be widely felt. It's not clear why the DDoS is succeeding, as most registrars have implemented sturdy DDoS protection since the attack on the root nameserver system back in 2002.' Some security experts have warned in recent weeks about DNS recursion attacks as previously discussed here on Slashdot, which can amplify the power of attacks launched from botnets."

yootje writes "Infoworld is running an interesting article about Akamai and the DDoS attack that hit the network of Akamai Tuesday. According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC." Yootje points to another article on this subject as well, this one at Internetnews.com. Update: 07/07 19:38 GMT by T : Note that Vixie's quote here is actually presented out of context; he was commenting by way of contrast on the diversity of the root DNS servers, not Akamai's content-serving system.

bleachboy writes "Last week I completed a new DNS server survey, since D. J. Bernstein's hasn't been updated for years. Not surprisingly, BIND wins. Why is it so hard for alternate DNS servers to gain favor, especially when BIND can be so frustrating sometimes? And yes, I'm shilling."

darthcamaro writes "Time for net admins to update BIND: version 9.3 has been released. internetnews.com has a story on it where they talk with Paul Vixie, the founder of BIND's keeper ISC. In it he details why after so many years BIND has finally decided to offer commercial support. 'Many of the companies who use our software free of charge have told us that their corporate risk management strategy requires them to have a bona fide support channel for all of their critical operations,' Vixie said. 'In other words we were told that having the best software wasn't good enough, and giving it away for free wasn't good enough, we also had to ensure that commercial support was available or they could be forced to switch to software they didn't like as well just to get support.' The full press release on the BIND 9.3 release is also available."

darthcamaro writes "Time for net admins to update BIND: version 9.3 has been released. internetnews.com has a story on it where they talk with Paul Vixie, the founder of BIND's keeper ISC. In it he details why after so many years BIND has finally decided to offer commercial support. 'Many of the companies who use our software free of charge have told us that their corporate risk management strategy requires them to have a bona fide support channel for all of their critical operations,' Vixie said. 'In other words we were told that having the best software wasn't good enough, and giving it away for free wasn't good enough, we also had to ensure that commercial support was available or they could be forced to switch to software they didn't like as well just to get support.' The full press release on the BIND 9.3 release is also available."

An anonymous reader writes "After .COM and .NET started using a wildcard, the internet community busily started creating patches to various pieces of software to circumvent this. It was said that this was a grave problem to the internet. Several official BIND patches were announced over the next few days. However, it turns out they weren't necessarily too well thought through. Usage of the patch unexpectedly broke at least 7 Top Level Domains, ISC announced 3 weeks later, after users started having problems. The .NAME registry has sent a formal letter to ICANN's Security and Stability Advisory Comittee to warn against using the BIND patch, which they will look into in their next meeting. The intention may have been good, but... Stability? Anyone?"

chromatic writes "The O'Reilly Network has just published an interview with Paul Vixie, chairman of the board of the Internet Software Consortium and a primary author of BIND. Topics include the recent VeriSign controversy, ISC's BIND patch in response, and other potential issues that might come to light in the near future." On a related note, dmehus writes with a link to the letter sent by David Maher, chairman of the Public Interest Registry -- the .org registrar, to ICANN President and CEO Paul Twomey. "The letter says that it supports ICANN's call for VeriSign to voluntarily suspend SiteFinder and the Internet Architecture Board preliminary position paper. It goes on to say that PIR will not be implementing any DNS wildcard to the .ORG zone. It urges ICANN to stand its ground, but also to implement a policy preventing registries from taking this kind of unilateral action in the future." The letter is in .doc format, but AbiWord and OpenOffice.org both open it fine.

chromatic writes "The O'Reilly Network has just published an interview with Paul Vixie, chairman of the board of the Internet Software Consortium and a primary author of BIND. Topics include the recent VeriSign controversy, ISC's BIND patch in response, and other potential issues that might come to light in the near future." On a related note, dmehus writes with a link to the letter sent by David Maher, chairman of the Public Interest Registry -- the .org registrar, to ICANN President and CEO Paul Twomey. "The letter says that it supports ICANN's call for VeriSign to voluntarily suspend SiteFinder and the Internet Architecture Board preliminary position paper. It goes on to say that PIR will not be implementing any DNS wildcard to the .ORG zone. It urges ICANN to stand its ground, but also to implement a policy preventing registries from taking this kind of unilateral action in the future." The letter is in .doc format, but AbiWord and OpenOffice.org both open it fine.

BrunoC writes "Following the story about VeriSign's new Site Finder, the Internet Software Consortium promises to release a patch to its (in)famous BIND that will block the controversial Site Finder. Wired News has full coverage of the ISC initiative against this name resolving atrocity."

BrunoC writes "Following the story about VeriSign's new Site Finder, the Internet Software Consortium promises to release a patch to its (in)famous BIND that will block the controversial Site Finder. Wired News has full coverage of the ISC initiative against this name resolving atrocity."

JohnQPublic writes "BIND author and all-around Internet personality Paul Vixie and Mirror Image Internet have recently received US patent 6,581,090, specifically '..technology that efficiently stores and retrieves content requests and balances Web traffic between origin servers to improve performance and speed' - sounds an awful lot like what Akamai do. There's a press release from last week that gives some lovely 'details', including this little gem from CEO Alexander M. Vik: 'We anticipate that these patents and our technology solutions will encourage large groups of corporations to become customers of Mirror Image services. We also recognize that this technology is a critical component of other content delivery services and weâ(TM)ll be attempting to work cooperatively with our competitors and their customers to address this issue.' Can you say 'patent infringement suit'?"

A Sorry End writes "It appears that one of the 13 root-servers, the core of DNS name resolution, have moved away from BIND to NSD since wednesday, Feb 19th, 2003, which is a Good Thing. Since the 26th of october 1990, all root-servers have been running BIND. According to this message, this change was designed to increase the diversity of software in the root name server system, the lack of which is widely considered to be a potential vulnerability. The nsd software has been designed from scratch specifically as an authoritative name server. It has no design commonalities with bind, the currently prevalent DNS implementation. In addition to that nsd provides a significant increase in the performance reserve of k.root-servers.net. NSD was developed at NLnet Labs in coorperation with RIPE."

maiden_taiwan asks: "We're constructing a free groupware application by integrating well-known open source components: apache webserver, inn news server, ircd chat, scp for file transfer, etc. Unfortunately, each app has its own incompatible concept of a 'user identity.' Apache has the htpasswd module, IRC has nicknames, scp has public keys, NetNews has the poster's email address, and so forth. Has anyone managed to integrate a similar suite of apps using a single sign-on model, where a user has a single identity that is understood and carried through all these apps?"

eecue writes "The world's most popular DNS package is once again vulnerable. Even the advisory says it's only a matter of time before worms are written.... just like a couple years ago. I guess this is why i run tinydns."

bootc writes "Last week we heard the news that Jordan Hubbard was leaving the FreeBSD Core Team. I received an email about the new OpenDarwin.org web site and had a look around, just to find that our friend Jordan was member of the OpenDarwin Core Team!" Apple has consolidated its Open Source web site, including Darwin, under its developer site, while the Internet Software Consortium is hosting the independent OpenDarwin.org, which will develop OpenDarwin with the developer community and collaborate with Apple to merge OpenDarwin technologies into Darwin and Mac OS X.

wizzy writes "Irelands toplevel domain registry has a notice on Microsoft and Apple DHCP clients sending dynamic DNS updates per RFC2136. The problem is they are not sufficiently careful about where they send it if they are in RFC1918 space - usually used for behind-firewall addressing, which is where they usually are.. This is resulting in bogus updates being sent at the rate of nearly one million an hour to root nameservers, only to be rejected - as reported on the NANOG mailing list."

The Dev was the first of several zillion to point out that security holes were found in BIND. The detailed table of known vulnerabilities will help clarify (and it has tarball links too), but the short version is, if you're running BIND 4 or BIND 8, set aside some time today to upgrade to 4.9.8 or 8.2.3 (not beta, betas of 8.2.3 are vulnerable). And now's a good time to reconsider version 9, too. SecurityFocus warns that the last time a BIND hole of this magnitude was found, it was followed by a "cyber-crime wave." Exploits for these holes were successfully created by COVERT Labs, but nobody seems to know whether they're in the wild yet. Obviously, they soon will be. Post your questions and answers about upgrading below.

resistant writes: "This article at Network World Fusion (seen at Linux Today) says, "In addition to DNSSEC, BIND 9 features support for IPv6, the ability to run on multiprocessor systems and improved scalability for handling large domain name zones." The urgent need (by Nike anyway, heh-heh) to forestall easy domain hijacking could be the sleeper issue that finally ushers in universal implementation of IPv6."

rwm311 writes: "linuxsecurity.com is running an interview with [Paul Vixie] and [David Conrad] about the ISC and BINDv9. It's a pretty good read. Vixie talks about his days at DEC and his motivation behind BIND while both Vixie and Conrad speak of the future of BIND - features they would like to implement and things that will be going away (such as nslookup)."

rwm311 writes: "linuxsecurity.com is running an interview with [Paul Vixie] and [David Conrad] about the ISC and BINDv9. It's a pretty good read. Vixie talks about his days at DEC and his motivation behind BIND while both Vixie and Conrad speak of the future of BIND - features they would like to implement and things that will be going away (such as nslookup)."

Eric Sun writes "After numerous release candidates and betas, the final stable release version of Bind 9 has been released. Looks like the homepage hasn't updated yet, but you can get a list of download servers from its page at Freshmeat."

Ben Galliart writes "The Internet Software Consortium released on Monday another patchlevel to their ever popular BIND software package. The ISC has recommended that everyone using BIND upgrade to this latest version (BIND v8.2.2 patchlevel 3) due to security holes existing in previous versions. If you are using a version previous to BIND 8.2.1 then pay special attention to the ISC configuration hints on a new required TTL setting which should be added to every zone file. More information on the TTL setting is also available in RFC 2038. On a side note, those who enjoy the bleeding edge should read the ISC future plans page which now has information on the thread-safe/multi-processor ready BIND version 9 (major rewrite) going beta in January. "

Ben Galliart writes "The Internet Software Consortium released on Monday another patchlevel to their ever popular BIND software package. The ISC has recommended that everyone using BIND upgrade to this latest version (BIND v8.2.2 patchlevel 3) due to security holes existing in previous versions. If you are using a version previous to BIND 8.2.1 then pay special attention to the ISC configuration hints on a new required TTL setting which should be added to every zone file. More information on the TTL setting is also available in RFC 2038. On a side note, those who enjoy the bleeding edge should read the ISC future plans page which now has information on the thread-safe/multi-processor ready BIND version 9 (major rewrite) going beta in January. "

Ben Galliart writes "The Internet Software Consortium released on Monday another patchlevel to their ever popular BIND software package. The ISC has recommended that everyone using BIND upgrade to this latest version (BIND v8.2.2 patchlevel 3) due to security holes existing in previous versions. If you are using a version previous to BIND 8.2.1 then pay special attention to the ISC configuration hints on a new required TTL setting which should be added to every zone file. More information on the TTL setting is also available in RFC 2038. On a side note, those who enjoy the bleeding edge should read the ISC future plans page which now has information on the thread-safe/multi-processor ready BIND version 9 (major rewrite) going beta in January. "

Ben Galliart writes "The Internet Software Consortium released on Monday another patchlevel to their ever popular BIND software package. The ISC has recommended that everyone using BIND upgrade to this latest version (BIND v8.2.2 patchlevel 3) due to security holes existing in previous versions. If you are using a version previous to BIND 8.2.1 then pay special attention to the ISC configuration hints on a new required TTL setting which should be added to every zone file. More information on the TTL setting is also available in RFC 2038. On a side note, those who enjoy the bleeding edge should read the ISC future plans page which now has information on the thread-safe/multi-processor ready BIND version 9 (major rewrite) going beta in January. "

Dan Marks sends us the followign:"DNS protocol is highly vulnerable to spoofing attacks, and Cylink and RSA may provide royalty-free (at least for a limited time) signature systems to prevent these attacks. The Internet Software Consortium is choosing a standard for DNS security. Read this article"