Net Neutrality Comments Overtaxed FCC's System

Presto Vivace writes with news that the FCC has had trouble dealing with the sheer volume of comments submitted about net neutrality. There were millions of them, and they caused problems with the agency's 18-year-old Electronic Comment Filing System (ECFS). When the FCC attempted to dump the comments into XML format to make download and analysis easier, problems with Apache Solr meant roughly 680,000 didn't make the transfer. The agency promised to release a new set of fixed XML files in January that include all of the dropped comments. Despite many reports that the comments were "lost," they're all available using the ECFS.

Comments

Need a slow lane

[Leuf]

If only the FCC website could have been put in a slow lane they wouldn't have had this problem.

Re:Need a slow lane

[davester666]

I'm sure Comcast and AT&T would be more than willing to give the FCC a fast lane for comments arguing against net neutrality.

Re:Need a slow lane

[jmac_the_man]

I'm sure Comcast and AT&T would be more than willing to give the FCC a fast lane for comments arguing against net neutrality.

Comcast is pro-NN because Net Neutrality was imposed on them as a condition of their merger with NBC. They want their competitors to be hamstrung by the same NN rules that they are.

Re:Need a slow lane

[davester666]

Um, no. While they do have some conditions imposed on them until 2018, they have sued over FCC's regulations for net neutrality and won. Hell, these conditions were IMPOSED on them for violating the laws and regulations w.r.t. internet traffic.

And from Comcast directly: “we do not support reclassification of broadband as a telecommunications service under Title II.”
http://corporate.comcast.com/comcast-voices/comcast-files-open-internet-reply-comments

Re:Need a slow lane

[jmac_the_man]

Part of the problem with Net Neutrality is that a bunch of groups use the name to mean different things. You, in presumably good faith, saw "Net Neutrality" and meant "Title II reclassification." I, also in good faith, meant it in the sense of "regulated settlement free peering," which is the condition that the FCC imposed on Comcast, but for the moment, few other companies. (Comcast wants that regulation extended to its competitors as well.)

Way out on the wacko scale, you have people trying to impose the now-discredited "Fairness Doctrine" on the Internet, and calling THAT Net Neutrality. There are people out there saying that every X times you type Fox News into your browser, it should bring you to NPR instead. To some people, that's net neutrality.

I think the bigger issue

[rsilvergun]

is the number of Astroturfer comments. I'd like to see some effort made to find and eliminate them.

Re:I think the bigger issue

[MillionthMonkey]

See link: http://sunlightfoundation.com/... Half of the petitions were anti-NN, and mostly came from a Koch-backed organization's form letter:

Dear Mr. Wheeler, As an American citizen, I wanted to voice my opposition to the FCC's crippling new regulations that would put federal bureaucrats in charge of internet freedom, and urge you to stop these regulations before they're enacted. If the federal government goes through these plans to regulate the internet, I know that the internet will change -- and not for the better. [ INSERT VARIANT PARAGRAPH COMMENT HERE ] Like many Americans, I believe that the internet should remain free of government control and unnecessary regulation -- just as it has for the last twenty years of unprecedented growth. Please stop the FCC's dangerous new regulations, and protect the future of internet freedom here in America. Sincerely, [APPLICANT NAME] [APPLICANT HOME ADDRESS]

As for the "VARIANT PARAGRAPH COMMENT", apparently you were given several selections to choose from, including the following:

The Internet is the biggest economic, intellectual, and artistic success story of the century, and it rose up because of free people, not stifling government. The federal government needs to keep its hands off the Internet. It is not broken, and it does not need to be fixed. It is the federal government, not the Internet, that is broken, and in need of fixing.

One can make an appeal to justice for persecuted cable companies:

Before our government can handcuff a citizen, it must have some reasonable evidence that they have done something wrong. Before the FCC places regulatory handcuffs on Internet providers, shouldn't the government present evidence that they have actually done something wrong?

Or maybe this is your style:

The ideological leader of the angry liberals calling for you to reduce the Internet to a public utility is Robert McChesney, the avowed Marxist founder of the socialist group Free Press. In an interview with SocialistProject.ca, McChesney said: âoeWhat we want to have in the U.S. and in every society is an Internet that is not private property, but a public utility...At the moment, the battle over network neutrality is not to completely eliminate the telephone and cable companies. We are not at that point yet. But the ultimate goal is to get rid of the media capitalists in the phone and cable companies and to divest them from control.â In a country of over 300 million people, even an extremist like McChesney can find, perhaps, millions of followers. But you should know better than to listen to them.

Re:I think the bigger issue

[charronia]

The FCC might be able to reduce the number of comments they have to read by running them through a letter plagiarism filter.

Re:I think the bigger issue

[Presto Vivace]

Mods,l please mod parent up

Re:I think the bigger issue

[MillionthMonkey]

It is a good tip for reducing the amount of astroturfing. I'm sure the FCC will get right on that very soon.

Re:I think the bigger issue

[Kuroji]

I'm wondering how long until someone just finally snaps and tries to go after the Koches.

Hmmm...

[mmell]

The wonderful people at the FCC will forgive me if I find the nature and timing of this announcement to be . . . convenient? I'm sure the many PAC lobbyists working for the major ISP's will find this news very welcome indeed.

Reading Between The Lines

[Greyfox]

I'd guess they probably tried to dump several gigabytes of comments into one gigantic XML document, and their... lessee 18 years... I'm going to say, DG/UX system couldn't handle a file of that size. Is that about right? I don't even want to know what hokey solution they duct-taped to their system to get it to spit out XML.

If only there were some sort of magical agency that knew how to deal with communications and could actually design a decent transfer format for these guys. I bet that very same agency might know why it would be important that the people who provide access to the Internet should not be allowed to constrain those channels based on who you decide to talk to.

Re:Reading Between The Lines

[Vladius]

Most of them are probably from some organization with a name like "People for Freedom of Speech" or "Americans for Liberty" or some other name to get the Tea Party's panties wet.

They only store them for us to read

[fustakrakich]

The FCC already has its orders. The 'comments' thing is just a pacification measure.

Re:They only store them for us to read

[PhrostyMcByte]

They've already given word they intend to throw out comments that don't give an in-depth analysis... so the great majority of these comments that are just like "hey i use this stuff, please don't fuck with it" will just be ignored.

Re:They only store them for us to read

[rsborg]

The FCC already has its orders. The 'comments' thing is just a pacification measure.

I'm guess it's more akin to "parallel construction" whereby if the comments provide sufficient cover for your existing orders, you can claim that it was a mandate of the constituency, and if not, then you have to do extra work to reframe it so that it is.

Still wondering why we can't have tax id used to authenticate messaging for such comment sites. I mean, like that's a guaranteed unique identifier, non? Its not like you're not putting your name/address on the comment anyway are you?

Re:They only store them for us to read

[Anonymous Coward]

The FCC is required to provide for a public comment period. They might even be required to consider the comments before making any rule change, but if so, it would be impossible to enforce. The only real question is what kind of backlash they might face, and from whom (Congress who can cut funding, the President who can fire the upper management, the citizens who can complain and hope to influence Congress or the President).

Re:They only store them for us to read

[Sir_Eptishous]

Wait, you're not insinuating that the FCC is only truly beholden to a cabal of incredibly powerful and wealthy elites who are the ones actually calling the shots via intense lobbying of spineless and paid off politicians and back room deals with utterly despicable business leaders?

Re:They only store them for us to read

[jmac_the_man]

Still wondering why we can't have tax id used to authenticate messaging for such comment sites. I mean, like that's a guaranteed unique identifier, non? Its not like you're not putting your name/address on the comment anyway are you?

Isn't the tax ID used for most individuals their Social Security numbers? Putting that on comments (which obviously need to be made public) is an invitation to identity theft.

It also doesn't solve the claimed problem. Groups putting out form letters would just include a field for SSN. Then you'd have a bunch of near identical comments with different social security numbers.

It also doesn't show that it's a real problem. The Constitution says that citizens are allowed to petition the government for a redress of grievances, right? What is a petition except a letter asking for the government to do something with a bunch of citizens signing it, ALL OF WHOM USE THE SAME WORDING? Why is the fact that they all use the same wording a disqualifier in the first place?

Re:They only store them for us to read

[fustakrakich]

Me? insinuate? Ohhhh no, never! It's just an observation of what invariably happens when people reelect corrupt politicians to 40 year careers in the expectation they will bring home some serious pork. It's what keeps the Great American Boondoggle alive. As far as democracies go we have nothing new here. The whole ordeal has become quite ordinary.

Now you have 2 problems

[Anonymous Coward]

XML? Easier?

XML???

[djbckr]

I really don't get it. XML wasn't designed for things like this, and yet people still want to use it as a database. It's one of the most goofy things I've come across in my years as a developer. I've been working in this industry since before XML was even an idea, and it's still a bad one (well, for things like this anyway). Admittedly it's good to get data from one system to another *in small chunks*, but don't try to move so much data in one block.

Re:XML???

[Jason Levine]

I'll admit that when I first used XML, I started trying to use it for as much as possible. (Blame "Shiny New Technology" syndrome.) I had written stuff to databases before but I thought this technology would make so many things easier. Years later, when I'm reviewing my old code, I'm finding that removing the XML and moving the data to a database improves everything. XML definitely has its place, but it also has limits. Trying to export a million comments as an XML file is almost guaranteed to run into problems.

Re:XML???

[Anonymous Coward]

There's nothing wrong with XML per-se and it's perfectly capable of million record exports.

The problem is stupid developers trying to load entire documents and manipulate them with DOM-like procedures when they should be using streaming parsers.

Re:XML???

[Anonymous Coward]

your homework assignment is to filter the duplicate "astroturfed" comments, using a streaming parser.

wtf

[Anonymous Coward]

http://apps.fcc.gov/ecfs/proceeding/view?name=14-28

Why is verizon the very first entry?

Shit is so corrupt, it boggles the mind.

Re:wtf

[Anonymous Coward]

Likely because it was the most recently entered, and you are looking at the "Recent Public Filings" list.

XML is being generous

[langelgjm]

Having looked at the XML files they provided, while some are marginally useful, about 1.4 million reply comments are simply dumped into CDATA sections without any consistent formatting or separators. After much regex splitting, it was readily apparent that the downloads they provided were missing about 700k comments. Glad to see they're actually admitting it and may even remedy it.

Government the Answer?

[Anonymous Coward]

These idiots can't get a web site to work, yet we want them to dictate rules to the Internet? Anyone remember the healthcare.gov web site? It didn't work either, because the government "didn't anticipate" the load of millions of people who were complying with the laws that they passed.

I'm going to go get a Koch

[Anonymous Coward]

I'm wondering how long until someone just finally snaps and tries to go after the Koches.

"I'm going to go get a Koch, and I'm going to shake that Koch vigorously until he spasms and goes limp."