Slashdot Mirror

The federal government already built the site to host arbitrary open data: https://www.data.gov/open-gov/

Just upload them there.

There are large databases maintained by many federal agencies/organizations. These include NOAA, Census Bureau and NASA. Some provide FTP access, some provide an API, and some require going through a web interface -- and some provide all three. Some of these can easily result in downloads of many gigabytes, sometimes zipped up into one custom file for your request. Yet, not one that I've run across even requires registration, let alone paying anything.

So, why would the courts charge for access to public data that is much more central to the proper functioning of a society?

It's like the courts really haven't gotten beyond the notion of paper archives with costly human workers digging through dusty file cabinets to retrieve the data and copy it onto dead trees. That's a little scary since these are the same organizations that are our last resort for civil and criminal justice.

If you want to motivate yourself, bring some data for a problem that interests you. Data.gov ( http://www.data.gov/ ) or your national equivalent, financial data from Yahoo, something that you find though an Open Data ( https://en.wikipedia.org/wiki/... ) site. Then develop your skills while working on real data. I code best in conjunction with unit tests, and for anything above a trivial exercise I like an actual data sample.

I refuse to call it a list of 'data centers', because of their changing definition of what a 'data center' is, but you used to be able to get lists from data.gov ... unfortunately, they've now got so much stuff in there that it's hard to find much of anything. The project to shut everything down goes by the name FDCCI:

http://www.data.gov/search/node/FDCCI

I've personally run into this problem on a couple of occasions when making FOI requests. Once I requested court transcripts from a case that I wanted to provide to the local newspaper as evidence of an incredibly incompetent prosecutor, but the county courthouse wanted thousands of dollars to copy the transcripts and would not allow me to simply come down and copy them myself. I ran into a similar problem with the Department of Transportation when trying to build a database of VIN numbers for a used car sales site 14 years ago. They had no electronic records and only companies with huge pockets could afford to send people down to photocopy the new VINs every month (stack of papers the size of several telephone books) and ship them off to India for data entry. It basically killed our business model. The first example felt like a local court playing CYA, while the second was DOT simply having no incentive to make its data accessible benefiting larger corporations who could throw money at it.

I do feel it's getting better though. Things like data.gov and the Open Data Initiative are things we should be applauding, because there are some incredibly useful datasets that we the taxpayers have funded and now have access to. When things happen like this story of the AP being effectively blocked from FOIA via a bureaucratic maneuver, we should be outraged, but let's not forget the progress we're making and let our cynicism override the truth that we can change the system.

This initiative has been going on for a while. Is the issue that not enough agencies are getting their data out fast enough, or comprehensively enough?

I'm also a little bit skeptical of relying on a random private company, GitHub, to be the canonical data host. What's wrong with hosting it on data.gov? Or if it's going to be hosted in the private sector, how about with a public-interest organization like the Internet Archive?

I think Science Debate is the greatest thing to happen to those of interested in science and politics. When they got Obama and McCain to answer science questions in the 2008 election, I immediately cancelled my membership to the Union of Concerned Scientists and started donating to this grassroots organization.

I have one issue that I vote on, and that's science. It's the only issue I understand well enough to evaluate the candidates on. If they know their science or have advisors that understand science, then I will trust them with most everything else. I summarized Obama's 2008 responses here, McCain's here, and my calls for who won on each issue. Obama's responses won on most issues, but McCain did not do poorly. Since Obama has taken office, he has impressed me with his support of science with Data.gov, Science.gov, a Memorandum on Scientific Integrity, proposed major increases in science funding, and put the Office of Science and Technology Policy back in the Whitehouse.

These might seem like small accomplishments, but compared to the Dark Ages of the Bush Administration they were a breath of fresh air. Unless Romney answers the science debate questions this election cycle, I won't even consider him.

My state does that as does the US federal government. My personal favorite is:
The MN DNR data deli

There are others as well:
Federal GIS data
MN legislature GIS resource
MN DOT
Minnesota MetroGIS
2011 US Tiger data set
USGS data
National Atlas
This data is all freely available and you can go do what you want with it. Granted you need some program that can parse and display shapefiles but those aren't too hard to come by as there are some very capable open source ones available.

Even at poverty in the US, you still get to go to school, have vaccinations and so on.

And a majority of those also have more than 2 color TVs (including 1 plasma), a cell phone, and a car.

Source

The 2011 update to data.gov actually allows whoever is submitting the data to describe it such that people can make use of it, including via visualization (maps, graphs, etc.) or via API to make custom applications.

So my question for Carl would be : What can we do to get more government agencies to actually put their data in there? And if they won't do it, should resource.org or similar groups work to put up something similar, so that people who have gotten information through FOIA can share it back out to wider audiences?

You mean like OODT ( ) ? or something more like iRODS ? Both are used by various 'big data' groups (NASA, NIH, NOAA, NOAO, super computing centers) to share data across multiple sites.

As for the indexes .... well, if science.gov and data.gov are any example, they could use some work. Although, hopefully in this case, you're describing bibliographic records, so the necessary metadata is a little more standardized.

In some cases, I'd be better to just put the records out there under standardized open APIs, and let interested parties make interfaces to the stuff they're interested.

In 2005, the typical household defined as poor by the government had a car and air conditioning. For entertainment, the household had two color televisions, cable or satellite TV, a DVD player, and a VCR. If there were children, especially boys, in the home, the family had a game system, such as an Xbox or a PlayStation*

* http://explore.data.gov/Energy-and-Utilities/Residential-Energy-Consumption-Survey-RECS-Files-A/eypy-jxs2

http://www.data.gov/

They're at least making efforts to add transparency.

I went to download 2005 Toxics Release Inventory data for the state of California and the only link was for a .csv. When I went to download it, up comes an .exe file. Why the binary executable?

The Government (a.k.a. Big Brother or THE ILLUMINATI) wants to infect your computer with uninvited wiretapping software, of course! And once the NWO buys control of Google, every time you visit a Google web site, your computer will send the collected data to Big Brother and the Vatican, so that they will know everything you do and can enforce their agenda for globalization upon you!!! DON'T BE FOOLED!!! ITS A TRAP!!!

Or maybe they're just idiots? They probably spend more time playing WoW in their cubicles than working on data.gov if my past experience working for the government is any indicator.

I went to download 2005 Toxics Release Inventory data for the state of California and the only link was for a .csv. When I went to download it, up comes an .exe file. Why the binary executable?

I see they've kept up with the latest in web design. When you go to search for geodata, the search list is constrained to a tiny rectangle in the middle of the page. You have to scroll within that tiny rectangle. On my monitor, the page is about a foot tall, and I'm tediously scrolling in this inch-high box.

I've learned to recognize state of the art web design when I see it. I bet it's even CSS compliant. They're not quite there yet. To be really great web design, it should be a Flash only site.

(close captioning for the sarcasm impaired: this was sarcasm)

The problem with anecdotal evidence, is that people arguing the exact opposite point can pull out a dozen examples too. In this article John Derbyshire pulls out a dozen examples of why Obama is trying to kill science in the United States. It's not convincing to anyone who knows about National Lab Day, Educate to Innovate STEM initiative, Computer Science Week, data.gov, and the Policy Forum on Public Access to Federally Funded Research... but this is all anecdotal too, a better resource would be an overview of all the projects being funded by the stimulus package or trends in government funding of scientific research.

However, I do completely agree that Tyson is being unfair to the American government. In fact, this is the same guy who previously argued Republicans were doing a great job of funding American science. The real issue here, and the one we are dealing with most in computer science, is American Culture's antipathy and outright contempt for science and academia. Kids aren't going into Computer Science, Physics, Chemistry, etc, because they are afraid of being associated with "geeks." The kids all want to be gangsters, models, and sports stars... not realizing how unrealistic those dreams are and that only a miniscule percentage of people succeed in those arenas.

We need a culture change, we need to be proud geeks and make others envy us. It'll help us out in the long run.

Yep. It's spending and campaign contribution data I want to see side by side.

Here you go: http://www.data.gov/suggestdataset

...how about making those divs/viewports/things-containing-the-checkboxes a little bigger here? There are 30 checkboxes and you can only see two at a time; I feel like a Cylon!

Wow these are some amazing data sets . http://www.data.gov/catalog#raw

It is limited to geological, environmental, and weather datasets. Ohh yea, and the usability of the website is about as good as you'd expect from the government.

$ curl -i http://data.gov/
HTTP/1.1 301 Moved Permanently
Date: Thu, 21 May 2009 18:13:00 GMT
Server: Microsoft-IIS/6.0

Bah!

Would you prefer this method? :)

$request=[system.net.httpwebrequest]::create("http://data.gov") ; $request.getresponse() | select-
object {$_.Server}

$_.Server
---------
Microsoft-IIS/6.0

Search for everything selected and...

A total of 27 results comes up. At least they got this database there for future use. Now all they need to do it create a part in the stimulus bill to hire all those newly graduated psychology degree holders to preform data entry.

$ curl -i http://data.gov/
HTTP/1.1 301 Moved Permanently
Date: Thu, 21 May 2009 18:13:00 GMT
Server: Microsoft-IIS/6.0

Bah!