Slashdot Mirror
Where's the Open Data?
blamanj asks: "There's a lot of open-source code around, and generally, it's quite easy to find. Finding open source data, on the other hand, can be quite a pain. Why isn't there a common reponsitory for public domain data sets? I'm thinking of things like lists of world cities, dictionaries of stemmed words, population data, etc., etc."
Right Here.
you can't have that data. you might use it to kill someone.
die fucker. information should be banned.
Follow da links. Use Google's cache.
:(
Data = Money
MIT's SuperArchive
Grabbe the link off of rootprompt in case any of you care
NOAA provides Bathymetry data and electronic navigation charts (vectorized) and NIMA (that's right, .mil, -- NIMA used to be the Defense Mapping Agency provides city lists and populations for all the countries in the world, as well as DEMs (digital elevation models--i.e. gridded topography). The National Atlas project provides boundaries of federal lands, outlines of states, locations of major cities, stuff like that.
ENJOY!
Why isn't there a common reponsitory for public domain data sets?
There is, it's right here.
(aka The Internet)
"And like that
the world almanac? Oh no, a book? made of paper? what's that? Just because it isn't in digital form doesn't mean it's not out there. Try the public library sometime.
The GeekNights podcast is going strong. Listen!
I understand the importance of Google and the like, but let me give an example from the MS side of things, and why we love it.
SQL Server and Access both come with the Northwind database. If I have some new query that I'm trying to write, for instance randomly returning different numbers of products for each product category, it is pretty darn handy to have a standardized data set to pull from for my example code.
Otherwise, I have to include DDL and DML just to create the example data. Instead, I can just say "Run this against Northwind."
The same applies for training and learning. Northwind is a pretty well known database, and most established developers won't have to learn a new schema in order to demonstrate a new concept.
So rephrase the question from "Where can I find some data?" to "Where can I find a data set that other developers are using so we can more intelligently exchange information?"
"Tax preparation software eliminates errors your[SIC] may make...." From IRS home page.
--here's a great site. refdesk.com. Matt Drudge's father runs this site,AFAIK, a boatload of data links.
Not only the WWW, but the entire Internet is at your fingertips. It's not that the data is available, it's that it's sometimes hidden.
That kind of data is out there if you want it, but it isn't always in the format you want it and sometimes it's hard to find (less hard with google). I started a site for this kind of data at www.tonsoflists.com, which contains data in MySQL tables that you can format into different kinds of sets,orderings, formats, etc. Just did it for fun, but haven't gotten any feedback.
nobody posted this - standardized data sets for training AI. It's a start, anyways - useful for comparing one machine learning system to another. Maybe you could use it for something else?
What's the deal with NASA data? Especially Hubble data? Sure would be nice to make some screensavers without those unsightly logos on them.
:-)
e s/images_index.php3
Oh, yeah... just remembered a nice bookmark!
http://earthobservatory.nasa.gov/Newsroom/NewImag
The NASA Earth Observatory. Don't know how open, though.
This is stuff you can't download for free from Yahoo, CBOE, or other places.
If I can just get access to this data, then I will make enough money to purchase the other data.
Keeping
For example, ever run out of trivia questions in your version of Trivial Pursuite? Or used up all the word cards in Taboo... etc etc.
I think in the event of running out of data for your board game, it would be nice to download more. (And this would make a cool website.)
Especially if they came with PalmPilot/Windows versions that would administer the game for you. For example Taboo consists of a word that you must get the other people to say, but there are 7 words that you CANNOT say as a clue. For example the word may be "George Bush" and you can't say "Texan", "President", etc. This game is fun but we calculate we'll use up all the data that comes with it in about 30 hours. The "electronic" version is $40. That's hardly worth it. If we could just download data, we could play forever. So .. um .. yes.. I want open data.
-Clio
Karma: Bad (mostly from not giving a fuck)
Blog: http://clintjcl.wordpress.com
When you go to Google to find software to fill some specific need, you already know quite clearly how to search. The problem with finding "open data" is that there currently is not any commonly used clear label on such texts, research and articles. I tend to mention that the content is released under the GNU Free Documentation License or FDL when I want to release something to be freely utlized by anyone. One such case is for example the Amazon Discoveries series. Not that it would be any useful for anyone :) This problem is a bit related to the problem of releasing your idea or concept under such license - there does not seem any clear practise how to go on about this :: what to do if your idea might be unique but you do not want to patent it. We have that exact problem with for example the Openchallenge concept submissions. Any ideas on what practises to use in that case would help us out.
I agree in theory that we need a Semantic Web where content is easier to find, but I don't think XML-etc can really help. [rant]
My current theory is that individuals need to build the 'Necessary Web' which consists, like an encyclopedia, of a page for each topic (or many pages by different authors, on their own websites). Four special traits make a page qualify as 'Necessary':
-- an attempt to be FAQ-like, and briefly cover all the important subtopics on a single page.
-- an attempt to sort thru and link all the best web-resources on the topic. (By reducing the linktext to one- or two-word [text buttons] you can fit hundreds of links into a useful page.)
-- a timeline, to present the most possible data in the neatest possible way. [theory]
-- The Open Web Content License to encourage others to recycle-and-update your content, requiring only that they clearly link your page as one of the original sources.
Most recent example of this format: Linux/Unix (timeline w/100s of links)
I believe that once a critical mass of authors adopt this format, taking on the most useful topics, there will be a rapid shift from the current search-frustrations to something very much like the Semantic-Web ideal, without even requiring any fancier technology than simple HTML.
it costs more money to get data than to make code.
The World Wide Web. Here's a library: http://promo.net/pg/ Go try google. Did you just make this question to see if you could get in?
Surprise. No one as yet seems to have mentioned ibiblio?
Timeo idiotikOS et dona ferentes
Check out and add to:
I realize this is slashdot, but maybe there's other stats geeks out there that like baseball. Baseball-Reference is hands down the best stats site out there. And it's based on the Lahman database which is freely available (newest version coming soon).
-Gabe
the entire Internet is at your fingertips
AND, it's on computers now.
Yeah, *THIS* post might be offtopic, but the above wasn't. That's the first post that I'd ever had moderated down, and this will likely be the second.
"Hmmmm, he mentioned MS in a fashion not suited to bashing, he must be moderated down." I bash MS quite often, but I will give them credit when it's due.
Why don't you reply so we can have a little discussion instead of hiding behind anonymity? Oh, never mind, I forgot that this is slashdot.
"Tax preparation software eliminates errors your[SIC] may make...." From IRS home page.
--I play trivial pursuit once a week on irc with some friends. We do it ad hoc round robin style, and the good part is, you come up with your own questions. It's your "turn" as long as you stump people, whenever someone gets the answer, it's their turn, and so on. Seems to work great. You start a session by locking in the theme of the questions. Everyone gets a chance to both share knowledge they have, and also to learn from the others.
I guess if you wanted to get it in some sort of doc form you could use a session log and tweak it.
There are plenty of government sources of data that is free and open to anyone. The Census Bureau, Energy Information Administration, Commerce Department, a good starting place is FirstGov .
In addition, most state governments and even county level governments publish large amounts of data.
"The old forget, the young don't know" --Japanese Proverb
Someone should mention the UCI Machine Learning Repository and the UCI Knowledge Discovery in Databases Archive.
Lots of great links there, but you left out The CIA's world fact book. They publish as much as they can so that anyone (including their own agents) can access the needed information, from anywhere. World Fact Book http://www.cia.gov/cia/publications/factbook/index .html
"Those who cast the votes decide nothing; those who count the votes decide everything." - Josef Stalin
While looking for available domain names, I came across OpenTrivia.com. I like the spirit of what he is doing, though the license is a little more restrictive than what I would prefer.
Much as you suggested, it uses XML to allow people leeway on structuring a quiz. It doesn't offer multiple choices, though. Thats not really a concern for the Trivial Pursuit application, but still, I'd like to have it as an option.
SpyDock: Scientific Python in a Docker container
Hi Slashdot. I'm in junior high school and have a book report due tomorrow and I can't seem to get logged into cheater.com. I'm really pissed I can't find some l33t paperz on the web to hand in as my work, can you help?
...you have to have open data types that is ubiquitous as well. i see a lot of the in-fighting and disagreement of the 'best way' to do things in the open source community as sometimes counterproductive to this end...
i am awaiting the obvious response that a beowulf cluster of xml will save the world...
Large print giveth, and the small print taketh away
A good starting URL - http://www.biotech.ubc.ca/bioinform.html#public
How the hell was the first comment offtopic? Oh, we aren't supposed to discuss stuff? The second reply came after the moderation, and I already said it was likely to be offtopic.
I'm sorry I've been around for a while and don't Karma Whore with everyone else. I post my opinion. Always. I don't cater to what I think will get posted up or down.
Use your points to mod intersting stuff up instead of wasting them by modding down stuff you don't agree with. Christ, there were how many trolls that weren't touched when my comment got modded down? My comment was the best one that could be found to be modded down?
This comment will be my third to get moderated down, and in the same thread no less. There are far more deserving comments that should have had the point used to be modded up.
Yeah yeah, YHBT, YHL, HAND, but I don't really care.
"Tax preparation software eliminates errors your[SIC] may make...." From IRS home page.