Open Data Needs Open Source Tools

macslocum writes "Nat Torkington begins sketching out an open data process that borrows liberally from open source tools: 'Open source discourages laziness (because everyone can see the corners you've cut), it can get bugs fixed or at least identified much faster (many eyes), it promotes collaboration, and it's a great training ground for skills development. I see no reason why open data shouldn't bring the same opportunities to data projects. And a lot of data projects need these things. From talking to government folks and scientists, it's become obvious that serious problems exist in some datasets. Sometimes corners were cut in gathering the data, or there's a poor chain of provenance for the data so it's impossible to figure out what's trustworthy and what's not. Sometimes the dataset is delivered as a tarball, then immediately forks as all the users add their new records to their own copy and don't share the additions. Sometimes the dataset is delivered as a tarball but nobody has provided a way for users to collaborate even if they want to. So lately I've been asking myself: What if we applied the best thinking and practices from open source to open data? What if we ran an open data project like an open source project? What would this look like?'"

Comments

Standards by Domain needed.

[headkase]

High-level: Save your differences from day to day, bittorrent those differences to others, merge back in differences from others. Low-level: OMG, we used different table-names.

Re:Already being done

[Hurricane78]

I've said this a thousand times before: Make Wikipedia a P2P project without a single control, and build a cascading network of trust relationships on top of it (think CSS rules, but on articles instead of elements, and one CSS file per user, perhaps including those of others), and you solve all problems with then not-existing central authorities, and so also with censorship.

The only caveat: People have to learn again, who to trust and who not. (Example of where this fails: Political parties and other groups with advanced social engineering / rhetorics / mass psychology skills, like marketing companies.)