Slashdot Mirror
Ask Slashdot: How To Both Mirror and Protect Crowdsourced Data?
New submitter cellurl writes "I run wikispeedia, a database of speed limit signs. People approach us to mirror our data, but I am quite certain it will become a one-way street. So my question is: How can I give consumers peace of mind in using our data and not give up the ship? We want to be the clearing house for this information, at the same time following our charter of providing safety. Some thoughts that come to mind are creating a 'Service Level Agreement' which they will no doubt reject, or MySQL-clustering, or rsync. Any thoughts, (technically, logistically, legally) appreciated."
You'll only be THE clearing house if you are the best source. Second, it's public data, stop trying to own it, you can't, it's not yours to own in the first place.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
There are plenty of publicly accessible sites that mirror data from trivial to critical. I would contact a few of them and see what agreements they have in place, if any.
I would think you would want to make sure they note their data is a mirror, and that updates should be sent to your site. That might be handled by doc files for each file, or some type of about file in each directory. You probably want something like that if for no other reason so as to note metadata.
I've seen quite a few sites that prefer that you go to a mirror to download actual data.
much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
Consider teaming up with a seasoned negotiator with good business sense, or hiring an attorney -- or both. If there is any value in your dataset, those who got in touch with you will not reject fees, SLA's, reciprocal updates, etc. It all depends on how much data you have, and how accurate it is.
On a separate note: your site is disfunctional on my tablet. I'm left wondering what it's about or how it's supposed to work.
create an API and provide an interface where your client base can interface with the data.
there are a lot of places out there that does this, as its considered Intellectual Property.
It's not a typo if you understood the meaning!
Might not translate exactly, but look into how the openbsd project mirrors their stuff. There is the main site, tons of mirrors. Everything is hashed. Grab a mirror, if you don't trust it get the hashes from the main site and check the files. Not sure if it would scale to what you're doing. And what do you mean by 'giving up the ship' exactly?
License the mirroring only in the event that:
1. It's visibly acknowledged that you are the source site
2. updates are either directly sent to you, or are sent to you by the other site within a time limit
3. All content on your site, including that sent to you by another (mirror) site, be watermarked as belong to your site. For pictures, this would be a visible watermark on the picture.
Be the best Make all information free Choose a good licence Expect to be taken over one day from something better, when that comes along ... help them
Make it easy for anybody to use your information
It is counterintuitive but the moment you put up protective barriers, you fall over. The moment you depend on an artificial barrier to protect your lead is the moment you will degrade the quality of your product. Happens every time on products and services that grow on openness and suddenly feel the reason they are good is more so because of their qualities than the openness.
If you develop a product/service based on a closed environment, that is a different story. It makes business sense to improve your model based on a closed environment until a disruptive product/service comes along.
This is a compilation of public data, with the legwork being done by others. You've got no real legal option in protecting the data, at least in regards to the US. You could perhaps try some technical means of controlling the data, but that would greatly reduce the utility. I would also consider in unethical to try and 'own' the results of work done by other unpaid volunteers. If you wish to be the center of this data collection, than make it as useful as possible.
This is my signature. There are many like it, but this one is mine.
You want to "Protect...Data", "not give up the ship", "follow...our charter of providing safety". But what is it that you don't what mirrors to do with the data? Less verbiage, more clarity, please.
Daily MySQL dump to S3. Done. Your welcome.
This project is terrible and has gotten only marginally better with time. The woman in the video sounds very attractive. She needs a sexy slashdotter to help her out.
If it's safety you want, I don't understand why you are trying to get other sites to freely back up your data.
Get a real backup service and tell people how it's backed up, poof! safety.
Or if you want to make a community resource you can do like sourceforge, ibiblio, etc, free mirrors that point back to your site.
You don't want to mandate people give you data. That will just get you bad data. Instead, make it as easy as possible for them to do - APIs, easy web forms, any method you can think of that will make the barrier to entry as low as possible. Encourage them to use it, but relax and set your data Free and don't try to force it. It's like Wikis... Somehow it works out OK.
Fully Homomorphic Encryption. FHE. See http://en.wikipedia.org/wiki/Homomorphic_encryption#Fully_homomorphic_encryption
As we keep getting told, information wants to be free.
Stop being a scummy copyright-enforcer and let people share the information.
Simple.
You don't lose anything if others mirror the data, instead you gain an even wider audience, it's all good.
From the home page "the sign you capture is copyrighted with your name since you found it".
How on earth can you copyright a speed sign, and even if you could, how can that copyright be relevant to anything?
The location and speed limit of a speed sign is a fact. How can that be copyrighted? How can it limit the rights of others who observer the sign to publish its location and speed limit?
If anybody were entitled to copyright a speed sign, it would be the authorities that put it there and who actually own it. How can the location of other peoples property be copyrightable? Looks like somebody took the concept one step too far...
Gets a thousand years of bad karma.
If you were blocking sigs, you wouldn't have to read this.
I've been working with phone directories for a few decades, where many companies are in basically the same position that you are - making a living from public information. Most data is collected from phone companies that dump their customer databases to the phone directory companies. This process and the associated tariffs are regulated by law. This data must be processed and cleaned up before it is passed on. Then there are data consumers - in the old days these were people reading the phone books. These days, data consumers are people browsing the web and all sorts of web apps that connect to the phone book through one of several apis. Most telephone directory companies provide search apis for their databases - usually not for free. Everything is a one-way street, of course. Information flows downstream, money goes upstream. No phone directory company that I know of will voluntarily mirror their database to anyone. Search APIs, yes. Mirrors, no. Phone directories are sometimes distributed to consumers and businesses on cd/dvd, but never without at least an attempt at scrambling and restricting its usage. You could probably make a business for a while selling an open, mirrored copy of your database. People will pay for subscriptions. The problem is, any one of your customers could choose become your competitor at any time. The more successful you are the more likely someone is to do that. Maybe you can protect yourself legally, but most people prefer to lock their door even in jurisdictions where trespassing is forbidden. Competition in your area would be nice for everyone else, your customers as well as your competitors, so as a member of "everyone else" I should say go for it. But you're no dummy. You got your company name posted on slashdot after all!
you want to be 'clearing house', so you want to be the main provider, if in fact you are that provider right now, then you can put a time delay on all new data that you gather, so that whoever mirrors it from you will get the data say 10 days after you collected it.
Oh, and API.
You can't handle the truth.
Distributed and redundant p2p web server and storage application.
Everyone is the site and your data! Or at least several dozen hashed pieces of it.
Information goes in and never gets deleted Or maybe not without the master key. Which can be entered from any application copy running.
Use all the tricks of spammers, botmasters, trojan writers, and virus experts.
With one huge difference. It is opt in.
What is the point? If you're too blind to read fucking traffic signs, how about not driving?
To have a right to do a thing is not at all the same as to be right in doing it
Just like the GPL but also closes the loophole that allows you to use an open source tool in SAS without giving back. I would investigate this licence. Also, map maker usually put distinctive voluntary mistakes in their maps to prove when data has been copied.
The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
Since you stated that themirros would not sign a SLA, but you are the one that is concerned about the customer expierence,
(a) make it easy for them to stay up to date with providing a proper API to retrieve updates
(b) monitor the usage of that API and mail your contacts if they need assistance
Finally you might want to publish the current state of all mirrors in public. You are trying to open your data, so do not rely on contractual enforcement, but transparency.
You could fix that first.
Sorry, this probably isn't what you want to hear, but it sounds like you're wasting your time. Your desire to protect the data simply doesn't make any sense; the best way to protect data is to make it more open. Why are you trying to place restrictions on it? I would forget the project entirely and add maxspeed data to OpenStreetMap. The project is much larger than yours, and has much more impressive maxspeed results.
OpenStreetMap
If you're worried mirroring MIGHT be a one way street, you don't understand the concept.
Mirroring = multiple people hosting "reflections" of the same master data. It's used primarily for large data or files that consume a lot of bandwidth. You have mirrors to reduce the bottleneck of a single host to download from, and the network latency of being further away from the master.
Mirroring != a collaboration mechanism or a crowdsourcing concept where anyone can contribute to any repo they want. Think (for example) of all the RedHat mirrors out there. They're not SUPPOSED to contain anything not in the master. This would be a PROBLEM with a mirror. People go to the mirror because they trust it's the same data they'd get from the master. And the master won't pick up stuff from the mirrors if they DO get something new in there. If someone wants to contribute, they contribute to the master. The master updates, and those updates flow out to the mirrors via replication so they're all in sync.
It sounds like (from T very terse FA) like you're expecting "mirroring" to allow other people to collect similar data and share it with you? If so, mirroring is a bad choice. So help us out - why do you want this?
i've always liked the idea of a torrent-based file-system. tfs, if you will, to run on linux's virtual file system. a radical solution that would require major programming, but it would be pretty cool, imo.
First, you make an obvious play on someone else's good name (Wikipedia). Then you want to somehow protect public data that you have collected. Sorry man. Not sure what you're trying to accomplish but if it's not getting laughed at and possibly sued you're doing it all wrong.
Go on Shark Tank.
A real contender for the most worthless website on the planet. Ergo, it will catch on and make millions in advertising.
I've thought about similar problems for years, and you simply cannot restrict the flow information in the noosphere - there's no practical and ethical way to do it. If someone wants to, they can scrape your data from a myriad of proxies (very easily scripted), and then dump it on BitTorrent. In trying to prevent this, you'd only be hurting the convenience of your site, meaning someone else can build a more convenient site that organizes the same type of data and attract your contributors. By making it easy for people to export your data, you'll make your users more conscious of the value that you're openly providing, and thus more likely to donate. You need to let your babies grow up and move out, and be confident that if you've raised them right they'll appreciate it...
In other words - release the data under a license that requests attribution, and ideally no other restrictions. You can always use Google / Archive.org / etc to timestamp and prove where the data originated. You can do that and still get all the credit and donations, because anyone trying to plagiarize credit will be easily exposed as fraud. We need to educate users to appreciate those who add value to their free software / data, and ostracize anyone who abuses that openness (like ad-filled Wikipedia mirrors that offer no added value). Search engines are gradually getting better at penalizing moocher sites, and rewarding the sites where the content originally appeared.
As for the means of distributing data - I don't recommend allowing the general public access to database replication. It's much more effective to do an SQL / JSON / CSV data dump, compress it (PPMd usually offers better and faster text compression over xz), and distribute it over BitTorrent with HTTP fallback. You can do a full data dump every X days, and a partial update every Y hours, depending on your data size. (This also has the advantage of boosting your data proliferation / backup-fu.)
What I would really like to see is something like pkgsrc / CPAN for data - an automatic way to install and update publicly available data sets into one's local database, with application of diff/patch files in sequence, automatic conversion between RDBMS systems (IMHO, PostgreSQL > MySQL), configurable filtering, BitTorrent downloading with configurable generosity of seeding time / bandwidth, etc. This would lead to a smarter and more reliable Internet, and make it impossible to censor sites like WikiLeaks or The Pirate Bay by making them so much easier to preemptively mirror.
One interesting analogy is how this principle would apply to Slashdot comments. For a very long time now, I've been just drooling over the idea of scraping all Slashdot comments since time immemorial, parsing it to SQL, and putting a data dump on BitTorrent. There's plenty of room in them terabyte storage arrays, but 40 million comments averaging say 5KB per comment (with RDBMS overhead and indexes) is only 200GB - or about 8GB in a compressed SQL dump. This way I can tag all my "AC" writings and make sure they're preserved, since Googlebot only sees the highest-ranking comments (and, well, my copyfree advocacy isn't very popular), and make all my past writings searchable. But, well, I'm lazy... Hope someone else beats me to it... ;-)
--libman
Many counterfeit jerseys will cheap custom Nike NFL Jerseys have the backing still attached on the inside of the jersey Only you need to sacrifice the time and patience In addition, there are some sports jerseys cheap shops, in fact, the overwhelming majority of the people to customize and design their own basketball jersey opportunity, simply choose your own color palette cheap wholesale jerseys By going through the online shops that are there, you will be able to get the order the particular items that you want at a very cheap price You really can taste just how fresh the seafood used in dishes is, so make sure you sample some of the high-quality cheap customized nba jerseys shellfish such as crab, scallops, cheap custom Nike NFL Jerseys lobster and oystersIn this case, you will hardly find a jersey of Rodgers from the Minor League, but you can also order the Aaron Rodgers Jerseys of this kindLight transparent material or mesh is the next in fashion when it comes to men's t-shirt fashion Because of this, these customers interchanged NBA jerseys alongside Iran game enthusiasts Some of the important reasons for damage of these essential parts cheap jerseys are extensive use and age of your laptop If you would like to get the most ideal sites, NBA Jerseys it is as simple as listening to what your friends and even family have to say about the sites that they have previously dealt with Ohio, you vixen, anyone! And in case you hang out cheap nike nfl jerseys wholesale while you're watching TV while using true hockey fans, you'll begin to post a little nike nfl jerseys cheap every now and then There are numerous advantages to this type of movie gathering Genuine NFL jerseys can be divided into three categories, from low to high are: offset fan version (Replica lynch jersey Jersey), high-level fan edition (Premier Jersey / EQT Jersey), and player version (Authentic)
If you custom jerseys cheap intend to have more children and use the jersey as a hand-me-down, buy only quality made items so the lettering does not crack on them Some of the tactics may consist of paid on the web advertising, promoting in cheap custom nike cheap sports jerseys nfl jerseys research engines, advertising on web sites, and even, posting in forums One of my favorites was the Ballet Studio, because I fostered my love for balletRelated ArticlesNew Euro 2012 Spain Home Soccer Jersey / Football ShirtEuro 2012 Football Shirts To Be More Popular Than EverWhich Euro 2012 Soccer Jersey Will Be The nike nfl logo tee images nfl jerseys cheap Bestseller?MOR
Maybe cheap authentic nike nfl jerseys not your fault, just because someone decided to cheap nike nfl jerseys wholesale go to the past and be in contact with a terrible turn Bob Ortiz Property Hat Even so lots of USA's participants instantly gives a powerful impulse, acknowledged people on no custom nfl jerseys cheap account situation FIFA points, dream to often be cheap nike nfl jerseys very helpful cheap mlb jerseys so that you could Iran athletic men and women, a fabulous desire could deepen cheap customized nba jerseys of brett lawrie uniform which knowledge involving youre a set issues In this economy however, it goes down to zero for some China offers plenty of cheap jerseys, which is actually in the store to buy their shares, but there are other packages which provide cheap custom Nike NFL Jerseys cheap, buy a larger unit They may have this support that you can need to have all day every dayawebd Coach NBA Jerseys Doc Rivers does not deny itRockstar & Pro Circuit/Monster KitsBoth the legendary Rockstar and Pro Circuit/Monster Energy team colors are available in CORE and PHASE kits so you can look like a factory rider and nike nfl jerseys cheap race like a champion History, heritage and a driving desire to cheap nike nfl jerseys create riding gear that enhances a rider's performance has made THOR the choice of champions for decades You can use this information to decide whether you should modify the ineffective website or stop running nfl jerseys cheap these advertisements There are so many options with Dish Network and dishnetwork channels deals G
discount football jerseys marshawn Larger number of avid fans, the basketball jerseys nfl logo tee images is not just the playersSummer T-shirt Fashion Trend 2012 for MenThis time the summer elite nfl jersey fashion t-shirt trend of 2012 is cheap nike nfl jerseys perfect for creating a fashion statement Only cheap nike elite nfl jerseys you need to sacrifice the time and patienceLong ago, the sea between Jersey and France was a low, marshy plain with a few low hills, of which today only the Ecrehous reef protrudes above the waterline I bought from one shop online which is wholesale ch