Slashdot Mirror

There aren't too many organizations public or private left in the world, that can replicate Google's indexing capabilities.

It's not necessary to have a company the size of Google to do Google's web search. CPU power and disk space have been increasing faster than the amount of content on the web. Indexing the web isn't all that big a job any more. Cuil, despite their problems, did it with about $20 million, 50 people and about 1000 servers. Their relevance algorithm was terrible and their crawler hit the same pages too often, and by the time they had those fixed, it was too late for them. But they did index all the pages they could reach.

Common Crawl, which is a modestly sized nonprofit, maintains a crawl of the whole web. So does Archive.org Neither maintains a search engine, though.

The next frontier in web search is to crack the provenance problem - figure out which pages are derived from other pages, and list the original source first. Google is not good at this. There are constant complaints on Google support forums about scraper sites outranking the original source. Google doesn't get this right for video, either. A video search system which found the best copy of something and discarded all the copies with ads, logos and recompression would be far better than what Google provides.

Google can't do that. They make money off AdSense ads on third-party content. Scraper sites are a gold mine for Google. 30% of Google ad revenue is from AdSense. Much of that is from copied content.

Warning heeded, but I saw this on a blog post at commoncrawl.org.

This bucket is marked with Amazon Requester-Pays flag, which means all access to the bucket contents requires an an http request that is signed with your Amazon Customer Id. The bucket contents are accessible to everyone, but the Requester-Pays restriction ensures that if you access the contents of the bucket from outside the EC2 network, you are responsible for the resulting access charges. You don’t pay any access charges if you access the bucket from EC2, for example via a map-reduce job, but you still have to sign your access request. Details of the Requeser-Pays API can be found here: http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RequesterPaysBuckets.html

If I understood that right, at least getting started with the tutorial will not result in me coughing up $200. Correct me if I am mistaken.

Isn't http://www.commoncrawl.org/ better?

It's obviously foolish to use public text verbatim as a key. Common Crawl has a 40 TB dataset that costs approximately $150 to MapReduce on EC2. Any key that happens to be a (reasonably short, say under 1KB) substring of that data costs $150 to break. Any key within a short hamming distance of a substring in that database costs roughly 2^hamming_distance more to break; two changed bytes is only worth $600. I imagine that large organizations who care have much larger databases including the text of most published books. It's such an obvious idea and until you realize that attackers have access to all the public source data that you do it sounds like a good idea to just pick a random string from a book to use as a passphrase. Don't kid yourself; no matter how obscure or unpopular a song is there will be lyrics for it somewhere on the Internet, not to mention in published books. You can take a published string and make it a reasonably secure passphrase by adding enough entropy to it, but you still have to remember the entropy that you've added. Why not just start with a diceware passphrase and memorize the entropy directly?

I guess reading comprehension isn't your strong suit. I'll assume that you are an ESL person rather than a moron. I said:

...It's even better if you *mis-remember* the quote/lyrics so that you're the only one who would come up with the result even if someone tried to brute force the key by scanning all your books and listening to all your music.

Get it now?

It's obviously foolish to use public text verbatim as a key. Common Crawl has a 40 TB dataset that costs approximately $150 to MapReduce on EC2. Any key that happens to be a (reasonably short, say under 1KB) substring of that data costs $150 to break. Any key within a short hamming distance of a substring in that database costs roughly 2^hamming_distance more to break; two changed bytes is only worth $600. I imagine that large organizations who care have much larger databases including the text of most published books. It's such an obvious idea and until you realize that attackers have access to all the public source data that you do it sounds like a good idea to just pick a random string from a book to use as a passphrase. Don't kid yourself; no matter how obscure or unpopular a song is there will be lyrics for it somewhere on the Internet, not to mention in published books.

You can take a published string and make it a reasonably secure passphrase by adding enough entropy to it, but you still have to remember the entropy that you've added. Why not just start with a diceware passphrase and memorize the entropy directly?

Because it's easier to remember something like
"Four score and seven years ago, the only thing we had to Fear was Fear itself. And in this hole, there lived a Hobbit. Live long, and prosper."

It's obviously foolish to use public text verbatim as a key. Common Crawl has a 40 TB dataset that costs approximately $150 to MapReduce on EC2. Any key that happens to be a (reasonably short, say under 1KB) substring of that data costs $150 to break. Any key within a short hamming distance of a substring in that database costs roughly 2^hamming_distance more to break; two changed bytes is only worth $600. I imagine that large organizations who care have much larger databases including the text of most published books. It's such an obvious idea and until you realize that attackers have access to all the public source data that you do it sounds like a good idea to just pick a random string from a book to use as a passphrase. Don't kid yourself; no matter how obscure or unpopular a song is there will be lyrics for it somewhere on the Internet, not to mention in published books.

You can take a published string and make it a reasonably secure passphrase by adding enough entropy to it, but you still have to remember the entropy that you've added. Why not just start with a diceware passphrase and memorize the entropy directly?