Slashdot Mirror

← Back to Stories (view on slashdot.org)

Huge Site Ranking Dataset Donated to the Common Crawl Foundation

Posted by Unknown on from the fuzzy-feelings dept.

Greg Lindahl writes "blekko is donating search engine ranking data for 140 million domains and 22 billion urls to the Common Crawl Foundation. Common Crawl is a non-profit dedicated to making the greatest (yet messiest) dataset of our time, the web, available to everyone, including tinkerers, hackers, activists, and new companies. blekko's ranking data will initially be used to improve the quality of Common Crawl's 8 billion webpage public crawl of the web, and eventually will be directly available to the public."

23 comments

  1. Can I Get A Copy ... by Anonymous Coward · · Score: 0

    Can I get a copy of the 'Leather Bound 1st Edition'? I'd prefer free shipping.

  2. I didn't know by plover · · Score: 2

    I didn't realize the web wasn't available to everyone, including tinkerers, hackers, activists, and new companies. Thank $(DEITY) the Common Crawlers are here to make sure that my port 80 hasn't yet been pried from my cold, dead fingers.

    --
    John
    1. Re:I didn't know by Nyder · · Score: 2

      I didn't realize the web wasn't available to everyone, including tinkerers, hackers, activists, and new companies. Thank $(DEITY) the Common Crawlers are here to make sure that my port 80 hasn't yet been pried from my cold, dead fingers.

      I have no idea what is going on here. I am stoned (I live in Washington State, it's like law or something) so I'll admit that maybe I'm not in the right state for thinking. (I did a pun there, sorry)

      I did actually go to the web site, saw they were hiring, and read the FAQ.

      Still have no idea why they are doing what they are doing. Hoping someone will explain the purpose of Common Crawlers in terms I can understand and maybe a car analogy.

      --
      Be seeing you...
    2. Re:I didn't know by Anonymous Coward · · Score: 0

      Yeah, having to use an Amazon map-reduce node to access it makes it harder to access than just going to port 80. Make a torrent of it. A really big torrent.

    3. Re:I didn't know by TheRealMindChild · · Score: 1

      It is a data set about the internet. This saves you from having to crawl the web, analyze it, and build your own database.

      --

      "When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
    4. Re:I didn't know by L1s4 · · Score: 5, Informative

      The idea is to give everyone access to crawl data. If you work at a large search company, you have access to crawl data. You can also set up crawlers to get the data yourself, but that is expensive and having countless crawlers doing duplicative work is not ideal. Our idea is that there should be one common repository for crawl data that anyone can use. Researchers are using it for NLP, IR, sentiment analysis and many other things like measuring the adoption of metadata formats http://www.webdatacommons.org/ Educators are using it as a real world dataset to teach big data techniques in the classroom. Developers and entrepreneurs are using it for startups. Sorry I don't have a car analogy :) Feel free to email me if you have any other questions lisa at commoncrawl dot org

    5. Re:I didn't know by Anonymous Coward · · Score: 0

      I don't think you understand how big the data is.

    6. Re:I didn't know by plover · · Score: 1

      I was actually mocking the slashdot story editor for claiming they were providing a copy of the web instead of providing a copy of web metadata obtained by crawling.

      Here's your car analogy: Web crawling is like a guy driving down every street in town and taking a picture of every vehicle he sees, and analyzing them to figure out make, model, year, license plate number, etc. The guy can either choose to sell the information, or he can make it freely available under a creative commons license and publish it on the web.

      So if you want to open a gas station and are wondering if you should install a diesel fuel pump, you can either drive all the streets yourself to count all the diesel cars, you can pay the one guy to find out how many diesel cars he counted, you can download a copy of the freely available car information, or you can make a guess. Only one of these choices costs nothing yet leads to making an informed decision.

      --
      John
    7. Re:I didn't know by Nyder · · Score: 1

      The idea is to give everyone access to crawl data. If you work at a large search company, you have access to crawl data. You can also set up crawlers to get the data yourself, but that is expensive and having countless crawlers doing duplicative work is not ideal. Our idea is that there should be one common repository for crawl data that anyone can use. Researchers are using it for NLP, IR, sentiment analysis and many other things like measuring the adoption of metadata formats http://www.webdatacommons.org/ Educators are using it as a real world dataset to teach big data techniques in the classroom. Developers and entrepreneurs are using it for startups.

      Sorry I don't have a car analogy :) Feel free to email me if you have any other questions lisa at commoncrawl dot org

      Thanks, that explains it better.

      --
      Be seeing you...
    8. Re:I didn't know by Anonymous Coward · · Score: 0

      I'm curious whether that website will include itself and its dataset in the dataset...

    9. Re:I didn't know by Anonymous Coward · · Score: 0

      Car analogy: it is like how $250k spent on a bus is more efficient for more people than 120 20k cars would be.

  3. Kudos to the blekko team by Anonymous Coward · · Score: 0

    I've met some of the guys from blekko, and they do solid work, for the web as well as for their site. QED.

  4. How does Common Crawl compare w/ Internet Archive? by Anonymous Coward · · Score: 0

    How does this project compare with the Internet Archive?

  5. I always wanted to be ranked... by 3seas · · Score: 1

    .... its better than being judged.

  6. Re:How does Common Crawl compare w/ Internet Archi by Anonymous Coward · · Score: 3, Funny

    How does this project compare with the Internet Archive?

    commoncrawl.org will be available on archive.org a lot longer than it will be available on commoncrawl.org

  7. Blekko has horrible search results by Anonymous Coward · · Score: 0

    Blekko has pretty horrible search results compard to Google, Bing or even Duck Duck Go. So the raw data may be useful, the actual ranking data should not be trusted as being anywhere close to good.

  8. Re:How does Common Crawl compare w/ Internet Archi by L1s4 · · Score: 3, Informative

    Hi I work at Common Crawl. Internet Archive is awesome and does really important work. The main difference between us and Internet Archive is that you can analyze our data. Internet Archive is a vault and is not available on a platform where you can run jobs against it. Because we put it on Amazon and other compute platforms, anyone can access our data and run jobs against it. If you wanted to do that with Internet Archive's crawl you would have to ask permission, get permission, and download it to your personal data center in order to analyze it. I don't know too many people with a personal data center :) Lisa

  9. Re:How does Common Crawl compare w/ Internet Archi by Anonymous Coward · · Score: 0

    CommonCrawl is awesome! We have already used it for multiple projects at my work. Very cool!

  10. Wikipedia already has a ranking of huge sites by Anonymous Coward · · Score: 0

    Here.

  11. Re:How does Common Crawl compare w/ Internet Archi by Anonymous Coward · · Score: 0

    Because we put it on Amazon and other compute platforms, anyone can access our data and run jobs against it. I don't know too many people with a personal data center :)
    Lisa

    Thanks. Hopefully you do know a few people with access to a data centre...like AWS.

  12. Re:How does Common Crawl compare w/ Internet Archi by colin_faber · · Score: 1

    Hi, Since you work there, do you have any idea how much data we're actually talking about here?

  13. Yacy by Anonymous Coward · · Score: 0

    What implications (if any) does this have for Yacy?

    http://yacy.net/en/ (the distributed search engine)