Slashdot Mirror

← Back to Stories (view on slashdot.org)

Is Microsoft Crawling Google?

Posted by CmdrTaco on from the put-on-your-foil-hat dept.

triplecoil writes "Jason Dowdell over at WebProNews has written a piece questioning a tactic Microsoft might be using to beef up its new search engine. He thinks they might be dipping into Google's results to supplement its own. Dowdell likens it to leaving your garbage on the curb--anyone could conceivably go through it and take whatever is there for their own."

9 of 480 comments (clear)

  1. They been crawling like mad lately by mpost4 · · Score: 5, Interesting

    I can say that they been crawling like mad as of late, Google, Yahoo, and MSN. I say this because on my site I have had a lot of traffic from all three, and my site is not a popular, or even an important one but I seen a lot of traffic from them. Not just once a week or a few times a week but every day. There are big updates coming. I was not surprised to see the article about google doubling their index, I know something was coming from the way they are crawling unimportant/unpopular sites.

  2. They really only need to seed their crawler... by JustNiz · · Score: 5, Interesting

    You can't get to every page on the internet just by starting at one page and recursively following links, therefore the more places you from, the more likely you are to have 100% coverage.

    I could imagine that Microsoft just needs a few thousand URL's evenly-spread across the internet just to seed their crawler, which they can get from Google by using a list of most popular queries.

    Once their crawler has so many starting points it can do the rest itself.

  3. Re:Does it violate Google's Terms of Service by TheRaven64 · · Score: 4, Interesting

    Do Google's terms of service have any legal standing? Click-through EULAs don't in many jurisdictions, and I don't remember ever even seeing Google's ToS, let alone agreeing to them.

    --
    I am TheRaven on Soylent News
  4. Re:Don't concern yourself with this crap... by mollymoo · · Score: 5, Interesting
    No offense dude, but you are the one who put the site out their publically. Now if they are DoSing you then you have a valid complaint but robots.txt is just there as a friendly suggestion.

    There's more to it than that. Google caches your pages and makes that cache of your copyright material available. Arguably if you have used your robots.txt file to tell it not to index (and therefore cache) your pages and it still does they are breaching copyright. OK, the Google cache is the world's largest breach of copyright anyway, but if you have told its spider not to index and it does regardless, that's a different ballgame.

    Putting it out there on the web does not give anyone the right to do with it as they please.

    --
    Chernobyl 'not a wildlife haven' - BBC News
  5. Re:Difficult to do if Google doesn't want them to by blamanj · · Score: 5, Interesting

    Yes, and don't think Google wouldn't notice. My company had a summer intern that once wrote a program that started sucking a lot of information out of Google. They blocked our entire site for about three days until everything got straightened out.

  6. Re:Microsoft stealing someone elses technology??? by netringer · · Score: 4, Interesting
    I fail to see how they are stealing any of Google's technology. Data maybe.
    Are are they stealing Google's innovations?

    Lo! Note how the review articles of the last few days mention the innovative NEW FEATURE of MSN search called, "Search Near Me" which stores the calculated lat/long of addresses on web pages and returns matches near you.

    Note how Google's long in beta Google Local (http://local.google.com) stores the calculated lat/long of addresses on pages and returns matches near you. Google Local works better.

    Another Microsoft innovation! Let's hope WE remember who had it first!
    --
    Ever dream you could fly? Get up from the Flight Sim. I Fly
  7. Re:Don't concern yourself with this crap... by liquidsin · · Score: 4, Interesting

    Hmmm...let's call "robots.txt" a "copyright control device" in that it states who may and may not have access to my copyrighted images directory. I'd bet a DMCA suit or two for circumventing your copyright control device would get them to pay attention...

    --
    do not read this line twice.
  8. Full Circle by Guppy06 · · Score: 5, Interesting
    "Dowell likens it to leaving your garbage on the curb--anyone could conceivably go through it and take whatever is there for their own."

    It's interesting to know that Bill Gates has been forced to go back to his roots...
    The best way to prepare [to be a programmer] is to write programs, and to study great programs that other people have written. In my case, I went to the garbage cans at the Computer Science Center and fished out listings of their operating system.
  9. Re:Try this term on MSN search by Red+Alastor · · Score: 4, Interesting
    Sure. Bill Gates is an atheist so he think that God is evil. Open Source too, specially that pesky browser eating his market share.

    Before you mod me down for that, I'd like to mention that this isn't Microsoft bashing since I am an atheist too and so are Linus and RMS.

    --
    Slashdot anagrams to "Sad Sloth"