Microsoft To Launch Homegrown Search Engine
Mr. Christmas Lights writes "While Google is currently the king-of-the-hill in search engines, Microsoft continues to lag in market share and uses Yahoo's technology/results. But Cnet reports that they'll launch on Thursday their own homegrown search engine , although it appears this is mostly a face-lift (despite a year of development and $100 million investment). According to Bill Gates, they 'will introduce a homegrown web crawler and algorithmic search engine ... later this year,' which is almost certainly their tech preview (you can look at this now) -- but will that be ready for prime-time in less than two months?"
This article is from june 30th
I wonder whether that's the bot that has been scanning my website for three days by attempting to "crawl" through all session ids and causing more then 1 GByte of traffic.
"msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
It was only stoppable by blocking the IP. (robots.txt was only read once before it started) Great, smart bot, really.
It pays attention to robots.txt directives (finally, a small amount of standards compliance!)
http://blog.nexusuk.org
It payed attention for me with:
/
:)
User-agent: msnbot
Disallow:
iptables -A INPUT -p all -s 65.54.0.0/16 -j DROP
Or even better, if you have the TARPIT module:
iptables -A INPUT -p tcp -s 65.54.0.0/16 -j TARPIT
http://blog.nexusuk.org