Microsoft Bots Effectively DDoSing Perl CPAN Testers

at_slashdot writes "The Perl CPAN Testers have been suffering issues accessing their sites, databases and mirrors. According to a posting on the CPAN Testers' blog, the CPAN Testers' server has been being aggressively scanned by '20-30 bots every few seconds' in what they call 'a dedicated denial of service attack'; these bots 'completely ignore the rules specified in robots.txt.'" From the Heise story linked above: "The bots were identified by their IP addresses, including 65.55.207.x, 65.55.107.x and 65.55.106.x, as coming from Microsoft."

Comments

So how do we DDoS Microsoft?

[drinkypoo]

Anyone know what sites on Microsoft's front-facing sites are most computationally intensive, and yet always dynamically generated? :D

Re:So how do we DDoS Microsoft?

[Anonymous Coward]

Bing? ...But that would only help them to DDoS Bing.

Re:So how do we DDoS Microsoft?

[Lennie]

http://blogs.msdn.com/

I've seen it fail many times

Re:So how do we DDoS Microsoft?

[SharpFang]

No, we just make mistakes writing our Perl programs for automatic downloading stuff from MSDN. Like, download() unless success, and forget to set success=true;

Re:So how do we DDoS Microsoft?

[jisatsusha]

All that'd serve to do is make them look more popular than ever. Traffic up 300%! Sounds like a good mar

Re:So how do we DDoS Microsoft?

[Anonymous Coward]

That exactly what i said. Dont you dare leech the score from me jackass!

Re:So how do we DDoS Microsoft?

[jlp2097]

Not necessary. A Bing Product Manager has already commented on the CPAN Testers blog entry upon which the article is based:

Hi,
I am a Program Manager on the Bing team at Microsoft, thanks for bringing this issue to our attention. I have sent an email to barbie@cpan.org as we need additional information to be able to track down the problem. If you have not received the email please contact us through the Bing webmaster center at bwmc@microsoft.com.

As said below, never ascribe to malice that which can be adequately explained by stupidity. (Insert lame joke about MSFT being full of stupidity here).

Re:So how do we DDoS Microsoft?

[John Hasler]

Seems like the CPAN admin has already solved the "issue".

Re:So how do we DDoS Microsoft?

[Anonymous Coward]

As much spam as I get from ir@infousa.com , I wish that someone would DDOS that damned company. If I knew of a way to get extra spam to ir@infousa.com I would probably do it so that company could get a taste of its own medicine. ir@infousa.com sent me unsolicited spam and it drives me nuts. Thanks for nothing, ir@infousa.com . It makes me want to call the company at (402)593-4500 and complain, but I don't have time. I guess I'll email them at ir@infousa.com instead. maybe.

Re:So how do we DDoS Microsoft?

[kulnor]

Well, with Barbie(TM) on the case, this should be quickly resolved (unless she's too busy with G.I.Joe(TM))

Re:So how do we DDoS Microsoft?

[PetoskeyGuy]

Why make things worse? Block the ip address or range and notify the admins. This isn't a chan mob.

Re:So how do we DDoS Microsoft?

[Zarf]

Clue: Subtle joke, deserves 'funny' moderation ;)

Subtle + Slashdot = FAIL

Re:So how do we DDoS Microsoft?

[Anonymous Coward]

"as we need additional information to be able to track down the problem."

IP addresses aren't enough? You're MS--if you can't fix the problem and IP addresses are given, damn, that's just sad. You're freaking massive multi-billion dollar tech companies, and this is the best you can do?

No wonder Chinese hackers own our asses.

Then again, it took Comcast 9 months to fix a security hole in customer accounts (which would have required an s to http to make pages SSL'd), and the only reason it was "fixed" was because they did their annual website makeover and changed their entire system to something Flash based. Then again, I had contacted a VP, VP's security, referred to web security, and talked to web security 3x, talked to a manager. The last 3 groups verified the problem. It was referred to their web applications team by that point, who sat on it.

Lovely world we live in.

Re:So how do we DDoS Microsoft?

[Penguinisto]

As said below, never ascribe to malice that which can be adequately explained by stupidity. (Insert lame joke about MSFT being full of stupidity here).

Given the back-story on the whole Danger data loss affair, stupidity is the FIRST thing I'd ascribe to Microsoft these days...

Re:So how do we DDoS Microsoft?

[Penguinisto]

So, really, the Perl guys deserve to be on the receiving end of some shitty code for once.

So, err, it's .NET versus Perl (okay, PHP) then in a battle over whose customer base can mis-use whose code the worst on the public Internet?

Fuck this - I'm going back to BBS.

Re:So how do we DDoS Microsoft?

[WinterSolstice]

Actually, your statement works better with 'INSERT LANG HERE'...

I'm always surprised by how people seem to think that any language has a monopoly of some sort on sloppy and/or lazy coders. Been doing IT a long time, and the one thing that never changes is the sloppy/lazy code issue. It even predates programming, you know - look at infrastructure around the world for examples of "just toss something out there, hope it works".

Re:So how do we DDoS Microsoft?

[Short Circuit]

A quick guess? Identifying unique sites by domain name, rather than by IP address, and either the bot or server not respecting HTTP 301 redirects.

With Rosetta Code, I once had www.rosettacode.org serving up the same content as rosettacode.org. My server got pounded by two bots from Yahoo. I could set Crawl-Delay, but it was only partially effective; One bot had been assigned to www.rosttacode.org, while another to rosettacode.org, and they were each keeping track of their request delay independently. I've since corrected things such that www.rosettacode.org returns an HTTP 301 redirect to rosettacode.org, and have was eventually able to remove the Crawl-Delay entirely.

I've since worked towards only serving up content for any particular part of the site on a single domain name, and have subdomains such as "wiki.rosettacode.org" redirect to "rosettacode.org/wiki", and "blog.rosettacode.org" to "rosettacode.org/blog". Works rather nice, though it does leave me a bit more open to cookie theft attacks.

YMMV; As I said, that was a quick guess.

Re:So how do we DDoS Microsoft?

[Spatial]

How horrible are your employees at their jobs when they require the assistance of their victims to fix the problem?

[Every IT worker on Slashdot looks around nervously]

Re:So how do we DDoS Microsoft?

[Hurricane78]

As said below, never ascribe to malice that which can be adequately explained by stupidity.

Must be really easy to just beat you in the face, and say “Ooops, I’m sorry, I’m so st00pid! *drool*”
I call bullshit on that rule.

My rule: Don’t make judgements at all (either way), about things that you just don’t know.

Re:So how do we DDoS Microsoft?

[jc42]

As said below, never ascribe to malice that which can be adequately explained by stupidity. (Insert lame joke about MSFT being full of stupidity here).

Yeah, though this particular sort of stupidity has been going on for a long time, and not just at Microsoft (though they seem to be the worst culprit).

I run a couple of sites that, among other things, has links to return the "content" in a list of different formats (GIF, PNG, PS, PDF, ...). Periodically, the servers get bogged down by search sites hitting them many times per second, trying to get every file in every format. The worst cases seem to come from microsoft.com and msn.com, though it happens with other search sites, too. Actually, the first attempts I saw at "deep search" like this came from googlebots around 10 years ago, though they quickly backed off and haven't been a serious problem since then. MS-origin "attacks" of this sort have been happening every few months, for nearly a decade.

I've generally handled them with a couple of techniques. One is to check the logs for successive requests from the same address, and insert sleep() calls with progressively longer sleeps as more messages arrive. The code prefixes the "content" with a comment explaining what's happening, in case a human investigates.

Another technique is to look for series of "give me this in all your output formats" requests, verify that it's a search bot, and add the address to a "banned" list of sites that simply get a message explaining why they aren't getting what they asked for, plus an email address if they want to get in contact. So far nobody at any search site has ever used that address. I did once get a response from a guy who was studying sites with such multi-format data, for a school project, to see how the various output formats compared in size and information content. I took his address off the banned list, and suggested that he add a couple-second delay between requests, and he finished his project a few days later.

I suspect that the googlebot folks may have read my explanation of the delays and added code to spread their requests out over time, since that's what their bots seem to do now. But I never heard from them. They must have gotten complaints (and bans) from lots of web sites when they started doing this, so they probably realized quickly that they should add code to prevent such flooding of sites.

Re:So how do we DDoS Microsoft?

[Alpha830RulZ]

You know, it's easy to poke fun at the Microsofty, but is it possible that he was just trying to find out what was being hit so that he could figure out who in his organization he should contact? Maybe there is some uber technical way he could have figured this out, or maybe he should have RTFB, but his response sounded well intentioned and responsive. What would you prefer? The microsoft of old?

Re:So how do we DDoS Microsoft?

[Jarjarthejedi]

That's not uncommon at all, ever hear of a bug report? Different systems/setups exist everywhere, it's impossible to test your system on all of them, just the most common and wait to hear from people with oddball systems and problems.

Different system's doesn't really apply but what if the site's robots.txt is slightly different (different newlines or something) which is causing an unforeseen error?

Re:So how do we DDoS Microsoft?

[John Hasler]

> ...his response sounded well intentioned and responsive...

Microsoft managers are very good at *sounding* well intentioned and resposive when the shit hits the fan. But why the hell can't they do things right the first time?

Re:So how do we DDoS Microsoft?

[Lonewolf666]

If he was trying to find out what was going on, he was doing it wrong. The CPAN blog probably gives all the information the guys at CPAN have. As outsiders, they probably don't know which department at Microsoft is running those bots. Except that we all can guess at Bing because that is the Microsoft search engine.

As a Microsoft manager in that situation, I'd try to reach someone in Bing network administration first. That someone might not have the tools or network privileges to track down the offending bots himself, but should at least be able to direct the call to the right people.

Re:So how do we DDoS Microsoft?

[MstrFool]

Same reason other folks can't, they are human. Look, I despise MS for a variety of reasons and am one of the rabid anti-MS folks. But honestly, they do enough that is legit to gripe about, no need to blow a mistake like this out of proportion. Considering all they do it was inevitable to happen at some point. Shit happens, any one that codes has had a mega-woops at one point or an other, and if they haven't they they are cookie cutter coding and not risking creativity. Hate them for needlessly locking the geeks from the systems, for locking the owners out of the systems while permitting hackers more remote access rights then they could get at the system it self. But this? 'eh, they goofed, get over it and worry about the real evil they are doing.

Re:So how do we DDoS Microsoft?

[Short Circuit]

The REAL solution to your problem is for everyone to abandon the dumb-as-shite "www" prefix.

Why bother with www.example.com and example.com? Get rid of it. Anyone who still puts "www." on their business cards is a dufus.

REAL solutions to immediate problems don't depend on the rest of the world changing to suit my needs. Also, the fact remains that there are links out there that point to "http://www.rosettacode.org/w/index.php?something_or_other", not all of those links will (or can) change, and I would be an absolute fool to knowingly break them, if I want people to visit RCo via referral traffic.

Re:So how do we DDoS Microsoft?

[raju1kabir]

Different system's doesn't really apply but what if the site's robots.txt is slightly different (different newlines or something) which is causing an unforeseen error?

There is a spec for robots.txt. If someone's not following it, then it's their fault. Given Microsoft's past history, I know where I'd point the finger absent any more concrete information.

Re:So how do we DDoS Microsoft?

[mounthood]

As said below, never ascribe to malice that which can be adequately explained by stupidity.

Must be really easy to just beat you in the face, and say “Ooops, I’m sorry, I’m so st00pid! *drool*” I call bullshit on that rule.

My rule: Don’t make judgements at all (either way), about things that you just don’t know.

How about: Don't mistake organizational stupidity for individual stupidity. This isn't the case of a single bad coder making a mistake, this is an organization that's chosen to how much effort to apply. How much testing and review? What failsafe's, logging and active monitoring? Will options for feedback be accessible and responsive? Stupidity and Malice aren't mutually exclusive for an individual, and certainly not for an organization.

Re:So how do we DDoS Microsoft?

[Chris Burke]

I've never liked that saying because of the implication that malice and stupidity are exclusive.

Dumb and mean are often found together.

Re:So how do we DDoS Microsoft?

[dissy]

Every once in a while, I still see sites that don't serve up unless you include "www." in the address - but it's like I said - a dufus.

Looks like someone hasn't read RFC 1178 and enjoys breaking interoperability.

Your method also breaks email by redelegating MX records one sub domain above where the control should be and MX's point to, thus breaks delegation of sub domains.

Re:So how do we DDoS Microsoft?

[HiThere]

OK, but since I still wouldn't ascribe any trustworthiness to them, I doubt the manager's story. It's not that it's implausible, it's that it's coming from Microsoft.

Put it this way:
If Microsoft said the sky was blue, I'd carry a raincoat.

Re:So how do we DDoS Microsoft?

[davester666]

And now we know why the web-crawlers are so slow to index small web sites for Bing. They're spending all their time crawling open-source web sites in an effort to slow them down.

Re:So how do we DDoS Microsoft?

[dimeglio]

That would be Ken(tm), if I recall correctly, G.I. Joe(tm) was not interested in Barbie(tm).

Re:So how do we DDoS Microsoft?

[mmontour]

Mission accomplished. I got this on the second link that I clicked.

We are currently unable to serve your request
We apologize, but an error occurred and your request could not be completed.
This error has been logged. If you have additional information that you believe may have caused this error please report the problem here.

Re:So how do we DDoS Microsoft?

[HiThere]

It's plausible. But the explanation came from Microsoft, so trusting it isn't reasonable. (Neither is claiming it's a lie. It *is* plausible.)

This is one case where I have to say "Well, they might be telling the truth", and leave it at that.

Still, blocking the addresses seems like the correct move. Even if the truth has leaked out of Microsoft, there's no telling how long it would take them to fix the problem.

Re:So how do we DDoS Microsoft?

[HiThere]

Agreeing with what you said, it's still true that some languages are worse than others (along some particular vector of evaluation). Perl is used to produce more crufty code BECAUSE it's used to quickly hack together solutions.

N.B.: It's not because Perl looks (to me) like line noise. APL is as bad for that (though not worse!). And several other languages are also noted for "write only code". But Perl was created specially for quick hacks. That is both it's power and it's weakness.

Re:So how do we DDoS Microsoft?

[Yakasha]

Clue: Subtle joke, deserves 'funny' moderation ;)

Subtle + Slashdot = FAIL

And what exactly are you hinting at?

Re:So how do we DDoS Microsoft?

[MikeFM]

Yeah I complained about a similar issue with being aggressively scanned by bots that ignored robots.txt and didn't identify themselves with a user-agent and their answer was to first ignore my question and then to almost stop scanning my site altogether. Bing sucks.

Re:So how do we DDoS Microsoft?

[__aaclcg7560]

I thought Ken(tm) was interested in G.I. Joe(tm) these days. :P

Re:So how do we DDoS Microsoft?

[dkh2]

Of course, you presume that the gateway is smart enough to route all traffic to the correct device or sub-domain, and that the under-budgeted admin actually knows how to do that.

I've seen a number of small companies with a very active digital presence for which the owner/president also manages the gateway and has the entire company running on a bank of repurposed workstation towers - each providing a specific service. The gateway box at domain.com doesn't provide anything but traffic cop services. The system named 'WWW' provides ONLY the HTTPd service. Likewise separate boxen provide POP, SMTP, etc...

In these cases the domain.com gateway is a bare bones implementation and actually requires the additional information to route requests correctly.

Re:So how do we DDoS Microsoft?

[gbjbaanb]

IP addresses aren't enough? You're MS--if you can't fix the problem and IP addresses are given, damn, that's just sad. You're freaking massive multi-billion dollar tech companies, and this is the best you can do?

I've seen and used Vista. The answer to your question is "yes".

Re:So how do we DDoS Microsoft?

[Anonymous Coward]

Never ascribe to malice and stupidity what can be explained by stupidity alone.

 

That better?

Re:So how do we DDoS Microsoft?

[Anonymous Coward]

Hey, great, sexism.

Re:So how do we DDoS Microsoft?

[budgenator]

The unobtainable fruit is always thought to be the sweetest.

Re:So how do we DDoS Microsoft?

[budgenator]

The http: part does make the www. part redundant.

Re:So how do we DDoS Microsoft?

[Fzz]

• Step 1. Turn on HTTP Compression on your server (msnbot supports it).
• Step 2. Write a little cgi script that checks if the agent is msnbot, and if so for every image on your web site, returns a really really large file of zeros. It wont cost you any bandwidth because gzip will compress all the zeros to very little for transmission.
• Step 3. Invest in shares of Seagate and Western Digital. Short Microsoft.
• Step 4. Profit!

Re:So how do we DDoS Microsoft?

[budgenator]

As for messing up the whole sub-domain naming scheme, please clarify?

Well let see Example Corp is a mega-sized multi-nation with a physical presence in numerous countries providing goods and services; it's global web portal is example.com, it's web portal for American operations is usa.example.com, Mexican operations site is mex.example.com and Swiss operations is at che.example.com. Putting an "www." on the front just complicates things a smidgen, but it's not a deal breaker.

Re:So how do we DDoS Microsoft?

[Anonymous Coward]

Even professionals.

You're implying "professionals" work there? Ha, ha ha. Ignoring robots.txt, particularly with the extraordinary resources they have to get it right, is incompetence, not professionalism.

Re:So how do we DDoS Microsoft?

[Passman]

Nah, G.I. Joe was interested in G.I. Joe these days. But don't bother asking, he won't tell.

Re:So how do we DDoS Microsoft?

[dissy]

You've made the classical mistake of mistaking the name of a thing with the thing itself.

The name of my web server is not 'www'. The machine has a real name, and www is a cname (Alias) to that.

In reality, for those of us that have been using the Internet since before 1992 and the web existed, some of us still run many other services than web servers, and some of them for a lot longer than web servers existed.
Why single out a relatively new service to hand the root of your domain over to?

Maybe 'example.com' points to my mail server, because I am an email company. That means I must use a subdomain, and it must be one my visitors KNOW IN ADVANCE.

This means 'screwball.example.com' would be unreachable to anyone I did not tell the computers name to ahead of time, since it would not be possible for just 'example.com' to redirect to the web server, being on a different machine than the mail server it already points to.

This means going to 'example.com' would end up TCP wise at my mail server, and 'www.example.com' would not exist. How would anyone ever find my web server?

Welcome to networking 101

In addition, if you want to base your case off of that logic, then OK.
If you think 'www' is the machines name out of 'www.example.com', then your solution of using just 'example.com' means your machine is named 'example' and part of the network 'com'.

That makes even less sense. (At least for all of us that are not ICANN/NetSol)

It's 2 decades out of date, and more importantly, specifically states It does not specify any standard.

Let's see what I said (Since you clearly didn't)

Looks like someone hasn't read RFC 1178 and enjoys breaking interoperability.

No, I don't see the word standard anywhere in there.

I see 'enjoys breaking interoperability', and following that I see a very precise example, which you ignored.

Re:So how do we DDoS Microsoft?

[merreborn]

Also, the fact remains that there are links out there that point to "http://www.rosettacode.org/w/index.php?something_or_other", not all of those links will (or can) change, and I would be an absolute fool to knowingly break them, if I want people to visit RCo via referral traffic.

That can be resolved with a single, simple apache rewrite rule.

Continuing to support www. -- if only by rewrite rule -- is unfortunately a necessary evil presently. If it isn't "www.*.com", the technically unsavvy majority doesn't understand it.

Re:So how do we DDoS Microsoft?

[31eq]

I don't see any need for exceptions because I can't find anything in RFC 1187 that we'd need to make an exception to.

I don't see any problem with MX records either. MX records exist precisely to solve this problem. The DNS tells you which server to send email to. That is, it delegates to a sub domain, if that's what you want. Nothing you do on your web server will break email.

Now, you may have services other than web and email, yes. And you may have some problem with routing by protocol. But, given that it isn't 1992 any more, and this web thing doesn't look like a passing fad, it makes sense to use the shortest domain name for the most used protocol. That does mean that the vast majority of "www" prefixes are completely redundant. But, hey, aren't people funny things?

Re:So how do we DDoS Microsoft?

[Eil]

Mod parent up. Any datacenter worth their salt has a way of blacklisting IPs at the router if a DoS can't be stopped at the server (although it honestly sounds like they didn't try that approach either).

Re:So how do we DDoS Microsoft?

[lena_10326]

I run a couple of sites that, among other things, has links to return the "content" in a list of different formats (GIF, PNG, PS, PDF, ...). Periodically, the servers get bogged down by search sites hitting them many times per second, trying to get every file in every format.

I don't understand why you exposed them as links in your html, which crawlers will easily pickup. If bandwidth/CPU is a concern (it almost always is) one shouldn't serve plain links to large data files; files with many duplicate formats; or files meant only for human consumption. Is there a reason you don't serve them from CGI with mandatory POST arguments? You turn off indexing on the data file directory to lock out crawlers and use the CGI to validate POST arguments, which serves a 302 redirect link to the actual data file. It can be done with GET and mandatory args but going with POST is gives you an extra layer because convention is crawlers don't do form POSTs. You can also throw in a CAPTCHA on the form page.

Inserting a form POST does throw up an extra page, but it's become a sort of status quo for user initiated downloads. Some nanny HTTP purists might bitch about misusing POST, but it's simple way to do it and can execute very fast with caching compiled CGIs because there's little CPU overhead with parsing CGI args and sending back a few headers.

As for automated downloads (invoked from installer programs), they should be using file manifests to locate files inside the non-indexable data directory.

Re:So how do we DDoS Microsoft?

[Al Al Cool J]

Why single out a relatively new service to hand the root of your domain over to?

Because that is the service that all of your internet-using customers will use to seek information about your company.

Maybe 'example.com' points to my mail server, because I am an email company.

Then that would be a stupid email company and deserves to go out of business.

I'm sorry, but if http://example.com/ does not bring up your company's website, then you are a dismal IT failure, and no amount of rationalisation or waving RFCs about will change that.

I understand and appreciate that there is often perceived to be a "right way" to do things in IT, but you still have to balance that against common sense, practical considerations, and user expectation. The "right way" may be right when seen within a specific and confined logical framework (networking 101), yet be completely moronic when placed within a broader context (business and marketing on the internet).

Re:So how do we DDoS Microsoft?

[fm6]

never ascribe to malice that which can be adequately explained by stupidity.

You'll pry my conspiracy theory from my cold dead hands!!!!!

Re:So how do we DDoS Microsoft?

[tomhudson]

The http: part does make the www. part redundant.

Thank you!

Reserve the extra stuff for subdomains. like blog.example.com, clients.example.com, specials.example.com, ads.example.com (so we can block that one more easily &lt:-0

We already know by the $PROTO http:/// portion that it's web traffic, not ftp or ssh or telnet or ...

Re:So how do we DDoS Microsoft?

[Chris Burke]

No, because you're still suggesting to discount malice in the presence of stupidity.

"If it's possible it could be a case of stupidity alone, assume there's no mal intent involved," is a bad assumption in many cases, and thus dumb.

Re:So how do we DDoS Microsoft?

[NNKK]

I encourage you to explain to people serving large quantities of content that in order to be in compliance with your personal view of network correctness, their routers must now perform NAT on all of their traffic.

I assure you, you will not get the hoped-for response from people saturating 10gbps uplinks.

Re:So how do we DDoS Microsoft?

[Emilio III]

This whole thing appears to be a fraud. The IP address range 66.55.96.0 - 66.55.111.255 belongs to Funds Xpress Financial Network, Inc. in Austin TX. 66.55.192.0 - 66.55.223.255 belongs to Great Works Internet of Biddeford ME. Who decided those IP addresses belong to Microsoft? Please check whois.arin.net

Re:So how do we DDoS Microsoft?

[spongman]

let's hope they don't store it compressed...

Re:So how do we DDoS Microsoft?

[drinkypoo]

Instead we have Slashtroglodytes screaming about conspiracies by MSFT.

Just for the record, since you're commenting under a thread I started, I do not believe that there was a conspiracy to attack CPAN. I think there is a conspiracy to continue accidentally attacking CPAN. The information provided ought to be more than sufficient to figure out what is going on. Remember, any time two people work to screw a third out of something, it's a conspiracy by definition.

Re:So how do we DDoS Microsoft?

[darthflo]

Maybe 'example.com' points to my mail server, because I am an email company.

Out of all the examples in the world you could pick, you went for the wrong one like a cartoon character falling into the desert and hitting the only cactus in a three-mile radius. There's a DNS record type called MX to identify Mail eXchanges for that domain. example.org. A may point to 10.2.3.4, which could be your web, telnet, irc and quake server yet example.org. MX would point to pizza.example.org. (which in turn has an A record to 10.4.5.6) and spaghetti.example.org. (10.1.1.1). You can even add several MX records with different priorities, so in the event of pizza failing, clients will try spaghetti. It's quite awesome.

Re:So how do we DDoS Microsoft?

[bcrowell]

I run a couple of sites that, among other things, has links to return the "content" in a list of different formats (GIF, PNG, PS, PDF, ...). Periodically, the servers get bogged down by search sites hitting them many times per second, trying to get every file in every format.

I've had sort of a similar issue, not with bots but with things known as "download managers" (example) Apparently people install a plugin in IE that is supposed to make their downloads go faster. If I'm understanding correctly, it opens up multiple http connections in order to retrieve the same file. I suspect it's basically snake oil. I suppose it might help in cases where the bottleneck isn't your own ISP but the overloaded server on the other end, although then you'd essentially be screwing the other users on the site in order to get more than your fair share. My site has a lot of books that are in the form of large PDF files. I'll get these users hitting my site, and it utterly brings my server to its knees. My apache logs show these people using up 50 Mb worth of data flow in order to download a 5 Mb pdf file. The only solution I've been able to find is to write a perl script that goes through my logs every 15 min looking for this pattern of usage. When it detects it, it writes to the .htaccess file to block that IP.

Re:So how do we DDoS Microsoft?

[budgenator]

Dude you are so dissing gopher

Re:So how do we DDoS Microsoft?

[bluefoxlucid]

Actually, you'd just have to run a little HTTP server on that box that replies with an HTTP 310 Moved Permanently to www.example.com

Re:So how do we DDoS Microsoft?

[tomhudson]

Most people aren't saturating 10gps uplinks. Also, most people ARE doing NAT anyway. So what's your point?

Re:So how do we DDoS Microsoft?

[marcosdumay]

Maybe... He got all the trouble of separating the punctuation from the adress, he could at least put a few href="mailto:..." tags in it. No, doesn't deserve the moderation :p

Oh! *Literally* Microsoft bots!

[Culture20]

Until I read the summary I thought it was another article about windows botnets and was wondering why the "microsoft" was tacked on since windows is the default OS assumption. Of course it would be interesting if these were new CPAN mirrors that MS was settings up.

Re:Oh! *Literally* Microsoft bots!

[Ardaen]

Probably not, if you look at other incidents: http://cmeerw.org/blog/594.html it appears they just like to push the limits.

Re:Oh! *Literally* Microsoft bots!

[Trailer Trash]

Until I read the summary I thought it was another article about windows botnets and was wondering why the "microsoft" was tacked on since windows is the default OS assumption.

I'm not sure these are mutually exclusive.

Testers blog link...

[flyingfsck]

Sooooo, lets all go to the testers blog and DDOS that too. Dumbass...

Re:Testers blog link...

[nicolas.kassis]

If he can handle the msnbots, he probably can handle the slashdot crowd.

I've seen it before

[LordAzuzu]

I manage some networks in my home city in Italy, and in the past year I've often seen strange traffic coming from some of their IP addresses. Guess they have been exploited by someone long time ago, and didn't even notice it.

Re:I've seen it before

[beadfulthings]

It's interesting to read this, as I've had some random and somewhat incomprehensible port scans coming from an IP address identified as one of theirs. If you're just an insignificant slob, you can't write to their abuse address, either; you'll get bounced. I simply blocked that particular IP address. Let them worry about who's gotten to them.

Re:Typical M$

[auric_dude]

Sounds like Microsoft.CN to me.

Check the blog...

[strredwolf]

Looks like Microsoft's Bing managers are on it. They'll make it worse in no-time flat. :)

BTW, the difference between a DDOS and a Slashdotting? You know why your site went down -- you got linked!

Re:Check the blog...

[Anonymous Coward]

BTW, the difference between a DDOS and a Slashdotting?

The DDOS bots actually read TFA.

Re:Check the blog...

[gothzilla]

They're not "on it." They admitted they were powerless to solve their own problems without help from their victims.

Re:Check the blog...

[Anonymous Coward]

Seems like they read everything but robots.txt.

Re:Check the blog...

[jc42]

They admitted they were powerless to solve their own problems without help from their victims.

Heh. It's another "damned if you do; damned if you don't" scenario. Usually, people criticise Microsoft for developing software without bothering to consult or test with actual customers. Now we have a manager of a MS dev group that actually does communicate (though not exactly with "customers"), and acts on what they say, so he's criticised for needing help from his "victims".

Ya can't win that game.

But the fact is that if you're developing server-side web software, you need to test it against real-world sites, not just the toy sites you've set up in your lab. And we all know the "Sourcerer's Apprentice" sort of bug that produces a runaway test that tries to do something as many times as it can per second until it's killed. Good testers will be on the lookout for such events, but it's understandable that they might fail occasionally

Among web developers, MS does have a bit of a reputation for hitting your new site with a flood of requests, trying to extract everything that you have (even the content of your "tmp" directory which your robots.txt file says to ignore). There are lots of small sites that block MS address ranges for just this reason.

It should be considered good news that there's at least one MS manager who understands all this, and is willing to talk to the "victims" and fix the problems. Now if they could fix the next-level problem, that this sort of thing happens repeatedly and their corporate culture seems to have no way to prevent it from happening again.

Re:Check the blog...

[schon]

They admitted they were powerless to solve their own problems without help from their victims.

Heh. It's another "damned if you do; damned if you don't" scenario.

Un, no. Not unless you're a rabid MS apologist.

Usually, people criticise Microsoft for developing software without bothering to consult or test with actual customers.

True.

Now we have a manager of a MS dev group that actually does communicate (though not exactly with "customers"), and acts on what they say, so he's criticised for needing help from his "victims".

Umm, exactly how did he act on what they said? According to the quote, they explicitly didn't act, which is the problem people are complaining about.

What's not?

[tjstork]

It's not like ASP.NET is the most efficient way to sling web pages to being with.

MS ineptitude?

[Anonymous Coward]

From TFA:

Hi,
I am a Program Manager on the Bing team at Microsoft, thanks for bringing this issue to our attention. I have sent an email to nospam@example.com as we need additional information to be able to track down the problem. If you have not received the email please contact us through the Bing webmaster center at nospam@example.com.

I mean, what additional information is needed wrt "respecting robots.txt" and "not letting loose more than one bot on a site at a time"?

Bing. Meh.

Re:MS ineptitude?

[Anonymous Coward]

It kind of depends on the individual robots.txt. Google, for instance, added a bunch of extended rules that they respect but which aren't officially part of the robots.txt spec (which is pretty limited). If they've added some of those rules in it could be that it's failing to validate when the MS bot hits it and therefore being ignored.

Re:MS ineptitude?

[ShecoDu]

I remember reading that the MSNBOT reads the "Robots.txt" file, but cpantesters has a lowercase filename:

http://static.cpantesters.org/robots.txt

http://static.cpantesters.org/Robots.txt doesn't exist, so basically MSNBOT only respects the robots.txt on case insensitive operating systems.

Re:MS ineptitude?

[John Hasler]

The standard clearly specifies lower case. However, if you are correct there's a simple way to send bingbots one way and all other bots another: create Robots.txt and robots.txt with different contents.

Re:MS ineptitude?

[gbjbaanb]

I wonder if it has something to do with fixing this

We asked Microsoft how it was planning improve Bing's indexing problem. "We're always working to improve the crawler," a Microsoft spokesperson told Ars. "With our latest crawler release still in beta, we doubled our crawling capacity worldwide. We increased our sitemap URL size to 50K and we made it easier for webmasters to control the crawler's aggressiveness."

Re:MS ineptitude?

[godless dave]

I mean, what additional information is needed wrt "respecting robots.txt" and "not letting loose more than one bot on a site at a time"?

To begin with, you would probably have to explain what robots.txt is.

Re:MS ineptitude?

[PAjamian]

They'll probably be asking specific questions, such as, "can we get a copy of your log entries so we can match the IPs and times to our own logs and see why this is happening?"

Re:MS ineptitude?

[Fnord666]

I remember reading that the MSNBOT reads the "Robots.txt" file, but cpantesters has a lowercase filename:

This is not likely to be the cause. In the article the author states that "It seems their bots completely ignore the rules specified in the robots.txt," and that "I know this because I can see the IP addresses in the logs. ", one would have to assume that cpantesters have reviewed their logs and that they would have noticed if robots.txt was not being returned.

Probably just a bug.

[tjstork]

I know everyone likes to assume that Microsoft is being evil here, but wouldn't the more realistic assumption be that they were just being incompetent?

Re:Probably just a bug.

[Lloyd_Bryant]

I know everyone likes to assume that Microsoft is being evil here, but wouldn't the more realistic assumption be that they were just being incompetent?

Sufficiently advanced incompetence is indistinguishable from malice. For additional examples, see Government, US.

The simple fact is that ignoring robots.txt is effectively evil, regardless of the intent. It's not like robots.txt is some new innovation...

Re:Probably just a bug.

[fish waffle]

I know everyone likes to assume that Microsoft is being evil here, but wouldn't the more realistic assumption be that they were just being incompetent?

Probably. But since incompetence is the plausible deniability of evil it's sometimes hard to tell.

Re:Probably just a bug.

[mspohr]

Occam's razor (or Ockham's razor[1]), entia non sunt multiplicanda praeter necessitatem, is the principle that "entities must not be multiplied beyond necessity" and the conclusion thereof, that the simplest explanation or strategy tends to be the best one.

Rough translation: "Never ascribe to malice that which can be adequately explained by stupidity."

Re:Probably just a bug.

[alexhs]

these bots 'completely ignore the rules specified in robots.txt.'

Microsoft ignoring standards is not incompetence, it's policy (NIH syndrome).

Re:Probably just a bug.

[djupedal]

> "I know everyone likes to assume that Microsoft is being evil here, but wouldn't the more realistic assumption be that they were just being incompetent?"

We assume MS is evil...

We know they are incompetent.

We feel this is typical.

We pray they'd just go away.

We think this will never end...

Re:Probably just a bug.

[gmuslera]

They are not ignoring robots.txt, probably just that they understand that file in their slighly different, but in the end incompatible, format. As every other file.

Re:Probably just a bug.

[Yvanhoe]

There is such thing as criminal incomptence. If a script kiddie can be arrested for having a virus "out of control" I don't see why Microsoft engineers DDOSing a website couldn't be charged.

By the way a philosopher once told that "evil" did not exist. That it was most of the time just a kind of hidden stupidity.

Re:Probably just a bug.

[MrMr]

The problem is, there is no evidence that:
Never ascribe to stupidity that which can be adequately explained by malice.
Is invoking more entities.
In fact, claiming that the commercially most successfull software company got there through stupidity rather than malice sounds extremely implausible to me.

Re:Probably just a bug.

[ztransform]

The simple fact is that ignoring robots.txt is effectively evil, regardless of the intent. It's not like robots.txt is some new innovation...

Since when did Microsoft feel existing standards were something to honour? How many times have its browsers changed behaviour? Re-defined entrenched URL standards (you cannot specify username/password in an Internet Explorer URL but this is a legal standard form of URL)?

It stands to reason Microsoft would take no notice of anything your website has to say.

Unless.. of course.. Microsoft define a certificate type that can sign your Microsoft-specific format exception list after payment on an annual licensing basis..

Oh hey, another Microsoft example: Vista! After all, why assume someone upgrading their operating system might expect the same if not better!

PS see http://support.microsoft.com/kb/834489

Re:Probably just a bug.

[Lundse]

That's a pretty rough translation!

You might be able to argue, that the latter saying is a corollary of the former, but in no way do they mean the same.

Occam says the simplest explanation is best - the better explanation is the one with least assumptions.

In this case, Occam affords us no help - we already know MS is both "evil" and incompetent. So the two explanations are equal in this regard. The "corollary" suggests, then, something else; namely that stupidity is a better explanation than "evil" in all/most cases (presumably because stupidity is more widespread).

Re:Probably just a bug.

[Rogerborg]

You're probably new here, but if you'd RTFA, you'd see that:

It seems their bots completely ignore the rules specified in the robots.txt, despite me setting it up as per their own guidelines on their site

Come to think of it though, isn't this what happens to most people who try to interoperate with Microsoft?

Amusingly, if I Google for "bing robots.txt" I get a link to a bing page titled "Bing - Robots.txt Disallow vs No Follow - Neither Working!" which has already been elided from history by Microsoft. CLassy.

Re:Probably just a bug.

[Anonymous Coward]

Excuse my ignorance, but isn't robots.txt compliance easily enforceable on the server? I remember something about hiding links to trap pages in order to indentify robots and then holding identified robots responsible for robots.txt infractions by blocking their IP address.

Re:Probably just a bug.

[drspliff]

Well, the last I heard Bing spider was looking for `Robots.txt` rather than `robots.txt` which would explain the file being "ignored" in this case.

Re:Probably just a bug.

[paiute]

I know everyone likes to assume that Microsoft is being evil here, but wouldn't the more realistic assumption be that they were just being incompetent?

Probably. But since incompetence is the plausible deniability of evil it's sometimes hard to tell.

"incompetence is the plausible deniability of evil"

  fish waffle, that is great sig material.

Re:Probably just a bug.

[Suki I]

Try saving a copy as robots.docx and see if that works ;)

Re:Probably just a bug.

[maxwell demon]

In fact, claiming that the commercially most successfull software company got there through stupidity rather than malice sounds extremely implausible to me.

So if certain Microsoft products are or were insecure and/or unstable, it wasn't incompetence, but malice? You think Microsoft was happy every time a user got the dreaded Blue Screen Of Death?

Re:Probably just a bug.

[init-five]

I know everyone likes to assume that Microsoft is being evil here, but wouldn't the more realistic assumption be that they were just being incompetent?

how about both?

Re:Probably just a bug.

[Opportunist]

Like my grandpa said, it doesn't matter how dumb you are. As long as you find someone even dumber to sell to.

Re:Probably just a bug.

[Xest]

Yes, and I like the solution too- rather than contact Microsoft to find out what the fuck is going on, post it to Slashdot and get Slashdotted as well.

Pure genius.

Re:Probably just a bug.

[afidel]

I wonder if it's a CR/CRLF bug =)

Re:Probably just a bug.

[horatio]

You think Microsoft was happy every time a user got the dreaded Blue Screen Of Death?

Yes, in a way. I never really thought about it until you asked, but it fits with their business model of forcing users into an expensive upgrade of their OS every few years. Look what has happened with XP. It doesn't blue screen [as] much, and they've met heavy resistance from folks not wanting to upgrade to Vista. (Never mind that Vista is crap.) So now they've re-packaged Vista as "Windows 7" and hope folks don't realize it looks the same and smells the same, because it basically is.

Re:Probably just a bug.

[hairyfeet]

But MSFT is a corporation, which thanks to our corporate butt kissing congress and courts can just go "ooopsie", maybe cut a small check at most, and walk away scott free.

And as for your philosopher? I saw an interview with Joss Whedon on writing evil characters that I thought really hit the nail on the head. He said, and I paraphrase "The villain never sees himself or herself as evil. To them there is a perfectly justifiable reason for their actions. I have known some truly evil people, those that have intentionally hurt their fellow man out of pure malice, and to them their actions were justified and noble. They simply didn't see what they did as wrong."

Which is how you get MSFT and Intel paying backroom deals to crush competition, or Jack Trammell and his "business is war" philosophy. To the ones making the decisions "the other guy would do it to us if they could, so why shouldn't we do it to them?". I'm sure that if you talked to Gates or the head of Intel you could never get them to believe that crushing your competition any way you can is wrong. To them that was/is business 101 and not evil. That is why I think Whedon was right, the villain always thinks they are noble.

Re:Probably just a bug.

[schon]

It has nothing to do with the RTFA.

their own guidelines on their site

As anyone who has ever read MS documentation can tell you, you need to read it, then implement a test, so you can see what it really expects, then adjust your test, then try it until it works.

Their problem is that they expected MS documentation to actually describe the expected behaviour.

Re:Probably just a bug.

[MrMr]

Sort of:
I'm saying that the assumption that these flaws persist through incompetence is not a less complex explanation.
The fact that issues were not solved in one of their later releases may very well be a deliberate commercial decision, which would make it indeed malicious rather than incompetent from the end-user perspective.

Re:Probably just a bug.

[CrazyDuke]

Something that bugs me about that statement: Out of curiosity, since when does a lack of evidence amount to an adequate explanation?

And, also, how does malicious incompetence fall under that false dichotomy? Or, for that matter, what of reckless incompetence and plausible dependability?

Oh, and for the record: Experience tells me such an outcome is often the result of a PHB or two and a few "I don't give a fuck anymore." engineers. It's fun to dismiss PHBs as merely incompetent. But, what they are competent in is convincing people their actions warrant promotion, regardless of the actual results of their actions.

Re:Probably just a bug.

[PinkyDead]

Microsoft don't have any tools that can effectively read that format.

Re:Probably just a bug.

[CFBMoo1]

Your links look odd.

Google: http://www.google.co.uk/search?q=bing+robots.txt
Bing: http://www.bing.com/community/forums/t/647019.aspx

My Bing: http://www.bing.com/search?q=bing+robots.txt

I get "Bing Not Honoring Robots.txt Directives?" as the second hit on Bing. The first is their own site which kind of makes sense since Bing is the first search term I used.

Re:Probably just a bug.

[Goaway]

I'm sure you heard that, but it's not actually true in any way.

Re:Probably just a bug.

[blueZ3]

What's amusing about the issue in the kb is that the problem that they're "solving" by breaking the username/password in a URL standard is NOT a problem with username/password URLs, but a problem with how IE displays the URLs. In other words, rather than fixing the behavior of IE's address and status bars to display such URLs correctly, they just stopped supporting them.

Incompetence at that level isn't just indistinguishable from malice, it IS malicious.

Re:Probably just a bug.

[blueZ3]

For the sake of argument...

Wouldn't you say that in most cases malice implies more complexity? For example, it only takes one stupid mistake by a coder to introduce a bug, whereas intentional introduction of flaws for some sort of business purpose supposes a concerted effort by a group?

Just asking :-)

Re:Probably just a bug.

[Pharmboy]

robots.rtf?

Re:Probably just a bug.

[MrMr]

Good thing there's a google-cache. Now if we only need a Bing-cache where Googles whitewashing is archived...

Re:Probably just a bug.

[mR.bRiGhTsId3]

That would be tremendously amusing. I can see the headline now. Bing robots DDoS attack every Unix hosted site by assuming Windows linefeeds.

Re:Probably just a bug.

[Pharmboy]

Wow, you must be new....to computers. I particularly liked you comment "A site could have quality links to non ignore sites." as justification for a bot to ignore robots.txt. Can I have your AOL email address so I can write you personally?

Re:Probably just a bug.

[b1t r0t]

What exactly do you mean by "elided from history"? I brought them both up, turned off the CSS (Google's version is broken), and tab-flipped betwen them. Not only is the page still there, it has all the same posts as the Google cache version, with small differences such as tags switching around, number of posts by users, and another stupid Blackpool adlink. Maybe you found some messages missing and then Google later re-cached it, but the thread itself is certainly not missing.

Re:Probably just a bug.

[Hurricane78]

The most realistic thing to do, would be to not make any stupid assumptions at all, about things that you know nothing about.

But who cares for actualy facts, nowadays, right? As long as you strongly prescribe to a side... “doesn’t matter which, as long as it’s mine!” ...you’re good. Right. :(

This world depresses me.

Re:Probably just a bug.

[ckaminski]

I think his point is that if you look at the Google Cache and the actual page on bing.com, the entire discussion is gone. Just the original question. The rest, poof gone!

Re:Probably just a bug.

[StuartHankins]

As anyone who has ever read MS documentation can tell you, you need to read it, then implement a test, so you can see what it really expects, then adjust your test, then try it until it works.

Mod parent up. I thought it was just me...

Re:Probably just a bug.

[b1t r0t]

I can't tell whether you're being ineptly sarcastic or really that stupid. The main purpose of robots.txt is to keep web spiders (aka "robots") from getting stuck in a tarpit of script-generated pages which are not only redundant but waste resources of the website, possibly bringing it to its knees. For instance, something like a button that says "full view" that shows the same page with more fancy formatting.

What it's not for is hiding stuff from view, because anybody can look at your robots.txt file and see that you have a /secret/ path in your web site. Yes, this actually happens, and people actually do find the secret information and have fun scattering it across the internets.

Re:Probably just a bug.

[Yvanhoe]

A pedophile that doesn't understand his pulsions and do not interest in existing treatments, who believes sexual drive come from the devil and are a challenge to his own faith, yes act more out of stupidity than of "evilness".

I once heard the story of the first serial killer arrested thanks to psychological profiling (someone who murdered old women and mutilated them). Do you know what was his reaction when he heard about how he was found ? He wanted psychological help. Until then he did not understand he was struggling with abnormal urges.

Re:Probably just a bug.

[darkpixel2k]

robots.rtf?

No, it's robots.wmf. Just take a screenshot of the URL you don't want them accessing...

Re:Probably just a bug.

[catman]

So if certain Microsoft products are or were insecure and/or unstable, it wasn't incompetence, but malice? You think Microsoft was happy every time a user got the dreaded Blue Screen Of Death?

Of course. "Oh, that's fixed in the next version, please upgrade. That'll be $nn, thank you. " Ka-ching!

Re:Probably just a bug.

[AnonymouseUser]

> We pray they'd just go away.

Therein lies the problem, and is why they never will go away. Instead of hoping/praying they go away, I do everything I can to make them go away. IOW, I avoid their products every chance I get, and recommend others use non-MS products whenever I can.

Re:Probably just a bug.

[tomhudson]

That's because they're not at version 3 of whatever they're working on that did the DDoS.

Either it's prior to version 3, in which case it should be labeled "beta"

Or it's after version 3, in which case it should be labeled "bloatware".

Of course, emailing a site you're accidentally DDoSing, you'd better hope their email server is on another machine ...

Re:Probably just a bug.

[dwiget001]

Which, for me, has been my biggest pet peeve in relation to their "Help" files and similar documentation.

They are, for the most part "not helpful", thank you very much.

Re:Probably just a bug.

[Silverlock]

It's not terrorism if you have a flag and it's not computer theft if you have a brand name.

Re:Probably just a bug.

[Jah-Wren Ryel]

I can't believe you are the ONLY person to point out that the guy took an obvious MS joke waaaay too seriously and then some dumbass mod came along and gave you a redundant - what the hell?

Re:Probably just a bug.

[rrohbeck]

You got it backwards :)

Re:Probably just a bug.

[metamatic]

Re-defined entrenched URL standards (you cannot specify username/password in an Internet Explorer URL but this is a legal standard form of URL)?

HTTP URLs never supported username and password in the URL, according to the actual standards. RFC 1738 was the original URL specification. Section 3.1 said that some schemes supported username (and/or password) in the URL, giving the example of ftp urls. However, http was not one of the schemes supporting usernames or passwords, as you can see from the syntax description in section 3.3. None of the followup RFCs added user or password support to http URLs. In fact, RFC2396 noted in section 3.2.2 that the feature was not recommended even when it was supported. RFC3986 then deprecated the feature, even for ftp URLs. So user and password in http URLs was a non-standard feature Microsoft should never have implemented in the first place, and they were right to remove it. As far as I know, the only URL scheme which still officially supports username and password without deprecation is telnet, presumably on the grounds that anyone still using telnet doesn't care about the username and password being hacked anyway.

Re:Probably just a bug.

[epine]

The main purpose of robots.txt is to keep web spiders (aka "robots") from getting stuck in a tarpit of script-generated pages

You mean provocating cause at point of consensus. Once something becomes ensconced as a facility of the commons, it's purpose takes on a life of its own, as practiced in the large.

Off-label use

Robots.txt also serves as a sentinel for which parties on the net are playing ball, but obeying robots.txt, and who is operating outside the bounds of conformity, DoSing whatever they please.

Re:Probably just a bug.

[metamatic]

What's amusing about the issue in the kb is that the problem that they're "solving" by breaking the username/password in a URL standard is NOT a problem with username/password URLs, but a problem with how IE displays the URLs.

No, it's much more than a display problem. URLs get cached. They end up in history files, downloaded files, cache files on proxies, log files, everywhere. You can't guarantee that all the software out there is going to dispose of URLs securely or sanitize out usernames and passwords, so it would not have been safe to put usernames and passwords in URLs even if Internet Explorer had taken pains to avoid information leakage.

Re:Probably just a bug.

[LihTox]

The villain never sees himself or herself as evil. To them there is a perfectly justifiable reason for their actions.

Add to that the phenomenon of emergence, where the whole is different from the sum of its parts. An organization can be evil even if its members are not.

Re:Probably just a bug.

[binaryspiral]

try robots.wtf

Re:Probably just a bug.

[Lundse]

Hm... Interesting.

Quite possibly, though it would be different for different domains. When it comes to bugs in software, obviously! To avoid Godwins law by a few borders, lets take the genocide in Rwanda as something which it would be pretty hard to explain by stupidity (without waxing philosophical).

So I would still not say that "stupidity is a better explanation than malice" is implied by Occams Razor, and certainly not that they mean the same.
But in certain domains, it could be a corollary...

Re:Probably just a bug.

[Suki I]

+1

Re:Probably just a bug.

[TheSpoom]

That's not a bug, that's by design. How else are they supposed to only DDoS Unix and Mac servers and leave Windows servers alone?

Re:Probably just a bug.

[watergeus]

"By the way a philosopher once told that "evil" did not exist. That it was most of the time just a kind of hidden stupidity."

Who was that?

Re:Probably just a bug.

[mgblst]

It seems they are looking for robots.docx instead. It is really the admins fault, for not converting the file to the accepted way.

Re:Probably just a bug.

[Lotana]

If there was a way to mod a comment to +10 Insightful: This post is it.

Thank you.

Re:Probably just a bug.

[aralin]

I think this thread is slowly approximating to the correct robots.wtf?

Re:Probably just a bug.

[aralin]

There should be a way to mark posts as favorite or submit to hall of fame of comments or something. This comment spot on describes my relationship with Microsoft during the last 14 years.

Fixing Bing's poor indexing

[AHuxley]

Its not a bug, its a feature to index a site with a new, rapid, powerful, direct, personalised crawler :)
http://arstechnica.com/microsoft/news/2010/01/microsoft-outlines-plan-to-improve-bings-slow-indexing.ars

This is a normal occurence for Bing

[Anonymous Coward]

I had a registration page - static content basically. The only thing that was dynamic was that it was referred to by many pages on the site with a variable in the querystring. Bing decided that it needed check on this one page *thousands* of time per day.

They ignored robots.txt.
I sent a note to an address on the Bing site that requested feedback from people having issues with the Bing bots - nothing.

The only thing they finally 'listened' to was placing "" in the header.

This kind of sucked because it took the registration page out of the search engines' index, however it was much better than being DDOS'd. Plus, the page is easy to find on the site so not *that* big a deal.

Bing has been open for months now and if you search around there are tons of stories just like this. Maybe now that a site with some visibility has been 'attacked', the engineers will take a look at wtf is wrong.

Re:This is a normal occurence for Bing

[The Cisco Kid]

Seems like a better solution would have been to setup a test for the either the User-Agent, or the IP/blocks that Bing was attacking your site from, and dropping those requests in /dev/null - your site would still exist on 'real' search engines, and Bing doesn't pound on your bandwidth anymore.

Re:This is a normal occurence for Bing

[The Cisco Kid]

Replying to myself: if testing the UA or the IP in the httpd itself was too much load, you could have also just nullrouted the IP blocks the Bing spider was coming from, either in the kernel table, or in your router.

Re:This is a normal occurence for Bing

[dkf]

Replying to myself: if testing the UA or the IP in the httpd itself was too much load, you could have also just nullrouted the IP blocks the Bing spider was coming from, either in the kernel table, or in your router.

I know of one site where this has been done for years (both with Bing and its predecessors). Sure it ruins the site's searchability for anyone using Bing, but like we care; that's better than having the site itself unreachable due to load and Google doesn't cause the same level of problems.

Flooding...

[Bert64]

I have noticed the microsoft crawlers (msnbot) being fairly inefficient on many of my sites...
In contrast to googlebot and spiders from other search engines msnbot is far more aggressive, ignores robots.txt and will frequently re-request the same files repeatedly, even if those files haven't changed... Looking at my monthly stats (awstats) which groups traffic from bots, msnbot will frequently have consumed 10 times more bandwidth than googlebot, but is responsible for far less incoming traffic based on referrer headers (typically 1-2% of the traffic generated by google on my sites).

Other small search engines don't bring much traffic either, but their bots don't hammer my site as hard as msnbot does.

Re:Flooding...

[Manfre]

Did you provide google with a sitemap file? If so, that explains why google does not need to check your site for changes as often.

Re:Flooding...

[Bert64]

I have 2 sites with sitemaps, but they were not the ones i was looking at as there are many more sites on the server.
That also wouldn't explain why search engines other than msn don't hammer the site.

Are you sure?

[Errol backfiring]

Are we sure this traffic comes from Microsoft? Could it not consist of forged network packets? You don't need a reply if you are running a DDOS. On the other hand, why would anyone, including Microsoft, want to bring down CPAN?

Re:Are you sure?

[Anonymous Coward]

Because they are coming out with P# and don't want the competition?

Re:Are you sure?

[Anonymous Coward]

You only see an IP in an apache log after a successfull TCP handshake. This is hard (not impossible, but really, really hard) to do with a forged IP.

Re:Are you sure?

[TheRaven64]

Are we sure this traffic comes from Microsoft? Could it not consist of forged network packets?

It's a TCP connection, so they need to have completed the three-way handshake for it to work. That means that they must have received the SYN-ACK packet or by SYN flooding. If they are SYN flooding, then that would show up in the firewall logs. If they've received the SYN-ACK packet then they are either from that IP, or they are on a router between you and that IP and can intercept and block the packets from thatIP.

You don't need a reply if you are running a DDOS.

You do if it's via TCP. If they're just ping flooding, then that's one thing, but they're issuing HTTP requests. This involves establishing a TCP connection (send SYN, receive SYN-ACK with random number, reply ACK with that number) and involves sending TCP window replies for each group of TCP packets that you receive.

On the other hand, why would anyone, including Microsoft, want to bring down CPAN?

Who says that they want to? It's more likely that their web crawler has been written to the same standard as the rest of their code.

Re:Are you sure?

[mikelieman]

On the other hand, why would anyone, including Microsoft, want to bring down CPAN?

Jealousy?

So block those IP ranges?

[Evro]

If they've identified the IP ranges, why not just block them? You can do it at the router or TCP level (drop packets), or just throw up a 403 Forbidden.

Re:So block those IP ranges?

[John Hasler]

> ...why not just block them?

They have.

Re:So block those IP ranges?

[Sarten-X]

For ignoring robots.txt, they don't deserve any more nor less.

Re:So block those IP ranges?

[delinear]

According to TFBlog, that's what they're doing (returning 403s), but it's still a nuisance as it's filling up the log file with thousands of requests per hour (I don't know if there's a way to prevent this being logged, I'm just relaying what they're saying).

Re:So block those IP ranges?

[thePowerOfGrayskull]

If they've identified the IP ranges, why not just block them? You can do it at the router or TCP level (drop packets), or just throw up a 403 Forbidden.

That's a good temporary solution -- but unfortunately as Bing continues to gain market share, blocking them will cost you. For this particular site it's probably not a big deal, but that's not really practical for other sites hammered by Bing (my own included: I've had problems with it ignoring robots, and frequently re-indexing the same unchanged pages - regularly consuming over twice the bandwidth that googlebots have in the same time.)

Re:Typical M$

[Anonymous Coward]

That's not a troll. That's common knowledge.

A more appropriate mod would be +5 Redundant.

Incompetent?

[omb]

Yes, Evil more so

Too easy for Microsoft

[BhaKi]

I suppose Microsoft can offer a simple explanation: "Our servers and other internal infrastructure are so vulnerable that they have been hacked and being used as remote-controlled botnets."

Re:Why?

[ozmanjusri]

Why? Bing?

They have to have SOME activity.

Sounds like there's more traffic from their bots than customers.

Robots.txt

[anomnomnomymous]

Can anyone here clarify what robots.txt stands for, as in:

Is it an 'agreement' to not scan the site at all (by a search engine bot), or is it meant to just not -display- those results in the search engine?
I'd assume, since everything on a site is more or less public, that it would be the second. And if so, I can't see anything wrong with what Microsoft's bots did.

I can see how scanning a site's content (even if you're not going to list the results in your search engine) can have some value to a company.

Re:Robots.txt

[Ogi_UnixNut]

It's the first. Whatever you specify in the robots.txt as no-follow etc... means not to spider the pages, so no scanning of them at all.

You use it for when you only want part of your site to appear in search results, such as just the front page (for example). The rest of the site should not be touched by the bot at all.

Re:Robots.txt

[afidel]

It's basically a rough pattern filter that the bot is supposed to follow on parts of the site not to crawl. One reason it's used is that you can have dynamically generated pages that create an infinite loop that's impossible for the bot to detect.

Re:Robots.txt

[TerranFury]

AFAIK you're not supposed to visit URLs that robots.txt tells you not to. The issue is more to do with load on the servers, side-effects from cgi programs, and the like (for instance, you don't want web robots clicking your "one-click ordering" button*) than it is to do with public visibility of the content: If you want to hide something, you don't put it up on a public webserver to begin with.

As usual, Wikipedia has more to say.

* ok, bad example; no purchase system actually works like this... but you get the idea.

Re:Robots.txt

[TerranFury]

I just learned something...

There are ways to achieve each of the various things you mention. See this, this, and this.

Re:Robots.txt

[anomnomnomymous]

Ahright. Never thought of that: That makes sense. Thanks for the answer :-)

Re:Robots.txt

[John Hasler]

Is it an 'agreement' to not scan the site at all...

It is a request not to scan part or all of a site. robots.txt

And if so, I can't see anything wrong with what Microsoft's bots did.

Every site does not have dozens of powerful servers and terabytes of bandwidth, nor is every site an ad-supported one that wants to maximize traffic. Common courtesy requires that a bot operator minimize his impact on any given site and honor requests not to index. Of course "courtesy" and "honor" are concepts that baffle Microsoft managers.

Re:Robots.txt

[Terrasque]

It is in the interest of the bot admins to respect it, since step 2 is usually full-out block of the ip's / user agent.

Re:Robots.txt

[John Hasler]

> ...step 2 is usually full-out block of the ip's / user agent.

Inconceivable to a Microsoft manager. No one could tolerate the resulting loss in traffic and therefor ad revenue. What's that you say? You operate a small, noncommercial site? Then it couldn't possibly have any content of interest to Bing users.

Re:Robots.txt

[Fnord666]

Here is some info that you might find helpful.

Or both

[cheros]

AFAIK, the one doesn't exclude the other.

However, assuming evil is more fun :-)

What the hell has become of the word "problem"?

[John Hasler]

> ...issues accessing their sites...

"Issues"? What's wrong with "problem"? "Issues" is marketing-speak. Microsoft marketing-speak.

And yes, get off my lawn.

Re:What the hell has become of the word "problem"?

[Spad]

Blame ITIL; you can't call it a problem until you've had multiple incidents, or something.

Re:What the hell has become of the word "problem"?

[FerociousFerret]

It depends on where you stand in the scenario.

In this case, for Microsoft, who is not directly affected, it's an issue. For CPAN, it's a problem.

Re:What the hell has become of the word "problem"?

[bipbop]

You can draw that distinction, if you like; but the word "issue" in this sense dates from the 14th century legal term "issue", and was used in this way long before you were born. See here for a discussion of different uses of the word: The Issue with Issues

Re:What the hell has become of the word "problem"?

[John Hasler]

I'm aware of the various meanings of the word "issue". It is now being used as a synonym for "problem", thus diluting the meaning of both words.

Send the lost bots home.

[N1ckR]

I redirect lost bots home, seems a polite thing to do. 301 www.microsoft.com

Re:Send the lost bots home.

[TheSpoom]

...which they'd look up in their cache and find that it's already been sufficiently indexed; the only thing it would do to them is add your site's (Microsoft equivalent of) PageRank to www.microsoft.com.

Re:The US government is competent.

[elvesrus]

you might want to read that over again

Re:The US government is competent.

[jimicus]

The US Gov't has successfully operated as a going concern for 220+ years, with a proven and reliable management structure. Few, if any corporations, have been able to do that.

Private corporations can go under with just a couple of bad years. Or even months, particularly if they're new businesses. Governments just have to raise taxes.

DDoS? Really?

[Siberwulf]

I'm pretty sure the first "D" in DDoS stands for "Distributed."

If it was really a DDoS, you wouldn't be able to filter the IP out with a simple regex (like the /^65\.55\.(106|107|207)/. from TFA).

To boot, TFA didn't even say DDoS. Maybe that's too much to expect the editors to oh... I don't know...say... RTFA or Fact-Check it?

I should drop my bar a bit, I suppose.

Re:The US government is competent.

[tjstork]

you might want to read that over again

Didn't say I was!

Re:The US government is competent.

[tjstork]

I don't think you actually know what you want and I think people of a similar mind will do much more harm to the US

I think that is a fair statement. I'm putting together a piece for the relaunch of my web site that takes the federal budget, breaks it down to # of days you have to work to support each line item, says, what happens if you don't do that, then, lets you cut to your heart's content, and then tallies the results for everyone to see what the averages are.

I don't think anyone even really gets the government at all, left or right.

No problem

[rgviza]

ipchains -A input -j REJECT -p all -s 65.55.207.0/24 -i eth0 -l
ipchains -A input -j REJECT -p all -s 65.55.107.0/24 -i eth0 -l
ipchains -A input -j REJECT -p all -s 65.55.106.0/24 -i eth0 -l

problem solved

Re:No problem

[j_sp_r]

Linux IP Firewalling Chains, normally called ipchains, is free software to control the packet filter/firewall capabilities in the 2.2 series of Linux kernels. It superseded ipfwadm, but was replaced by iptables in the 2.4 series.

You're a few kernels behind.

Re:No problem

[Anonymous Coward]

He's just running Debian stable. SCNR

Complain to Upstream Providers

[jchawk]

The CPAN folks could complain to their ISP and have them drop the traffic that's coming in to their boxes.

Most ISP's will work with you to correct DDOS problems.

Astroturfing Idiot

[omb]

If you dont know, you should Google it, that will make it clear /. is not a -help mailing list and this was stupid, feckless and criminal, as in mis-use of a computer system beyond authorisation.

Re:Astroturfing Idiot

[John Hasler]

Stupid and obnoxious, but not criminal unless there was deliberate intent to interfere with use of the site. Robots.txt is not access control. If you want to strictly limit your site to authorized users install an authorization system. The Web is public by default.

Re:Typical of Bots

[AHuxley]

MS will just blame outsourcing, Danger engineering, pink, a new team just took over, oh wait they used that.
Outsourcing works best.
If you think about the other options it gets more interesting..
Are google, yahoo, ms ect all passing robots.txt over?
As a share holder, why waste the cpu time, storage and power costs if its of not of any direct short term or long use?
Put that cpu time, storage and power usage to good for profit calculations, indexing faster or quality ads.
If not who is paying for the ignore sites and why..

Re:The US government is competent.

[Lloyd_Bryant]

I'm a right winger and I like to see smaller, less intrusive government, but, I think it is wrong to say that the US government is competent.

The US Gov't has successfully operated as a going concern for 220+ years, with a proven and reliable management structure. Few, if any corporations, have been able to do that.

Let's see...

War on Poverty - yeah, that worked out *real* well, didn't it.
War on Drugs - See any results there?
War on Terror - With this one, I can't really tell if it's bungling, or actual malice

Those are just the "big names".

The US Government is a well-designed structure. And it worked pretty darn well for a while. But as federal power has increased, the effectiveness of that structure has decreased. In short, the Republic of the Founding Fathers is showing it's years.

And, for the record, there are organizations such as Lloyd's of London that can trace their existence, in one form or another, back to the 17th century. And if you want a truly old corporation, look at Stora Kopparberg Bergslags Aktiebolag in Sweden, which has been around since the 1300's!

Most corporations wither and die after a while, since their markets can wither and die. The "market" of a government does not - the only way for a government to "die" is by armed force, either from within or without.

Re:Happy Dead Nigger Day!

[woody.jesus]

How dare you sir (or madam)!! How dare you! It is clear from the title of your post that you were not so subtly casting aspersions on an organization who I hold dear -- namely the Hirsute Dungeons n' Dragons society. You can frame your remarks in some obscure racial epithets, but to those of us who twirl our mustaches or stroke our beards while rolling dice, your insidious implication is brazenly clear. As the leader of a group of men (and women) With decorative facial hair who play Dungeons n' Dragons every Wednesday night, I cannot help but express the strongest offense to your euphamisticaly delivered hidden acronym. In the future, should you have such thoughts I would urge you to Do Not Say them.

Re:Why?

[Mitchell314]

You mean the mods have to read TFCs? D:

Re:Happy Dead Nigger Day!

[ckaminski]

You know women with decorative facial hair? mkaaaay....

Re:Typical of Bots

[jack2000]

Disallow a directory in robots.txt if anyone opens it have a link there along the lines of: If you open this your ip will be blocked. Everyone that requests that link gets nullrouted for a week if they do it again they get nullrouted forever.

I can't wait till the MS bots index private data

[StuartHankins]

What happens when the MS bots (which apparently ignore the robots.txt file) start indexing some site which provides pay-per-view information? Can we expect a fix to the problem then? All it takes is to get some lawyers involved, you know how that snowball goes.

IP Spoofing

[jkantola]

How's it possible that, on Slashdot of all sites, *I*, of all people, need to tell you that IP packets do not necessarily come from the address inscribed in their headers?

Re:IP Spoofing

[John Hasler]

> IP packets do not necessarily come from the address inscribed in their
> headers?

TCP/IP connections do necessarily come from the address inscribed in their headers.

Re:IP Spoofing

[Anonymous Coward]

Because you don't know what you're talking about?

Understand, then post.

Re:IP Spoofing

[Fnord666]

How's it possible that, on Slashdot of all sites, *I*, of all people, need to tell you that IP packets do not necessarily come from the address inscribed in their headers?

Maybe it's because most people here know how TCP works?

Mod parent up

[Lonewolf666]

While he could be more polite, it is indeed embarrassing for Microsoft if they cannot check their own network
a) for the existence of computers with given IPs
b) what these computers are doing

I think that deserves an "insightful" that cancels out the "flamebait".

Re:Why?

[darkpixel2k]

Bing?

Ned? Ned Ryerson?

Re:US Government is good.

[Nadaka]

Nothing you listed under the "War on Drugs" has anything to do with the war on drugs.

The war on drugs has made America a police state where the government can seize any of your property and auction it for profit before your trial. Even if you are found innocent, or the charges are thrown out for insufficient grounds, you will not be compensated for your lost money or profit. It has made an America where more people are imprisoned than any other nation on earth. It has made a nation where the cheapest and most effective drug for curing glaucoma and mitigating the pain and nausea associated with cancer treatments is a crime. Its made a nation where at least half its citizens are criminals.

Re:I can't wait till the MS bots index private dat

[John Hasler]

Robots.txt is merely advisory. Ignoring it is discourteous and oafish but not illegal.

hello? firewall?

[v1]

if it's a scan (TCP established stream, taxing the SERVERS, not the NETWORK) that's the problem, as opposed to a SYN flood etc, and the IP addresses are in a very small range, why aren't they just using a hardware firewall at the router and blocking the IPs? There's not a whole lot to "distributed" when it's coming from a pair of C's.

Not saying they should be DOING it, but this is not a Denial of Service, it's a Denial of Stupid.

I was just noticing this...

[faedle]

Wow, this article is prescient.

I was just noticing in my web logs that small, out of the way sites that I host that used to get 1,000 hits a month were suddenly getting 1,000 hits PER DAY. Sure enough, anybody care to guess what netblock the 26,000 hits came from?

Microsoft.com just earned a ban.

Re:Why?

[iluvcapra]

Watch out for that first step, it's a DOOZY!

Re:Happy Dead Nigger Day!

[MstrFool]

Sweet, got any room in your group? I have my own dice, mustaches and beard, though sadly lacking women for some reason.

Re:US Government is good.

[tomhudson]

You left out the "War on Drugs"

A total failure to treat a social problem. Wasn't Prohibition ! enough for you.

Legalize it and tax the crap out of it.

Who is threatened by that? The crooks. As long as there's a "War on Drugs", crooks are guaranteed monopoly profits and monopoly access, all supported by your tax dollars keeping people in jail.

The first rule of consulting is "No matter what they say, it's ALWAYS a people problem." Well, it's true here to.

Right now, the US has more people in jail than any other country in the world. And the #1 reason is the "War on Drugs." Stop it and you'll reduce crime, reduce drug use, save money and lives, and fix the deficit.

Re:US Government is good.

[tjstork]

Stop it and you'll reduce crime,

No, we'll just legalize it. So now we'll have corporations buying out advertising to convince people to ruin their lives by purchasing smack.

Re:The US government is competent.

[tjstork]

I'm a right winger and I like to see smaller, less intrusive government, but, I think it is wrong to say that the US government isn't competent.

Fixed that.

Re:Evil? What "evil"?

[jbengt]

Evil does not require malice

Re:US Government is good.

[tomhudson]

No, we'll just legalize it. So now we'll have corporations buying out advertising to convince people to ruin their lives by purchasing smack.

Do the same as tobacco - high taxes, illegal to advertise, gross packaging, fines of up to 2/3 of a million dollars for illegal distrbution, etc.

bing is written in perl

[bingoUV]

Got it! Bing is written in perl. They do regular expression matching while crawling and forgot to have a \E ... \Q escape sequence for the regex matching. They got so much perl code on CPAN, full of special characters, that somehow the crawler engine went into an infinite loop.

Re:The US government is competent.

[SpaceLifeForm]

He said he was a right winger.

Simple solution:

[Anonymous Coward]

Add to your .htaccess file:

deny from 65.55.207.
deny from 65.55.106.
deny from 65.55.107.

Re:Looks like a simple bug to me

[chromatic]

Looks like cpan removed their robots.txt file at least from where I am sitting.

The file in question is robots.txt for cpantesters.org, which does exist.

Re:US Government is good.

[tjstork]

Do the same as tobacco - high taxes, illegal to advertise

I'm down with that.

download locally first and then test indexing

[Anonymous Coward]

Bing should have used Wget first to download the articles to a local hard drive, and also to add a 2 to 3 second wait. Let it run over the weekend. Then test the search indexing algorithms on the local HTML files. They were probably performing indexing tests. I know they have smart people working for them, so it probably involved a contractor who didn't think about performance issues.

robots.txt is to protect servers, not spiders

[billstewart]

If you remember the history of robots.txt because you were there are the time, rather than because you read it in some history book somewhere, the purpose was to protect small web servers from being trashed by big search robots, initially altavista, and secondarily to protect them from other well-behaved web crawlers of whatever sorts. There were no script-generated pages back then, or at least hardly any; just handing out static html could be difficult enough if you had a small pipe and a slow server, though serving images to a robot obviously a waste of time back then.

Tarpits of various sorts existed soon after robots.txt, as a way of trapping spammer-run crawlers that ignored robots.txt, but that was as much for fun as for necessity :-)

And yes, people did have /private/ directories back then and still do now, thinking that because Google's polite about not looking in directories robots.txt says not to that there aren't humans or impolite robots that won't look there.

No, Blocking won't make them behave

[billstewart]

It'll just keep them from bothering you, and you're (almost by definition) too small for them to care that they're not indexing your site.

Advertising their IP address block with BGP, if your ISP is careless enough to let you do that, now *that* would get their attention :-)

As an intermediate level of annoyance, you could set up your DNS server to respond to queries from Microsoftland to return entertaining IP addresses, such as 127.0.0.2 or bing's IP addresses or whatever.

Re:Robots.txt is there to protect servers

[billstewart]

The primary reason for robots.txt was to protect small slow web servers from being swamped by Altavista's big fast web crawlers. Dynamic pages weren't a problem back then. On the other hand, after robots.txt became common, setting up dynamic pages to trap crawlers that ignored it into infinite loops became common also, because most of them were run by spammers of various sorts.

Linking to SEOs encourages scum

[billstewart]

Search engines try to tell humans what web sites would have interesting contents based on their queries. They use robots and content models to approximate that so they can produce results quickly and economically. SEOs try to get the robots to tell the humans "my page is really interesting", when it usually isn't, which is scummy lying, and you shouldn't encourage such people.

They've really got three things to offer:

• Telling the web site owner how to structure their content so that the robots can find it. That's legitimate and useful, but that's also something that can be covered in 1-2 pages of documentation. On the other hand, sometimes less-technical web site owners find it worthwhile to hire consultants to do that for them, which is fine, but usually the consultants they really need call themselves "web designers".
• Telling web site owners how to make their content look more interesting (keeping it up to date, adding new material, etc.) To the extent that it's making the content actually more interesting to human readers, cool, but if the consultant is just adding features to make it look attractive to robots so humans will look at it and generate advertising revenue, and not to make it actually interesting to actual humans, it's still sleazy. If you want to hire somebody to make your site actually interesting to humans, the consultants you should hire usually call themselves "editors" or "authors" or "web designers", not "SEOs".
• Lying to robots so the robots will lie to the humans, using whatever tricks still work, whether that's link farms or astroturfing popular discussion sites or blog comment spam or whatever. This is sleazy scum behaviour that makes search engine results less useful to humans, and at best it's done because of ignorance and greed, though it's increasingly often done to distribute malware. And yes, if you want to hire somebody to lie to robots to boost your search engine position, the consultant you're looking for will call probably themselves an SEO.

Re:Linking to SEOs encourages scum

[TerranFury]

Good points. I hadn't noticed my sources. Anyway, my purpose had only been to figure out how the various robots.txt and HTML META directives are interpreted to respond to great-great grandparent.

Re:Linking to SEOs encourages scum

[billstewart]

I'm guessing you probably looked on Google for references to robots.txt and HTML META - if SEO scum are any good, they should be among the first references you'll find, because that is one thing they know about, and they'll use that as part of their self-promotion.

robots.txt

[petit_robert]

I don't see any requests for Robots.txt in my logs. It's always lower case :
65.55.106.138 - - [19/Jan/2010:01:00:46 +0100] "GET /robots.txt HTTP/1.1" 200 30 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

Re:CPAN webserver broken

[Slashcrap]

They likely have a web-server written in perl

You are likely a fucking idiot. I saw your sig about spite. You're still a fucking idiot.

Actually why the fuck did you spend so much time writing all that totally speculative bullshit just to try and prove that MS aren't at fault? Do you want to explain that? Because case sensitive web servers written in Perl are a bit of a fucking stretch of the imagination.

And no, I don't want to show sound reasoning to disprove some wacky shit your diseased brain made up.

Here it is!

[LordAzuzu]

Here it is another one from some minutes ago:

IPv4: 65.55.34.139 -> 83.211.46.34
      hlen=5 TOS=192 dlen=162 ID=46000 flags=0 offset=0 TTL=0 chksum=7990
Payload: Priority Count: 5
Connection Count: 6
IP Count: 7
Scanner IP Range: 78.130.238.2:212.90.12.134
Port/Proto Count: 7
Port/Proto Range: 80:40210

65.55.34.139 resolving to col0-omc3-s1.col0.hotmail.com

Re:CPAN webserver broken

[lpq]

My, seems like I struck a nerve. The the web server doesn't have to be written in perl for it not to ignore case. The point was it doesn't.

As for my comment about them writing a web server in perl being wacky -- you obviously know nothing about the perl community. There is nothing that can not be done better in perl -- including a web server. I don't seen that being an unwarranted comment. May not be true in this circumstance, but I'm sure it's been done. Did you even bother to check cpan for a cpan webserver? I'll take my bemusings, that you erroneously call speculations, over your sad, dim existence speculation any day. Tell me "CPAN::Mini::Webserver" isn't meant to server up a copy of cpan -- maybe not used for the main site, but...maybe with a squid accelerator front end -- yeah...I could see it!

Too bad you are such a hateful, spiteful diseased thing. You really should get some help or consider doing mankind a favor and stop wasting the planet's resources with your continued existence. It would be the responsible thing to do.

-l

Re:The US government is competent.

[DragonWriter]

Private corporations can go under with just a couple of bad years. Or even months, particularly if they're new businesses. Governments just have to raise taxes.

Governments can fail quickly, too. Sure, they usually fall to different problems than private entities do -- governments usually that fail early generally due so, if it is early in their life, because of violent reactions by existing governments, and otherwise (early or not) because they so fail the populace that they see a violent reaction from them.

New attempts to start governments probably fail about as frequently as attempts to start businesses.

And, like any other government policy, raising taxes only works to the extent that the governed populace is willing to accept it.

Re:CPAN webserver broken

[chromatic]

May not be true in this circumstance....

Did you even bother to check the headers of the CPAN Testers site? It's Apache httpd. You spent longer typing your speculation and its defense than it would have taken you to verify for yourself.

Re:CPAN webserver broken

[lpq]

You miss this the main point. Funny how people focus on the unimportant details when they don't like the main statement of the post. Robots.txt says to ignore case. It only makes sense for their webserver to
also ignore case for the file name. It sure would make retrieving cpan modules much easier. I'm always forgetting where some specific author has decided to put caps - because it isn't done consistently. It would be far smarter to allow case insensitive searching and usage given how capricious case usage is. They got hoisted by their own petard.

Most web servers ignore case. Theirs doesn't because they like to give authors the ability to randomly force
users to remember random combination of case. Yipee. The bit about the perl server was a piece of dry wit for reasons I've previously stated. It wasn't meant as an insult. That you took it that way shows you aren't a true perl affectionado, so stop complaining.

If I remember apache defaults to options to set case insensitivity. So they'd have to explicitly disable case insensitivity to enable this vulnerability.

Re:You've been Bing'ed

[Macrat]

Bing


Bing

Re:CPAN webserver broken

[chromatic]

Funny how people focus on the unimportant details when they don't like the main statement of the post.

If you can't get any trivially verifiable details correct (including which site this is), why should anyone take your random speculations seriously?

Exploited servers

[nurb432]

Ya, give them an excuse to get away with it. "it wasn't us attacking our competition, really"

Re:Exploited servers

[LordAzuzu]

I just can't believe they are so dumb, that's why.
No way I'm trying to defend them.

Network Solutions Domain Information

[DJRumpy]

http://www.networksolutions.com/whois/results.jsp?ip=65.55.207.0

Re:Network Solutions Domain Information

[Emilio III]

Yes, I'm an idiot. How I managed to get a typo in a copy-and-paste job I haven't figured out yet.

Re:Network Solutions Domain Information

[DJRumpy]

I was actually wondering if your browser or dns had been hijacked ;)

No worries...