When Does Website Monitoring Go Too Far?

jafiwam asks: "Recently, the IT department of the company I work for and a 3rd party monitoring and security firm got into a pissing match about how much monitoring is too much. They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers. They then proceeded to sell them monitoring services for things such as server up-time, defacement detection, email up-time and DNS testing. While I welcome anything that lets our customers use the internet effectively, their set of monitoring servers filled an entire 18 gig partition full of web server logs (causing the server to crash on a weekend) and choked an email server with 40k some messages that could not be delivered, and they failed to properly brief the hosting customers about what would happen to their log analysis software when faced with 99% traffic from a small set of IPs. These things caused down-time, lost productivity and a damaged reputation. What is appropriate for monitoring a web site and email server? Who should be allowed to monitor? Where should the give and take lie in this situation? I am interested in finding out what admin-on-the-street has to say about this."

"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.

Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)

Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."

Comments

It's gone.

[Jonas the Bold]

Try it, type in a non existant .com, it no longer works.

Re:It's gone.

[zcat_NZ]

djbdns released a patch to ignore verispam's wildcard DNS entry the same day the change happened.

bind released a patch a day or two later.

Judging by the 'calm and measured commentary' I've been reading on various NOG mailing lists, I'd expect many ISP's to be ignoring verispam by the end of the week.

Re:It's gone. - No, it isn't

[rock_climbing_guy]

I just checked again and non existant .com addresses still resolve to Verisign. The trick is that your ISP may have blocked it. I'm on a university network that has blocked it. However, when I log into a remote machine and use lynx, non existant pages still resolve to Verisign. Also, keep in mind that this is only for .com and .net addresses.

Re:It's gone. - No, it isn't

[macdaddy]

Actually only the newest TLDs to do this are com and net. Numerous ccTLDs and one additional gTLD already do this. The complete list of TLDs that return bogus information follows:

gTLDs (Generic Top-Level Domains):

• com
• net
• museum

ccTLDsCountry-Code Top-Level Domains:

• ac
• cc
• cx
• mp
• nu
• ph
• pw
• sh
• tk
• tm
• ws