When Does Website Monitoring Go Too Far?

jafiwam asks: "Recently, the IT department of the company I work for and a 3rd party monitoring and security firm got into a pissing match about how much monitoring is too much. They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers. They then proceeded to sell them monitoring services for things such as server up-time, defacement detection, email up-time and DNS testing. While I welcome anything that lets our customers use the internet effectively, their set of monitoring servers filled an entire 18 gig partition full of web server logs (causing the server to crash on a weekend) and choked an email server with 40k some messages that could not be delivered, and they failed to properly brief the hosting customers about what would happen to their log analysis software when faced with 99% traffic from a small set of IPs. These things caused down-time, lost productivity and a damaged reputation. What is appropriate for monitoring a web site and email server? Who should be allowed to monitor? Where should the give and take lie in this situation? I am interested in finding out what admin-on-the-street has to say about this."

"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.

Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)

Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."

Comments

One word:

[stor]

Nagios.

http://www.nagios.org/

Cheers
Stor

Re:One word:

[@madeus]

Both are good monitoring packages, it's up to personal preference really.

Actually Nagios is a lot more powerful that BB (which really doesn't do all that much), and aisde from that Big Brother is not 'free' (often people just don't bother to read the Terms and Conditions and think it's free).

You can use BB with no charge to monitor certain systems, but if you provide certain types of services you are required to by a license, and these days most medium and large ISP's fall under this category.

Big Brother is amazingly basic, I don't understand why people get so excited about it (I could re-write it in a day, and I'm far from a rocket scientist). Nagios, in contrast, is a full network and service monitoring system, and would have been much more useful in this instance and you could have used it to more easily identify the source of the incoming traffic.

monitoring

[Feyr]

we typically set our monitor software to check every 5 minutes, with one request PER SERVER not per site. if it is down it will send an email to our support address, if it is STILL down the second time around, it fires off an email to the cell phone of the on-duty admin, plus one email when it comes back up

i've had some services set up for monitoring as low as 30 seconds, but those are specific cases.

obviously a 1 seconds check is WAY too low, not only it's a waste of bandwidth, it's prone to false positives. what happen when you have a slight delay in one of the core routers that cause your packet to get dropped/delayed by 1000ms ?

Re:Log partitioning

[MikeFM]

I'd think somebody would have noticed the high usage and firewalled off that site too. I mean jeez that must have been thousands and thousands of hits to use up that much space. I'd suspect a DoS attack if I saw that in my logs.

I also suggest anyone running servers to have some sort of program monitoring disk usage. If the disk gets dangerously low on space it should notify staff and take action such as rotating logs. Have the server page an admin or set an alarm off (where it'll be noticed) or something. Whatever you'd do if an attempted intrusion was detected. I usually have the server send warnings at 90% and 95% and at about 97% usage it should give me a good loud yell.

Re:Confidentiality & TOS & Abuse

[vt0asta]

What he said...

Sounds like you've got an open and shut legal case to recoup those costs they're causing you to incur.

First things first. These are your servers. Your network. I am assuming you have the standard abuse clause in your TOS. You need a lawyer.

Unfortunately, you are in a bad situation. They apparently have more resources than you, because they can bring your setup to it's knees. Not saying it's right, not saying it's fair.

A lookup of your TLDs each second makes sense if you are Yahoo! or Google. Their web monitoring levels don't appear to be reasonable. You already know the technical answer.

Personally, I would be worried about them stealing your customers. I mean the argument is going to be simple from their side. They will simply say, "hey look, their stuff folded under 'normal' monitoring, we have a hosting company we can 'recommend'" or they will just have the hosting company call them up out of the blue and ask if they are "unhappy" with thier current service..."oh, it goes down a lot"..."they can't handle simple monitoring"..."gee, that's a shame"..."well, we've worked with that monitoring company before, and we have never had any problems, in fact we routinely get 5 9s"...etc

Honestly, talk to legal, explain the potential situation, and have them make contact with the monitoring company. A couple of tortious interference this, and cease and desist that, will put the monitoring company on it's toes and maybe get them to leave your customers alone, or possible play nice with your servers. Notify your customers yourself and explain that they are being investigated by your legal team, etc.

Re:Confidentiality

[vt0asta]

IANAL, but if you'll allow me to shoot from the hip for a bit, I'll take a shot at it...

1) Tortious interference with business relationships. The solicited the customers. They directly interfered with the business relationship by bringing the servers down by overzealous monitoring.

2) The outage was caused by the monitoring company. If just one customer leaves to another hosting company because of outages or what not, or if that customer lost business due to downtime. The damages are realizable.