When Does Website Monitoring Go Too Far?

jafiwam asks: "Recently, the IT department of the company I work for and a 3rd party monitoring and security firm got into a pissing match about how much monitoring is too much. They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers. They then proceeded to sell them monitoring services for things such as server up-time, defacement detection, email up-time and DNS testing. While I welcome anything that lets our customers use the internet effectively, their set of monitoring servers filled an entire 18 gig partition full of web server logs (causing the server to crash on a weekend) and choked an email server with 40k some messages that could not be delivered, and they failed to properly brief the hosting customers about what would happen to their log analysis software when faced with 99% traffic from a small set of IPs. These things caused down-time, lost productivity and a damaged reputation. What is appropriate for monitoring a web site and email server? Who should be allowed to monitor? Where should the give and take lie in this situation? I am interested in finding out what admin-on-the-street has to say about this."

"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.

Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)

Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."

Comments

hm

[revmoo]

From your description, i.e. "Once per second", that is quite beyond monitoring, and that is an EXCESSIVE use of bandwidth and resources.

Now, if you charge your customers based on gigs transferred, it seems like this would fill up their quota for the month quite quickly. What are your customers going to think when they get a large overcharge bill for the bandwidth? They signed up for the service after all.

If you aren't hosting for money, then you probably aren't able to profit from this monitoring companies actions in the same way, so I suggest you blackhole their ip's. Downloading files from your server once per second goes way beyong monitoring, and into the realms of denial of service(It crashed your server you say).

What I would do? Make a change to the aup for your service stating that customers that use monitoring services that abuse bandwidth will have their accounts revoked, or be charge for the excess bandwidth used. There's no reason in the world why these people need to hit your servers as often as they are.

If you are unable to do business with your servers being hammered, then I suggest blackholing the monitoring service's IP's. It's only sensible.

How much is too much?

[Alien Being]

Here's a common sense reaction.

They are in the business of measuring Net availability. They should learn to set the scale on their instruments before they connect them to the circuit. And they should back off when availability drops because they might be the cause of the drop. If their traffic represents more than about 10x that caused by an individual customer, then as a "juror" I'd think they were being irresponsible.

You are in the business of supplying Net availability. You should install circuit breakers. Too many connection from one host/network? Start dropping packets. Too much raw incoming traffic from one source? Get on the horn quickly to the netadmin.

Your customers don't care who's at fault, they want what they paid for. But they can't expect miracles.

I work in network management...

[Ranger Rick]

And I can tell you that if they're polling at 1 a second of *anything*, they don't "know what they're doing". That is complete overkill, there's no way the amount of bandwidth being used for testing is worth the 59-second jump on knowing what went wrong. Humans generally have to react to it, that kind of resolution is just crazy.

I haven't been impressed with monitoring companies

[eric76]

A couple of years ago, a so-called "security expert" sold the president of my company on the idea of installing a firewall.

To some extent, that was fine with me. I'd been arguing for that for a very long time but had gotten nowhere because the "security expert" said that firewalls weren't necessary! I guess someone finally bothered to break into his system.

The security expert's idea was to have a third party monitoring company do it all. So I spent a couple hours on the telephone one day talking to the monitoring company's personnel about our network requirements and traffic. We went into great detail over exactly which servers had to handle which services.

The firewall arrived and the security expert plugged it in. It didn't work at all. All it did was block everything. I was 600 miles away at the time and it took me a week to convince them to take it off.

They decided the firewall was defective and the monitoring company set up another one. By the time it arrived, I was back in the office. The big day came and the security expert had one of his employees come out and plug it in.

It didn't work at all.

I caught the employee of the so-called security expert before he could leave the building and had him remove it. The idiot didn't even bother to check to see if it was working.

After he left the building, I started looking at how he had it plugged in. He still had a cable plugged into the firewall from an internal hub.

He had connected the untrusted side of the firewall to the internal network. I assume that the cable from the Cisco router was plugged into the trusted side of the firewall.

But it really didn't make much difference. I also found the rule set for the firewall. The monitoring company had set it to pass nearly everything in both directions.

The only thing they configured was to block incoming traffic containing our IP addresses. Since it was plugged in backwards, it really just stopped all traffic from going out.

At this point, it would take a lot of convincing to get me to advocate using a monitoring company's services.

By the way, the same so-called "security expert" declared that rules on the Cisco router to block traffic attempting to connect to port 135 and other similar ports constituted a security list and removed them.

Re:Confidentiality

[Maserati]

Firewalling them is good, your customers have no authority to allow them that kind of access to your network. Have your corporate attorney send them a polite C&D letter. By polite, just the followup contact - this time on an attorney's letterhead. Also consult the attorney for what you should/can tell your customers, then do so immediately.

Be very clear to your customers that your objection is the nearly-criminal (it's a DOS) heavy-handedness, mind-numbingly unethical and pathetically incompetent behavior of the monitoring company. It's not unreasonable for one of your customers to retain a third party to provide professional services of this nature; by professional I mean 'do it right' not in the sense of professional as a term of law. Loading your website at regular intervals and parsing their logs for them is fine. Right now, these guys are probably reporting the outages they caused.

Billing your clients for bandwidth used by the monitoring company they hired is not completely unreasonable. Be sure to document every cost associated with this in every way, including time reading responses to this article as 'best practices research'. I'm not kidding, if you worked late you add the pizza in or the taxi home. Every penny in fine detail. Your lawyer will be keenly intereste, so might law enforcement if the polite C&D letter didn't do it.

Since the offered protection, aka monitoring services and then caused damage to your systems you could make a case that a protection racket is being run. If, adding in their fees for their services (paid by your customers) to the damages calculated above you have more than a certain threshold, probably US$50,000, then the FBI will be interested. Also have the monthly and annual total of your revenue from the customers either employing the monitoring service plus those affected by the damage cause (probably all of them). If things go sour with them and you do go to law enforcement, wave your revenue totals around to help get DAs and FBI interested.

Basically, you call your lawyer and then contact your customers. Your lawyer asks them to behave themselves. Then you meet with the lawyer, discuss the response and post another Ask Slashdot.