Slashdot Mirror
When Does Website Monitoring Go Too Far?
"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
A server should not choke if the log partition is full. Is the log in a separate partition, isn't it?
Nagios.
http://www.nagios.org/
Cheers
Stor
"Yeah well there's a lot of stuff that should be, but isn't"
I agree with most of what you say but simply pinging ports in no way guarantees functionality. Our out sourced mail host commonly has issues yet responds to ping and allows logins still.
we typically set our monitor software to check every 5 minutes, with one request PER SERVER not per site. if it is down it will send an email to our support address, if it is STILL down the second time around, it fires off an email to the cell phone of the on-duty admin, plus one email when it comes back up
i've had some services set up for monitoring as low as 30 seconds, but those are specific cases.
obviously a 1 seconds check is WAY too low, not only it's a waste of bandwidth, it's prone to false positives. what happen when you have a slight delay in one of the core routers that cause your packet to get dropped/delayed by 1000ms ?
From a business perspective, monitoring is a service *you* should offer to your customers. Since it is your network, you have the ability to provide a much more effective and accurate monitoring service, and can set the resolution of the service according to your customers needs. All the problems you describe are because they are operating from the outside. What that monitoring service is effectively doing is stealing your bandwidth, and selling to your customers. If you want to get your lawyers involved, send them a C&D since they are affecting your ability to conduct business. personally I would firewall then as the CTO has done, and offer the same service internally.
First things first. These are your servers. Your network. I am assuming you have the standard abuse clause in your TOS. You need a lawyer.
Unfortunately, you are in a bad situation. They apparently have more resources than you, because they can bring your setup to it's knees. Not saying it's right, not saying it's fair.
A lookup of your TLDs each second makes sense if you are Yahoo! or Google. Their web monitoring levels don't appear to be reasonable. You already know the technical answer.
Personally, I would be worried about them stealing your customers. I mean the argument is going to be simple from their side. They will simply say, "hey look, their stuff folded under 'normal' monitoring, we have a hosting company we can 'recommend'" or they will just have the hosting company call them up out of the blue and ask if they are "unhappy" with thier current service..."oh, it goes down a lot"..."they can't handle simple monitoring"..."gee, that's a shame"..."well, we've worked with that monitoring company before, and we have never had any problems, in fact we routinely get 5 9s"...etc
Honestly, talk to legal, explain the potential situation, and have them make contact with the monitoring company. A couple of tortious interference this, and cease and desist that, will put the monitoring company on it's toes and maybe get them to leave your customers alone, or possible play nice with your servers. Notify your customers yourself and explain that they are being investigated by your legal team, etc.
No.
IANAL, but if you'll allow me to shoot from the hip for a bit, I'll take a shot at it...
1) Tortious interference with business relationships. The solicited the customers. They directly interfered with the business relationship by bringing the servers down by overzealous monitoring.
2) The outage was caused by the monitoring company. If just one customer leaves to another hosting company because of outages or what not, or if that customer lost business due to downtime. The damages are realizable.
No.
This is not legal advice. Find a lawyer, ask them what to do.
It seems as though you've got a tort of negligence on your hands, insofar as they seem unaware, or oblivious to, the damages they are causing you. They do not seem, from your statements, to be wilfully causing damages, but negligence torts need not show (at least in the commonwealth) either wilfulness or intent. You need only show damages, which are an indirect consequence of their actions.
Take into account that torts are, by most accounts, very expensive, though the threat of a tort is often sufficient, or binding arbitration (though that is apparently not oft met with success), or mediation (same deal as binding arbitration). If you do have to litigate, the general rule is somewhere north of $100,000 in damages to justify the transaction cost, from what I have heard. See the first line, though - find a lawyer.
In the least you can establish damages in support of a trespass if you inform them that their actions cause damage, in which case their actions are thereafter wilful, which may make for a cleaner case. The onus in trespass is on the defendent (them) to defend against damages established, not the plaintiff (you); and whereas in negligence, the onus is on the plaintiff (you) to show damages.
Ok, so in gist, take everything I said with a grain of salt, and seek legal counsel. Your jurisdiction may have many options with respect to small claims or public dispute resolution, and I would suggest those because they are significantly cheaper.
Hope that helps.
Of course, a webhost also needs to communicate what their customers are paying for. If you claim unlimited bandwidth for $9.95 a month, don't be surprised when somebody takes you up on it. These customers should have had some sort of bandwidth limit where the overmonitoring would cause their site to get defaced with the webhost's "This site has exceeded it's bandwidth limit, come back next month!" page or start running up a huge bill. The customers should know better not to invite such an attack on the server, and should be the ones feeling the pain. That'll put this monitoring company out of business in a hurry...
If they're letting their logs get huge before rotating them it would cause a problem every time the server tries to append data at the end of the file.
And they shouldn't be keeping the logs on the server anyway. It's static data that only they could need access to. It should be moved off site to a standard IDE harddrive for processing.
Statistical data should be created as the data comes in and not from the log files if they intend to let the customers have statistics for whatever.
As for my own site, I have Apache doing the combined log format and wrote custom software to process and analyze the data. Every month I move the log off the server and every 10 megs or so I rotate the logs and move the data into a second cumulative file that Apache doesn't work off of.
Ben
Work Safe Porn
It is very important for a bigger hosting firm to have a good moniroting strategy which shows trhe external perspective.
The timing need not be more than 15 minutes in most cases. The plan should include the network, web server and applications, and possibly supporting servers such as email or DNS.
The external capabilities are critical - if you are going to do external, use a firm who has profressional managed remote stations in many places.
Tim Goeke
http://www.globalnetwatch.com
one of such monitoring tools is nagios. it allowes for multiple users, with access limited to view information only on specific hosts/host groups. it's a pain to set up initially, but in the end it works quite nicely. www.nagios.org
--- d'oh
I work for a large hosting company. We have a lot of customers who have monitoring companies monitor their websites (we actually use some). We obviously monitor our services ourselves, but it is not always objective doing this. Having said that, monitoring once per second is *stupid*, generally 5 minutes is appropriate and we monitor some things internally every 60 seconds. We charge for bandwidth and disk usage (including logs), so if people want to monitor every second, go for it, your credit card will get dinged next month. For a smaller provider, I can see this being a problem, I would blackhole the IP. It is a DOS attack and I'm pretty sure you would have the legal right to do that. You do have a provision in your policies that you can take necessary action to protect your network, right? We do and will use it when necessary. Right after 9/11, we had a *very* popular and large image on our servers (the "eagle", if anyone has seen it). We "chmod 0"'d it and called the customer. They didn't realize what happened (getting so many hits), understood (once we explained bandwidth charges), and where happy we did it. Monitoring every 5 minutes is reasonable and will catch almost all outages.
Cricket.
http://cricket.sourceforge.net/
Microsft spel chekar vor sail, worgs grate !!!