Slashdot Mirror
When Does Website Monitoring Go Too Far?
"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
...that either don't have the time / money to go after people like this, such as the webhotel I'm involved in in my sparetime, I'd recommend firewalling. Simply block all incoming connections from over zealous monitor-companies.
Ofcourse this doesn't do anything to fix the bad reputation they may have given you by flooding your servers, but its a quick and easy antidote against future problems.
From your description, i.e. "Once per second", that is quite beyond monitoring, and that is an EXCESSIVE use of bandwidth and resources.
Now, if you charge your customers based on gigs transferred, it seems like this would fill up their quota for the month quite quickly. What are your customers going to think when they get a large overcharge bill for the bandwidth? They signed up for the service after all.
If you aren't hosting for money, then you probably aren't able to profit from this monitoring companies actions in the same way, so I suggest you blackhole their ip's. Downloading files from your server once per second goes way beyong monitoring, and into the realms of denial of service(It crashed your server you say).
What I would do? Make a change to the aup for your service stating that customers that use monitoring services that abuse bandwidth will have their accounts revoked, or be charge for the excess bandwidth used. There's no reason in the world why these people need to hit your servers as often as they are.
If you are unable to do business with your servers being hammered, then I suggest blackholing the monitoring service's IP's. It's only sensible.
I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
Here's a common sense reaction.
They are in the business of measuring Net availability. They should learn to set the scale on their instruments before they connect them to the circuit. And they should back off when availability drops because they might be the cause of the drop. If their traffic represents more than about 10x that caused by an individual customer, then as a "juror" I'd think they were being irresponsible.
You are in the business of supplying Net availability. You should install circuit breakers. Too many connection from one host/network? Start dropping packets. Too much raw incoming traffic from one source? Get on the horn quickly to the netadmin.
Your customers don't care who's at fault, they want what they paid for. But they can't expect miracles.
And I can tell you that if they're polling at 1 a second of *anything*, they don't "know what they're doing". That is complete overkill, there's no way the amount of bandwidth being used for testing is worth the 59-second jump on knowing what went wrong. Humans generally have to react to it, that kind of resolution is just crazy.
WWJD? JWRTFM!!!
A couple of years ago, a so-called "security expert" sold the president of my company on the idea of installing a firewall.
To some extent, that was fine with me. I'd been arguing for that for a very long time but had gotten nowhere because the "security expert" said that firewalls weren't necessary! I guess someone finally bothered to break into his system.
The security expert's idea was to have a third party monitoring company do it all. So I spent a couple hours on the telephone one day talking to the monitoring company's personnel about our network requirements and traffic. We went into great detail over exactly which servers had to handle which services.
The firewall arrived and the security expert plugged it in. It didn't work at all. All it did was block everything. I was 600 miles away at the time and it took me a week to convince them to take it off.
They decided the firewall was defective and the monitoring company set up another one. By the time it arrived, I was back in the office. The big day came and the security expert had one of his employees come out and plug it in.
It didn't work at all.
I caught the employee of the so-called security expert before he could leave the building and had him remove it. The idiot didn't even bother to check to see if it was working.
After he left the building, I started looking at how he had it plugged in. He still had a cable plugged into the firewall from an internal hub.
He had connected the untrusted side of the firewall to the internal network. I assume that the cable from the Cisco router was plugged into the trusted side of the firewall.
But it really didn't make much difference. I also found the rule set for the firewall. The monitoring company had set it to pass nearly everything in both directions.
The only thing they configured was to block incoming traffic containing our IP addresses. Since it was plugged in backwards, it really just stopped all traffic from going out.
At this point, it would take a lot of convincing to get me to advocate using a monitoring company's services.
By the way, the same so-called "security expert" declared that rules on the Cisco router to block traffic attempting to connect to port 135 and other similar ports constituted a security list and removed them.
Firewalling them is good, your customers have no authority to allow them that kind of access to your network. Have your corporate attorney send them a polite C&D letter. By polite, just the followup contact - this time on an attorney's letterhead. Also consult the attorney for what you should/can tell your customers, then do so immediately.
Be very clear to your customers that your objection is the nearly-criminal (it's a DOS) heavy-handedness, mind-numbingly unethical and pathetically incompetent behavior of the monitoring company. It's not unreasonable for one of your customers to retain a third party to provide professional services of this nature; by professional I mean 'do it right' not in the sense of professional as a term of law. Loading your website at regular intervals and parsing their logs for them is fine. Right now, these guys are probably reporting the outages they caused.
Billing your clients for bandwidth used by the monitoring company they hired is not completely unreasonable. Be sure to document every cost associated with this in every way, including time reading responses to this article as 'best practices research'. I'm not kidding, if you worked late you add the pizza in or the taxi home. Every penny in fine detail. Your lawyer will be keenly intereste, so might law enforcement if the polite C&D letter didn't do it.
Since the offered protection, aka monitoring services and then caused damage to your systems you could make a case that a protection racket is being run. If, adding in their fees for their services (paid by your customers) to the damages calculated above you have more than a certain threshold, probably US$50,000, then the FBI will be interested. Also have the monthly and annual total of your revenue from the customers either employing the monitoring service plus those affected by the damage cause (probably all of them). If things go sour with them and you do go to law enforcement, wave your revenue totals around to help get DAs and FBI interested.
Basically, you call your lawyer and then contact your customers. Your lawyer asks them to behave themselves. Then you meet with the lawyer, discuss the response and post another Ask Slashdot.
Veteran, Bermuda Triangle Expeditionary Force, 1992-1951
I use nagios to monitor local servers and servers at a remote site.
Local servers: ping once a minut
Remote servers: http/ssh checks every five minutes
Miscellaneous services, i.e. free disk space, cpu usage, number of users, etc: 15 minutes
The admins at the remote site will reboot a server for me within 15 minutes of a service request.
I would think checking every second would create its own problems - DDOS my own servers.
I know this is feeding trolls, but... who is to say how bad it is for something to be disclosed?
not to mention, it's probably already quite possible for your users to find out you were down for 30 seconds or so; even if they don't know it was cause the ceo tripped over a network cable, and knocked some network equipment down
Need a Catering Connection
These guys don't know what they're doing if they are banging on your servers every second. It is a strategy that is bound to make any competent admin irate and probably break things. Anything more than once-a-minute is probably overkill. Once every 5 minutes is a good window for most things. Your people are quite entitled to block them at the firewall.
Your sales people have to figure out how to appease the customers. That's their job. You are a tech and you'll just foul things up using tools like fairness and logic. I've been there.
Lastly, if they overflowed your log partition, you aren't monitoring enough things. It isn't enough to make sure that your sites are up, you need to make sure that the disks they depend on have enough free space, that the servers they run on don't have unnacceptible load spikes, etc... Comprehensive solutions are hard, but quick-and-dirty solutions aren't. Remember though that it's hard to send pages from a dead server and design accordingly.
Unfortnately, we're missing a key part of proving the "protection racket" scheme here, proof that the monitoring company illegally got ahold of a customer list. If this company just spread by word of mouth though the customers and advertising aimed at webmasters in general, then there's nothing illegal and they'll defend themselves by tar and feathering the webhosting operation for not being able to handle the level of trafic they promised the customers.
The customers should have run up huge bandwidth bills by causing their traffic to suddenly multiply by thousands with the auto-checking for site defacement (trans: re-spidering their site at an insane rate), and that'd be the way to recoup costs and then come off as the good guys by waiving thousands in excess fees...
Charge for it. Notify yer customer (by perl of course *tee hee*) that their logs are causing their account to approach its space limit. They can either move the logs, delete the logs, stop the logging software or remove the logging software. Warn them that if this is not taken care of additional hd space fees will apply.
Make sure they know that cleaning up logs should be *cough* easy and pain free!
Ahh, yes, lawyers. Sue sue sue.
C&D what? Block them entirely with firewalling, that's your right. But lawyering this, you're asking for trouble. The very nature of TCP protocol is that THEY ask for info, the ISP network acknowledges and then GIVES them the correlated data. Absolutely nothing illegal here. The ISP defaulted and let them in.
As to billing your customers, how nice. The way I look at it, 2 companies screwed up and now you want the customer to pay. The ISP--hey, how about setting up the servers right so freakin logs don't crash the machine. Sending out alerts to your damn admins. Why didn't the weekend admin or admin on call at least notice this? Seems like the ISP is trying to save face and pass the buck. They were contributory to the fault and hassle that was caused.
To the monitoring service--quit being asses to potential customers.
As to the protection racket, nice slippery slope. The facts aren't entirely clear what caused the harm to the ISP side of things--sounds more like someone was caught sleeping and is now trying to pass the back. Really--did the third party system hammer their systems, or did the customers sign up, resulting in hammering of the systems? If the latter, it's not a protection racket. If the former, you have a chance.
Under the circumstances, the ISP has a better chance at a tort claim than a criminal case, and the tort claim would be stretching things a bit by itself.
Sometimes you need more checks then ping.
At one of the companies I worked for, we had a pretty large farm of web servers running, and some hefty database servers on the backend. Not to mention all the support servers; running specific tasks. Some scheduled, some triggered.
For our web application, ping wasn't enough. Sure, the server would be running, but since the application wasn't coded in pure html, we needed to make sure it was actually working.
We set up scripts to test the functionality of various application functions. We also had to monitor all the web servers and database servers individually. We also had to monitor the "service status" of the entire system; ie two web servers can fail and it's not an emergency - but if the application is not functioning through the load balancer, it was.
Ping doesn't always cut it. With any somewhat complicated web application you need to monitor the functionality of the application, not just the server.
To add, 5 minutes is a big deal. If you have a web application that's heavily used and with paying customers, it's important for you to be up and running. If something unfortunate should happen, you need to know right away. We had some of the simple checks running every 2 minutes, and some of the more intensive checks running every 3.
Obviously, running a check every second is ridiculous, especially if it's something dumb like TLD servers. An hourly check on that would be more then enough because you can't fix it quickly anyways. Not to mention that you must be aware of the monitoring system in place and make sure your servers won't choke from it. Make sure you have enough log space. Make sure you're not affecting application performance from monitoring.
- It's not the Macs I hate. It's Digg users. -
Some $9.95 a month websites don't even get a "real" user once per hour. So, for them that'd be a sudden multiple of traffic...
What this really smells like is a webhost who oversold their server on the theory that everybody would never take their accounts to the promised limits at the same time, and then that's just what happened and the webhost got exposed as not being able to handle it...
ok ok...about the only thing I find remotley factual in this article is the fact that this guy works for a 'company'...however it looks like he works for a company doing exactly the things he is asking about.
First of all, lack of any knowledge of partion or disk utilities to prevent such an occurance is unacceptable. I would not admit that in public about my company even if I used the phrase 'a company I work for', just on the off chance my negligence would be able to be tracked back to me.
Second, why are you not able to offer these services yourself? You make a claim that these people know what they are doing, so if you are at such a level to recognise what they are doing, how come you havent done it already? Did customer service become just a novelty to you? so I doubt this line very much... While I welcome anything that lets our customers use the internet effectively
Doing hosting myself, Im well aware of the tactics you speak of, being that I get bounce mail for nonexistant addresses sent to such titles as; president, ceo, owner, support, tech...and so on. And Im not sure exactly what you mean by 'choked up' your mail server. How do 40k NONEXISTANT addresses manage to slow down your mail server? Is it a 286?
The whole article just smells funny to me, as it seems like you are just pretending to care about the ISP's end and more concerned about the backlash of doing these things. What do you mean how far is to far? Again, if the people in charge cant figure these things out on their own, I would be very hesitant to admit that in a public forum.
Get your technical skills and decision making in line...THEN question how to outsource it..
Here's my funny story: I was using Perfmon (NT monitoring utility) to monitor usage on this half dead database server, trying to get some compelling figures so I could argue for a new server.
So it's got all these options, and I wasn't paying attention, so I just said, "Monitor EVERYTHING...At 5 sedcond intervals."
Fortunately, I'm not a complete idiot, and it only filled up the directory I'd set for it, not the whole harddrive, but it did teach me an important lesson about log files: You can get a gig of useless information in less than an hour, OR you can monitor the IMPORTANT stuff, and get a gig of useful information in 2 or 3 days.
In case anyone is wondering, my logs proved 2 things: 1) That they needed a new database server and 2) That the people who were bitching about it being slow ALL the time, were actually only working about an hour a day.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
They are not a reputable company. They are a bunch of retards who should be driven out of the industry with sharp sticks. More to the point, they should be reported to the FBI for conducting a malicioius attack against your network - and you have tangible damage to prove it.
Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
Here's a hint for you: Do they offer web hosting services themselves? You may have to dig real deep to find the connection, but if I had to guess, I'll bet they do. And I'll bet they offer it to your customers, based on the fact that they crashed your servers. "Your current service seems to have a lot of downtime. Perhaps you should consider moving to another host. We can make recommendations."
If you find any evidence that they offer any kind of competition to your hosting, report them to the FBI. They may well be a criminal organization engaging in a well orchestrated scam.
Or maybe they're just fucking stoopid. It's hard to tell from here.
If a company did this kind of thing, even if taken to court they could produce the logs that verify the artificial downtime in order to defend themselves against accusations of lying to customers. Then, when asked if their once-per-second monitoring could have been the cause of the problem in the first place, they could make some fanciful BS claims like "a good server should be able to handle that."
My apologies for spinning an entirely hypothetical, and possibly paranoid, scenario. This was the first thing to pop into my incredibly suspicious mind - plus, it has the makings of a good scam if it hasn't already been done. :)
Note: OP means the monitoring company with "they". Pissing off your customers by telling them that they themselves are under legal investigation is a kind of SCO idea...
I suffer from attention surplus disorder.