Slashdot Mirror
When Does Website Monitoring Go Too Far?
"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
They must be a way to enforce that they could check, say, only once every hour. And BTW, isn't your company missing an opportunity here? If you're already checking the servers, etc., why not make the tools available to the customers? They'll be more satisfied with the tools, and not having to pay the outside firm. You'll have more satisified customers and less churn....
Don't give a company of strangers the key to the front door. There's no reason someone from your company wasn't there to say 'when.' As for when too much is too much, it'd be when the efficiency of your main product is impaired to the point that you lose customers or reputation.
Banaaaana!
Nagios.
http://www.nagios.org/
Cheers
Stor
"Yeah well there's a lot of stuff that should be, but isn't"
From your description, i.e. "Once per second", that is quite beyond monitoring, and that is an EXCESSIVE use of bandwidth and resources.
Now, if you charge your customers based on gigs transferred, it seems like this would fill up their quota for the month quite quickly. What are your customers going to think when they get a large overcharge bill for the bandwidth? They signed up for the service after all.
If you aren't hosting for money, then you probably aren't able to profit from this monitoring companies actions in the same way, so I suggest you blackhole their ip's. Downloading files from your server once per second goes way beyong monitoring, and into the realms of denial of service(It crashed your server you say).
What I would do? Make a change to the aup for your service stating that customers that use monitoring services that abuse bandwidth will have their accounts revoked, or be charge for the excess bandwidth used. There's no reason in the world why these people need to hit your servers as often as they are.
If you are unable to do business with your servers being hammered, then I suggest blackholing the monitoring service's IP's. It's only sensible.
I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
we typically set our monitor software to check every 5 minutes, with one request PER SERVER not per site. if it is down it will send an email to our support address, if it is STILL down the second time around, it fires off an email to the cell phone of the on-duty admin, plus one email when it comes back up
i've had some services set up for monitoring as low as 30 seconds, but those are specific cases.
obviously a 1 seconds check is WAY too low, not only it's a waste of bandwidth, it's prone to false positives. what happen when you have a slight delay in one of the core routers that cause your packet to get dropped/delayed by 1000ms ?
Here's a common sense reaction.
They are in the business of measuring Net availability. They should learn to set the scale on their instruments before they connect them to the circuit. And they should back off when availability drops because they might be the cause of the drop. If their traffic represents more than about 10x that caused by an individual customer, then as a "juror" I'd think they were being irresponsible.
You are in the business of supplying Net availability. You should install circuit breakers. Too many connection from one host/network? Start dropping packets. Too much raw incoming traffic from one source? Get on the horn quickly to the netadmin.
Your customers don't care who's at fault, they want what they paid for. But they can't expect miracles.
And I can tell you that if they're polling at 1 a second of *anything*, they don't "know what they're doing". That is complete overkill, there's no way the amount of bandwidth being used for testing is worth the 59-second jump on knowing what went wrong. Humans generally have to react to it, that kind of resolution is just crazy.
WWJD? JWRTFM!!!
A couple of years ago, a so-called "security expert" sold the president of my company on the idea of installing a firewall.
To some extent, that was fine with me. I'd been arguing for that for a very long time but had gotten nowhere because the "security expert" said that firewalls weren't necessary! I guess someone finally bothered to break into his system.
The security expert's idea was to have a third party monitoring company do it all. So I spent a couple hours on the telephone one day talking to the monitoring company's personnel about our network requirements and traffic. We went into great detail over exactly which servers had to handle which services.
The firewall arrived and the security expert plugged it in. It didn't work at all. All it did was block everything. I was 600 miles away at the time and it took me a week to convince them to take it off.
They decided the firewall was defective and the monitoring company set up another one. By the time it arrived, I was back in the office. The big day came and the security expert had one of his employees come out and plug it in.
It didn't work at all.
I caught the employee of the so-called security expert before he could leave the building and had him remove it. The idiot didn't even bother to check to see if it was working.
After he left the building, I started looking at how he had it plugged in. He still had a cable plugged into the firewall from an internal hub.
He had connected the untrusted side of the firewall to the internal network. I assume that the cable from the Cisco router was plugged into the trusted side of the firewall.
But it really didn't make much difference. I also found the rule set for the firewall. The monitoring company had set it to pass nearly everything in both directions.
The only thing they configured was to block incoming traffic containing our IP addresses. Since it was plugged in backwards, it really just stopped all traffic from going out.
At this point, it would take a lot of convincing to get me to advocate using a monitoring company's services.
By the way, the same so-called "security expert" declared that rules on the Cisco router to block traffic attempting to connect to port 135 and other similar ports constituted a security list and removed them.
Firewalling them is good, your customers have no authority to allow them that kind of access to your network. Have your corporate attorney send them a polite C&D letter. By polite, just the followup contact - this time on an attorney's letterhead. Also consult the attorney for what you should/can tell your customers, then do so immediately.
Be very clear to your customers that your objection is the nearly-criminal (it's a DOS) heavy-handedness, mind-numbingly unethical and pathetically incompetent behavior of the monitoring company. It's not unreasonable for one of your customers to retain a third party to provide professional services of this nature; by professional I mean 'do it right' not in the sense of professional as a term of law. Loading your website at regular intervals and parsing their logs for them is fine. Right now, these guys are probably reporting the outages they caused.
Billing your clients for bandwidth used by the monitoring company they hired is not completely unreasonable. Be sure to document every cost associated with this in every way, including time reading responses to this article as 'best practices research'. I'm not kidding, if you worked late you add the pizza in or the taxi home. Every penny in fine detail. Your lawyer will be keenly intereste, so might law enforcement if the polite C&D letter didn't do it.
Since the offered protection, aka monitoring services and then caused damage to your systems you could make a case that a protection racket is being run. If, adding in their fees for their services (paid by your customers) to the damages calculated above you have more than a certain threshold, probably US$50,000, then the FBI will be interested. Also have the monthly and annual total of your revenue from the customers either employing the monitoring service plus those affected by the damage cause (probably all of them). If things go sour with them and you do go to law enforcement, wave your revenue totals around to help get DAs and FBI interested.
Basically, you call your lawyer and then contact your customers. Your lawyer asks them to behave themselves. Then you meet with the lawyer, discuss the response and post another Ask Slashdot.
Veteran, Bermuda Triangle Expeditionary Force, 1992-1951
I'd think somebody would have noticed the high usage and firewalled off that site too. I mean jeez that must have been thousands and thousands of hits to use up that much space. I'd suspect a DoS attack if I saw that in my logs.
I also suggest anyone running servers to have some sort of program monitoring disk usage. If the disk gets dangerously low on space it should notify staff and take action such as rotating logs. Have the server page an admin or set an alarm off (where it'll be noticed) or something. Whatever you'd do if an attempted intrusion was detected. I usually have the server send warnings at 90% and 95% and at about 97% usage it should give me a good loud yell.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
Why are you putting up with this crap?
As several posters have already mentioned, firewall them off, and then report them to the legal authorities.
Jesus tap-dancing Christ! They are attacking your network. I feel like flaming the original poster for his incompetence. Acquire the BOFH nature. After you firewall them, file a report with the FBI's cybercrime division. Tell them you are a hosting company, and you have the IP of someone who is costing your company $BIGNUM dollars per day because they are DOS-ing your network. That should keep this "monitoring company" busy for a while, and it will teach them a lesson.
Whining about it on slashdot is the last thing you should be doing. Get a clue.
$ units bits/second bits/day
* 86400
So you're looking at (roughly) 100K hits per day per file downloaded per site. If they're downloading 15 files per site, and you've got 100 sites on the box, then you're looking at an increase of about 120 million requests per day. My acess log has an average of 200bytes/er line, so you're now looking at 120Mrequests*200bytes/request == a sudden jump of 24gigabytes of logging per day.
Then you've got the effective mail-bombing to deal with.
The article author said that these people sounded like they know what they're doing, so that leaves (in my mind), two likely possibilities:
- They're really really good snow-job artists. They understand the terminology, but they have no real sense of methodology or purpose.
- They really do know what they're doing, and they're trashing your servers with intent.
I mean -- for crying out loud: Multiple files once per second? And just how long did it take them to inform your customers that they'd managed to crash the servers? Monitoring granularity of more than about one quarter the normal notification time is a complete waste of resources -- and that's giving them lots of leeway to waste.And Tens of thousands of undelivered emails??? If those emails didn't get delivered, then what did the company do when they didn't arrive in short order? Why didn't they stop the transmission and diagnose why the emails weren't coming thru? If the emails really are undeliverable, then how in the world did you manage to conclude that they know what they're doing?
Other notes (mostly mentioned elsewhere)
-
are you charging your customers based on their net volume? If so, have you informed your customers of what sort of costs these, uhm, people are imposing on them in addition to their monitoring fees?
- I'm guessing that your AUP includes a clause on activities that wilfully or negligently cause inappropriate server load, outages, etc. I think that this company's "services" classifies.
- I think that you had better seriously consider possibility #2 above. Meticulously document what they've done to your servers (including somehow scamming your customer list). Have that information ready to present to your customers and/or a judge. If all goes well, you won't need it, but I'm not expecting all to go well, given how they've gone so far.
One last point -- Even though you may be dealing with a company that you think has a (otherwise) good reputation, doesn't mean that you're not dealing with an inept department of an otherwise good company. Sometimes the VP Engineering puts his/her stupid cousin in some group where they're not likely to do much damage, and then finds out that the goofball has managed to get out 'in the wild' with a 'bright' idea.Sometimes boldness is in fashion. Sometimes only the brave will be bold.