Slashdot Mirror
When Does Website Monitoring Go Too Far?
"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
They must be a way to enforce that they could check, say, only once every hour. And BTW, isn't your company missing an opportunity here? If you're already checking the servers, etc., why not make the tools available to the customers? They'll be more satisfied with the tools, and not having to pay the outside firm. You'll have more satisified customers and less churn....
Don't give a company of strangers the key to the front door. There's no reason someone from your company wasn't there to say 'when.' As for when too much is too much, it'd be when the efficiency of your main product is impaired to the point that you lose customers or reputation.
Banaaaana!
Their 'harvesting' your IP block is tacky at the least. That said, the current range of InternetSeer type monitoring is flat out overkill, and doesn't even work right half the time. According to some of them, my site is constantly down, but it *never* is. I know, since I'm an access_log nerd and always play with it; people are always going through it without any large 'dead' blocks appearing. All you need is a remote monitoring system to let you know when your major ports aren't functional, and to have it mail you ONLY when it's down. These 100k emails dripping with HTML to let you know that your site is still up are a complete waste of good bandwidth. Ping your damn site on your major ports, and that's all you need.
Dude, where's my packet?
They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers.
Sounds like you've got an open and shut legal case to recoup those costs they're causing you to incur.
NO CARRIER
At a choke point preferrably, that ought to get their attention rather quickly....they may then have issues with OTHER customers not on your network.
Jonah Hex
Horror & SciFi Erotic Nudes
Okay so you're telling me that a 3rd party company is contacting your web customers and selling them monitoring services that you already provide and some other services that you may or may not provide. They then begin to access your system to do said monitoring but it's crashing your servers.
:)
Lets put it this way.
You provide your customers a service. Part of that user agreement (This is doubly important in a shared server enviroment) that the customer cannot install any software/script/service that impacts the performance of the servers beyond what you say they can. Even the act of using 3rd party monitoring that is causing this problem is in violation of your AUP your customers are contractually binded to. Now I cant see your AUP but I hope there are provisions in there stating this.
Now as far as the 3rd party company goes. You need to have your legal department file a cease letter to them with a explanation of the problems they're causing and until things can be worked out they are not permitted to conduct business across your network.
You also need to notify your customers the actions you're taking on this company and why. Also pointing out your AUP/SLA's with them and the un acceptable behavior of the company that was selling them services. Tell them what you can monitor and explain what they really need.
In the assumption of a web/email then all you need to do is monitor the ports and maybe a script that will verify the email server is accepting connections on a minute basis. That's all you need for that setup. Also if they're allowed to telnet into the box (SSH I hope) then you'd also monitor the SSH port as well to ensure they can connect to their equipment.
If you're co-locating: Then I would suggest getting a Nagios setup running and sell some sort of monitoring to your customers. A good example would be the system that springboardhosting.com provides to their users. We use them as our colo partner and I've had no complaints. Though we only use the basic monitoring I do have advanced tools at the house and my laptop should I feel I need to watch any critical services. And I use webmin to monitor peer servers and page my phone in case there are any problems.
You're in a pickle at the moment but I think your customers will appreciate cutting off the source of the outages. Nobody needs to know if their service is up by the second unless it's some sort of huge database application and then you'd have special provisions to monitor it and not remotely.
That company is basically DDOS'g your servers to death. So it's basically them or you. I think the choice is simple
Hope that helps.
Remember they are WORKING FOR YOU.
If they cop some sort of we are smarter than you attitude, again, YOU ARE THE CUSTOMER, and YOU probably KNOW BETTER than they do, because YOU are in the business. They are just software vendors.There is no spoon or sig.
Buddy, you're living in denial. They've made a right mess of your services. Right? So their reputation doesn't mean a thing. If you'd mentioned their name (who are they?) they'd be suffering tomorrow after making the front page of Slashdot. You're discounting your own crediblity to judge if something is reasonable or not. From your description, their tools have already caused a denial of service attack on an email server.
My solution would be to attempt once more to get in touch with these goons. If they're still unresponsive, ban them permanently. Notify your customers that you do not wish your customers to use this service - and tell them why (because it is bad for your ability to provide them with the services that they've paid YOU for) - and that they should ask for a refund from this monitoring service.
If you think your customers feel this is a service that they need, you should look into providing some sort of monitoring system free for your customers (should not be hard if you have an in-house perl / python script wizard on hand - hell, I could do something like this in python in an afternoon).
Also, why do your customers feel the NEED for such a service? Are there any reliability issues that should be patched up with your network / services? Because there's no point fixing the symptoms if you don't fix the cause..
Another tactic would be to charge for the monitoring traffic. Surely your customers don't have unlimited bandwidth? Is the monitoring stuff being included in that bandwidth total? It damn well should be, status emails included. They'll see the light when the monitoring system eats up 80% of their monthly bandwidth.
Your system should have been set up to attribute the log file to the disk space of each client, causing them to eventually hit their limit and lose their abilty to log any further. No set of requests from the outside world should be able to bring down your server short of a vicious DOS attack, which clearly this wasn't. This was a an overload level of legit traffic, if your server can't handle it then you need a better server.
:)
You should be able to create a few new services and convince your clients that they don't need to pay a 3rd party to monitor their server, that you can tell them all they need to know, and besides that you don't go down anyway.
It would have been an absoulte fiasco if one of your customers were to attract a Slashdotting...
Monitoring your servers is a security function. A security company should strive to appear beyond reproach. Wether they got your customer list by looking through your ip logs or from a former employee, that is unsuitable behavior. I would contact my customers tell them that a security firm you do business with has "acquired" a customer list of yours and you are unsure of their intentions but you are sure that they acquired it dishonestly. None of your customers will hire them. The down side is, be careful not to tell your customers in a way that makes you look stupid, because you might look it.
It seems to me that unless your company signed some kind of waiver in case their monitoring did any damage, you have a case for negligence.
Even with a waiver, generally, you can't waive somebody's negligence. Their actions sound negligent in that they used excessive resources such that your servers crashed.
Additionally, it sounds like there may be some form of defamation claim when they complained to your customer base about you. Though defamation claims, especially slander (spoken defamation), are thorny claims that can be hard to prove, it sounds like you may have a number of incidents that may show intentional defamation (much better when seeking damages).
I think, at the very least, your general counsel should be asking for compensation for your downtime.
-A
They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers. They then proceeded to sell them monitoring services for things such as server up-time, defacement detection, email up-time and DNS testing.
In other words, they upsold your customers without your consent. That in itself it unethical and any thought in my mind that this is a 'reputable' company would go away at that point.
You go on to describe how they DoS'd your boxes, and complained to your customers when you took action to protect your customers from the DoS attack.
If their behavior is really as you described, why are you bending over backwords to say how reputable and legitimate they are? They are neither.
There should be no reason to add 3rd party security IF your security is in place. There are a lot of ways to protect your environment that do not require outside monitoring.
Alert your users of this fact - send them all an E-mail to alert them of this scam!
You run the show -- not some 3rd party. You set the rules and the security policies. You do the monitoring internally.
I can't believe that monitoring consumed 15GB of space. There's something else going on there. I helped work on a data warehouse to capture all of Worldcoms routers data every 5 minutes -- every router's SNMP logs and for years dumped all that data into an Oracle database so we could report on it. That's a bunch of routers and a ton of data. For your company to consume that much log data in a single weekend doesn't make sense.
Block the 3rd party polling IP at the routers and do the job internally.
Banjo - The more I know about Windoze, the more I love *nix
If I understand you right:
1. You have some customers to which you sell services such as email and web space.
2. Some of these customers contracted this monitoring service to watch the servers.
3. The monitoring service caused problems with your servers.
And the answer is:
Correct your hosting contract. Your hosting contract should include provisions for how much usage is reasonable and how the situation will be handled when the customer's usage exceeds those parameters. If the customer insists on doing something stupid which brings the server to its knees, then the customer should pay you enough for you to be able to afford a seperate server for them.
If the sales force insisted that they'd lose sales by bothering the customer with such notions, now would be an excellent time to point out that they just lost sales because they didn't.
As to how much monitoring is too much, the answer is simple: anything the customer is willing to pay for is fine. Anything more is too much.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Why are you putting up with this crap?
As several posters have already mentioned, firewall them off, and then report them to the legal authorities.
Jesus tap-dancing Christ! They are attacking your network. I feel like flaming the original poster for his incompetence. Acquire the BOFH nature. After you firewall them, file a report with the FBI's cybercrime division. Tell them you are a hosting company, and you have the IP of someone who is costing your company $BIGNUM dollars per day because they are DOS-ing your network. That should keep this "monitoring company" busy for a while, and it will teach them a lesson.
Whining about it on slashdot is the last thing you should be doing. Get a clue.
Ha, tell that to all the webmasters with non-compliant HTML out there.
I just thought of a good idea, a web page upload form or something which scans the webpages which gives a nice little dialog about a webpage being non compliant, and may not display correctly in many browsers :) Now to get ANY ISP to implement it... HA!
"Oh dear God! You've been pinged! The sky is falling!! Whew. It's a damn good thing you installed our over-priced over-hyped personal firewall thingy because we just saved your ass!"
Think I'm kidding? Don't. These ass clowns prey on guilible users that simply don't know any better. It's just like what many auto repair shops do to those people whom they don't think know jack about cars.
The belt on your carburetor are about to break. We also had to grease you exhaust bearings and reprogram your warp convertors. That'll be $700 please.
If only we can eliminate stupid people and those that would prey on them (including the media) the world would be a much better place.
One of the biggest problems with monitoring something is that you inevitably affect it, a la Heisenberg in the Physics world. The more closely you try to monitor something, the more you affect it. This is a basic principle of monitoring.
Absolutely. This isn't monitoring--this is load testing.
But they can't expect miracles.
;-)
Of course they can, and do. They won't get them, but that's different.
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
$ units bits/second bits/day
* 86400
So you're looking at (roughly) 100K hits per day per file downloaded per site. If they're downloading 15 files per site, and you've got 100 sites on the box, then you're looking at an increase of about 120 million requests per day. My acess log has an average of 200bytes/er line, so you're now looking at 120Mrequests*200bytes/request == a sudden jump of 24gigabytes of logging per day.
Then you've got the effective mail-bombing to deal with.
The article author said that these people sounded like they know what they're doing, so that leaves (in my mind), two likely possibilities:
- They're really really good snow-job artists. They understand the terminology, but they have no real sense of methodology or purpose.
- They really do know what they're doing, and they're trashing your servers with intent.
I mean -- for crying out loud: Multiple files once per second? And just how long did it take them to inform your customers that they'd managed to crash the servers? Monitoring granularity of more than about one quarter the normal notification time is a complete waste of resources -- and that's giving them lots of leeway to waste.And Tens of thousands of undelivered emails??? If those emails didn't get delivered, then what did the company do when they didn't arrive in short order? Why didn't they stop the transmission and diagnose why the emails weren't coming thru? If the emails really are undeliverable, then how in the world did you manage to conclude that they know what they're doing?
Other notes (mostly mentioned elsewhere)
-
are you charging your customers based on their net volume? If so, have you informed your customers of what sort of costs these, uhm, people are imposing on them in addition to their monitoring fees?
- I'm guessing that your AUP includes a clause on activities that wilfully or negligently cause inappropriate server load, outages, etc. I think that this company's "services" classifies.
- I think that you had better seriously consider possibility #2 above. Meticulously document what they've done to your servers (including somehow scamming your customer list). Have that information ready to present to your customers and/or a judge. If all goes well, you won't need it, but I'm not expecting all to go well, given how they've gone so far.
One last point -- Even though you may be dealing with a company that you think has a (otherwise) good reputation, doesn't mean that you're not dealing with an inept department of an otherwise good company. Sometimes the VP Engineering puts his/her stupid cousin in some group where they're not likely to do much damage, and then finds out that the goofball has managed to get out 'in the wild' with a 'bright' idea.Sometimes boldness is in fashion. Sometimes only the brave will be bold.
p.s. Why is using perl funny?
How about partitioning your servers properly so they don't crash when they fill the logs?
Basic sysadmin 101, people. You're going to piss off customers by doing what the parent suggests.
Please help metamoderate.
If the customer is only paying $9.95/month for the site they either have traffic limits set in terms of rate of activity (i.e. your site will never send out at a speed that would tax a 28.8 modem-- this is not a common approach, if it's used at all) or total periodic bandwidth allowance (you can't transfer more than a set GB limit a month without paying extra). Some script-allowed hosts will also set CPU limits on CGIs.
This rate of monitoring is no way going to come in under the transfer caps at the end of the month and these discount hosting customers would get SCREWED in terms of their bill, I'd think. Or maybe they deserve those bills for being so braindead about the impact of the monitoring service on the servers and the network.
What this really smells like is bad admin in terms of log size/rotation policy. Never once did the poster mention that there was a choke on transfer rate-- rather that the servers went down due to software crashes (running out of disk/RAM can do that, no?).
I would lay blame on everyone involved, personally. The "monitoring company" for being a crappy service and not working with the ISP. The ISP for having bad server management policies that lead to crashing during perfectly predictable events (what happens if one of their customers gets Slashdotted? would their logs have been able to handle that, too?). And the customers who arranged for this monitoring without talking to the ISP about it first or at least properly understanding the impact it would have on the network.
I do not have a signature