Slashdot Mirror
When Does Website Monitoring Go Too Far?
"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
They must be a way to enforce that they could check, say, only once every hour. And BTW, isn't your company missing an opportunity here? If you're already checking the servers, etc., why not make the tools available to the customers? They'll be more satisfied with the tools, and not having to pay the outside firm. You'll have more satisified customers and less churn....
What are these guys selling?
Are they running things on your servers?
Are they running your servers?
Did you hire them to do this?
Don't give a company of strangers the key to the front door. There's no reason someone from your company wasn't there to say 'when.' As for when too much is too much, it'd be when the efficiency of your main product is impaired to the point that you lose customers or reputation.
Banaaaana!
A server should not choke if the log partition is full. Is the log in a separate partition, isn't it?
If not, and they screw up the server, then it's their own fault. Warn them, don't warn them, whatever... just make sure you charge appropriately when they screw up their servers.
Their 'harvesting' your IP block is tacky at the least. That said, the current range of InternetSeer type monitoring is flat out overkill, and doesn't even work right half the time. According to some of them, my site is constantly down, but it *never* is. I know, since I'm an access_log nerd and always play with it; people are always going through it without any large 'dead' blocks appearing. All you need is a remote monitoring system to let you know when your major ports aren't functional, and to have it mail you ONLY when it's down. These 100k emails dripping with HTML to let you know that your site is still up are a complete waste of good bandwidth. Ping your damn site on your major ports, and that's all you need.
Dude, where's my packet?
They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers.
Sounds like you've got an open and shut legal case to recoup those costs they're causing you to incur.
NO CARRIER
Let's all pitch in on a little scheme. We will each agree to buy a service plan to have one non-existant .com web site monitored. If we could get lots of people to do this, we could DDOS Verisign off the internet!
Wh47 d1d j00 541, 31337 15n't t3h r0xor5 ne m0r3???
Nagios.
http://www.nagios.org/
Cheers
Stor
"Yeah well there's a lot of stuff that should be, but isn't"
When it exceeds the point of being far enought!
Kind of depends on how rapidly you can respond to a problem with something being monitored - obviously every second or even every minute is too rapid. Every hour sounds better.
$2B OR NOT $2B = $FF
At a choke point preferrably, that ought to get their attention rather quickly....they may then have issues with OTHER customers not on your network.
And anyone who is a lawyer, is denied access to all computing systems.
This is my sig.
...that either don't have the time / money to go after people like this, such as the webhotel I'm involved in in my sparetime, I'd recommend firewalling. Simply block all incoming connections from over zealous monitor-companies.
Ofcourse this doesn't do anything to fix the bad reputation they may have given you by flooding your servers, but its a quick and easy antidote against future problems.
From your description, i.e. "Once per second", that is quite beyond monitoring, and that is an EXCESSIVE use of bandwidth and resources.
Now, if you charge your customers based on gigs transferred, it seems like this would fill up their quota for the month quite quickly. What are your customers going to think when they get a large overcharge bill for the bandwidth? They signed up for the service after all.
If you aren't hosting for money, then you probably aren't able to profit from this monitoring companies actions in the same way, so I suggest you blackhole their ip's. Downloading files from your server once per second goes way beyong monitoring, and into the realms of denial of service(It crashed your server you say).
What I would do? Make a change to the aup for your service stating that customers that use monitoring services that abuse bandwidth will have their accounts revoked, or be charge for the excess bandwidth used. There's no reason in the world why these people need to hit your servers as often as they are.
If you are unable to do business with your servers being hammered, then I suggest blackholing the monitoring service's IP's. It's only sensible.
I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
we typically set our monitor software to check every 5 minutes, with one request PER SERVER not per site. if it is down it will send an email to our support address, if it is STILL down the second time around, it fires off an email to the cell phone of the on-duty admin, plus one email when it comes back up
i've had some services set up for monitoring as low as 30 seconds, but those are specific cases.
obviously a 1 seconds check is WAY too low, not only it's a waste of bandwidth, it's prone to false positives. what happen when you have a slight delay in one of the core routers that cause your packet to get dropped/delayed by 1000ms ?
... that you waited as long as you did.
;)
I don't care who they are - they frell up my servers they're gettin' buckshot in the ass first, and the ones who can still walk answer questions later.
Of course, that attitude may seem a little harsh in some circles, but in other circles they substitute AK-47s for the buckshot... so who's the extremist here?
Mnem
"It's a thankless job, but I have a lot of Karma to burn off."
Jonah Hex
Horror & SciFi Erotic Nudes
...your boss (or someone higher) is found to be browsing "questionable" sites.
In that case, jobs are at jeopardy if word gets out- and i *don't* mean your boss's job...
Here's a common sense reaction.
They are in the business of measuring Net availability. They should learn to set the scale on their instruments before they connect them to the circuit. And they should back off when availability drops because they might be the cause of the drop. If their traffic represents more than about 10x that caused by an individual customer, then as a "juror" I'd think they were being irresponsible.
You are in the business of supplying Net availability. You should install circuit breakers. Too many connection from one host/network? Start dropping packets. Too much raw incoming traffic from one source? Get on the horn quickly to the netadmin.
Your customers don't care who's at fault, they want what they paid for. But they can't expect miracles.
I find it difficult to imagine that the unnamed monitoring company hasn't overstepped its bounds. Perhaps you should send a calm e-mail to your customers explaining the issue at hand. Compare the once-per-second monitoring to calling a call center every second to check that it still works, or perhaps walking in and out through the front door of a store just to check if it still works. Both things are fine in moderation, but every second is entirely too much.
The monitoring company may have crossed some legal lines as well. Have your company's lawyer review your state's relevant laws and send a polite but firm cease-and-desist to the monitoring company, threatening relevant legal action. Don't insist that they stop alltogether, but suggest they exercise the legally-ambigious "common sense" and "good judgement" the rest of us seem to have figured out.
And I can tell you that if they're polling at 1 a second of *anything*, they don't "know what they're doing". That is complete overkill, there's no way the amount of bandwidth being used for testing is worth the 59-second jump on knowing what went wrong. Humans generally have to react to it, that kind of resolution is just crazy.
WWJD? JWRTFM!!!
from first hand experience, application monitoring is far more important than stupid webserver monitoring. It makes absolutely no sense to monitor the DNS and webserver beyond what most ISP's provide. Why in the world they are monitoring the domain once a second is beyond me. that is the lowest risk in my mind. monitoring application performance, server load, and mean process time of the application is very useful. From the details provided, it doesn't sound like they know what they are doing.
Okay so you're telling me that a 3rd party company is contacting your web customers and selling them monitoring services that you already provide and some other services that you may or may not provide. They then begin to access your system to do said monitoring but it's crashing your servers.
:)
Lets put it this way.
You provide your customers a service. Part of that user agreement (This is doubly important in a shared server enviroment) that the customer cannot install any software/script/service that impacts the performance of the servers beyond what you say they can. Even the act of using 3rd party monitoring that is causing this problem is in violation of your AUP your customers are contractually binded to. Now I cant see your AUP but I hope there are provisions in there stating this.
Now as far as the 3rd party company goes. You need to have your legal department file a cease letter to them with a explanation of the problems they're causing and until things can be worked out they are not permitted to conduct business across your network.
You also need to notify your customers the actions you're taking on this company and why. Also pointing out your AUP/SLA's with them and the un acceptable behavior of the company that was selling them services. Tell them what you can monitor and explain what they really need.
In the assumption of a web/email then all you need to do is monitor the ports and maybe a script that will verify the email server is accepting connections on a minute basis. That's all you need for that setup. Also if they're allowed to telnet into the box (SSH I hope) then you'd also monitor the SSH port as well to ensure they can connect to their equipment.
If you're co-locating: Then I would suggest getting a Nagios setup running and sell some sort of monitoring to your customers. A good example would be the system that springboardhosting.com provides to their users. We use them as our colo partner and I've had no complaints. Though we only use the basic monitoring I do have advanced tools at the house and my laptop should I feel I need to watch any critical services. And I use webmin to monitor peer servers and page my phone in case there are any problems.
You're in a pickle at the moment but I think your customers will appreciate cutting off the source of the outages. Nobody needs to know if their service is up by the second unless it's some sort of huge database application and then you'd have special provisions to monitor it and not remotely.
That company is basically DDOS'g your servers to death. So it's basically them or you. I think the choice is simple
Hope that helps.
I think anyone should be allowed to monitor on your clients behalf however they want. You should bill them for the added bandwidth incurred by the monitoring software, just like you'd bill them for amu bandwidth, unless you have some "unlimited bandwidth" deal, in which case your clients will probably realize the tradeoff if the monitoring software is chewing half of their practical limit.
As for your damaged reputation, if you don't have monitoring software in place to tell you when your log partition is filling up, and/or some software in place to dump the logs before they kill a system, then that is your fault. Your reputation was damaged deservedly. I mean yeah the other company caused it, but it could just as easily have been a press blurb touting the site, or a slashdot reference to a site, or something else that caused the log overload. If you're supposed to be managing the web servers, you made a mistake. Correct it, explain it, but apologize - it was your fault.
From a business perspective, monitoring is a service *you* should offer to your customers. Since it is your network, you have the ability to provide a much more effective and accurate monitoring service, and can set the resolution of the service according to your customers needs. All the problems you describe are because they are operating from the outside. What that monitoring service is effectively doing is stealing your bandwidth, and selling to your customers. If you want to get your lawyers involved, send them a C&D since they are affecting your ability to conduct business. personally I would firewall then as the CTO has done, and offer the same service internally.
A couple of years ago, a so-called "security expert" sold the president of my company on the idea of installing a firewall.
To some extent, that was fine with me. I'd been arguing for that for a very long time but had gotten nowhere because the "security expert" said that firewalls weren't necessary! I guess someone finally bothered to break into his system.
The security expert's idea was to have a third party monitoring company do it all. So I spent a couple hours on the telephone one day talking to the monitoring company's personnel about our network requirements and traffic. We went into great detail over exactly which servers had to handle which services.
The firewall arrived and the security expert plugged it in. It didn't work at all. All it did was block everything. I was 600 miles away at the time and it took me a week to convince them to take it off.
They decided the firewall was defective and the monitoring company set up another one. By the time it arrived, I was back in the office. The big day came and the security expert had one of his employees come out and plug it in.
It didn't work at all.
I caught the employee of the so-called security expert before he could leave the building and had him remove it. The idiot didn't even bother to check to see if it was working.
After he left the building, I started looking at how he had it plugged in. He still had a cable plugged into the firewall from an internal hub.
He had connected the untrusted side of the firewall to the internal network. I assume that the cable from the Cisco router was plugged into the trusted side of the firewall.
But it really didn't make much difference. I also found the rule set for the firewall. The monitoring company had set it to pass nearly everything in both directions.
The only thing they configured was to block incoming traffic containing our IP addresses. Since it was plugged in backwards, it really just stopped all traffic from going out.
At this point, it would take a lot of convincing to get me to advocate using a monitoring company's services.
By the way, the same so-called "security expert" declared that rules on the Cisco router to block traffic attempting to connect to port 135 and other similar ports constituted a security list and removed them.
Remember they are WORKING FOR YOU.
If they cop some sort of we are smarter than you attitude, again, YOU ARE THE CUSTOMER, and YOU probably KNOW BETTER than they do, because YOU are in the business. They are just software vendors.There is no spoon or sig.
Buddy, you're living in denial. They've made a right mess of your services. Right? So their reputation doesn't mean a thing. If you'd mentioned their name (who are they?) they'd be suffering tomorrow after making the front page of Slashdot. You're discounting your own crediblity to judge if something is reasonable or not. From your description, their tools have already caused a denial of service attack on an email server.
My solution would be to attempt once more to get in touch with these goons. If they're still unresponsive, ban them permanently. Notify your customers that you do not wish your customers to use this service - and tell them why (because it is bad for your ability to provide them with the services that they've paid YOU for) - and that they should ask for a refund from this monitoring service.
If you think your customers feel this is a service that they need, you should look into providing some sort of monitoring system free for your customers (should not be hard if you have an in-house perl / python script wizard on hand - hell, I could do something like this in python in an afternoon).
Also, why do your customers feel the NEED for such a service? Are there any reliability issues that should be patched up with your network / services? Because there's no point fixing the symptoms if you don't fix the cause..
Another tactic would be to charge for the monitoring traffic. Surely your customers don't have unlimited bandwidth? Is the monitoring stuff being included in that bandwidth total? It damn well should be, status emails included. They'll see the light when the monitoring system eats up 80% of their monthly bandwidth.
I work for a company that builds hardware to facilitate http traffic to web servers. Customers of ours are constantly calling and saying
'We just hired a 3rd party to do a security audit on our network and they reported that port 80 is open on your box. They said that is a security issue and should be closed.'
I then have to explain how fscking difficult it is to run a web farm without allowing port 80 traffic through!!!!
I see this 'reporting' company doing something very similar. They focus on one, and only one, small facet of the big picture. And because of this, they over-extend their capabilities well beyond usefullness.
But they feel the more they report, the better they are than their competitors, and the more you, the end customer, will like it.
When in all actuality, uptime and overall accesibility of a site are all that a lot of webmasters care about.
See if it say they can do what they're doing. If it doesn't, tell them to stop or you'll take them to court. If it does, tell them to stop or you won't renew the contract when it ends.
It sounds like they're doing a bunch of stuff that's not strictly necessary for them to do their job. It may convenient for them to scan your servers every second, but if that impacts your business, they need to stop. They're supposed to be there to enhance your business, not impede it.
And if they snagged a copy of your customer database to peddle their own products.. that's just plain wrong. Check the contract again, and if it doesn't explicitly sat they can do that, take them to court.
I appreciate your position that they do a good job, but I'm sure that there are other companies out there that can do an equally good job without the downside that you're having with your current vendor.
I am NOT a man!
I am a free number!
Your system should have been set up to attribute the log file to the disk space of each client, causing them to eventually hit their limit and lose their abilty to log any further. No set of requests from the outside world should be able to bring down your server short of a vicious DOS attack, which clearly this wasn't. This was a an overload level of legit traffic, if your server can't handle it then you need a better server.
:)
You should be able to create a few new services and convince your clients that they don't need to pay a 3rd party to monitor their server, that you can tell them all they need to know, and besides that you don't go down anyway.
It would have been an absoulte fiasco if one of your customers were to attract a Slashdotting...
It seems there has been an unusual amount of downtime to your web and email servers. Probable cause: we over monitored them. Sorry.
"She's a West Texas girl, just like me" - G.W Bush Iraqis
First things first. These are your servers. Your network. I am assuming you have the standard abuse clause in your TOS. You need a lawyer.
Unfortunately, you are in a bad situation. They apparently have more resources than you, because they can bring your setup to it's knees. Not saying it's right, not saying it's fair.
A lookup of your TLDs each second makes sense if you are Yahoo! or Google. Their web monitoring levels don't appear to be reasonable. You already know the technical answer.
Personally, I would be worried about them stealing your customers. I mean the argument is going to be simple from their side. They will simply say, "hey look, their stuff folded under 'normal' monitoring, we have a hosting company we can 'recommend'" or they will just have the hosting company call them up out of the blue and ask if they are "unhappy" with thier current service..."oh, it goes down a lot"..."they can't handle simple monitoring"..."gee, that's a shame"..."well, we've worked with that monitoring company before, and we have never had any problems, in fact we routinely get 5 9s"...etc
Honestly, talk to legal, explain the potential situation, and have them make contact with the monitoring company. A couple of tortious interference this, and cease and desist that, will put the monitoring company on it's toes and maybe get them to leave your customers alone, or possible play nice with your servers. Notify your customers yourself and explain that they are being investigated by your legal team, etc.
No.
These guys don't know what they're doing if they are banging on your servers every second. It is a strategy that is bound to make any competent admin irate and probably break things. Anything more than once-a-minute is probably overkill. Once every 5 minutes is a good window for most things. Your people are quite entitled to block them at the firewall.
Your sales people have to figure out how to appease the customers. That's their job. You are a tech and you'll just foul things up using tools like fairness and logic. I've been there.
Lastly, if they overflowed your log partition, you aren't monitoring enough things. It isn't enough to make sure that your sites are up, you need to make sure that the disks they depend on have enough free space, that the servers they run on don't have unnacceptible load spikes, etc... Comprehensive solutions are hard, but quick-and-dirty solutions aren't. Remember though that it's hard to send pages from a dead server and design accordingly.
Monitoring your servers is a security function. A security company should strive to appear beyond reproach. Wether they got your customer list by looking through your ip logs or from a former employee, that is unsuitable behavior. I would contact my customers tell them that a security firm you do business with has "acquired" a customer list of yours and you are unsure of their intentions but you are sure that they acquired it dishonestly. None of your customers will hire them. The down side is, be careful not to tell your customers in a way that makes you look stupid, because you might look it.
It seems to me that unless your company signed some kind of waiver in case their monitoring did any damage, you have a case for negligence.
Even with a waiver, generally, you can't waive somebody's negligence. Their actions sound negligent in that they used excessive resources such that your servers crashed.
Additionally, it sounds like there may be some form of defamation claim when they complained to your customer base about you. Though defamation claims, especially slander (spoken defamation), are thorny claims that can be hard to prove, it sounds like you may have a number of incidents that may show intentional defamation (much better when seeking damages).
I think, at the very least, your general counsel should be asking for compensation for your downtime.
-A
Poor monitoring can be just as bad as too much, though too much monitoring isn't necessarily poor.
It seems that the processes regarding monitoring and maintenance of the monitoring system(s) failed and caused the problems which ensued. If the proper preparation had been done to plan for the level of monitoring which was being done, i.e. 18 GB is didly squat compared to the hundreds of GB at many other enterprise sites, then this likely wouldn't have happened. Like wise, proper levels of logging and tuning are required to have a truly healthy and useful monitoring system.
Also level of service, SLAs mainly, dictate what level of monitoring is also required. It's very easy to go crazy monitoring everything, but you can monitor hundreds of servers and not generate much in the way of logs, or you can monitor one and generate many GB of logs.
--SuperBug
Try it, type in a non existant .com, it no longer works.
Everything seemed to be going so nice
'till the end of all beings punched right through the ice
The monitoring company just hit you with a Denial of Service attack. Plain and simple.
Now the next step is not technical, but legal. SLAP 'EM WITH A LAWSUIT WORTH MORE THAN THEY'RE EVER MAKE!!!
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
If there was damage to your business, the simply take them to court. More than likely with that kind of leverage you can come to some sort of agreement and drop the case once they come to a new agreement or give you a settlement and you find a different company.
“Common sense is not so common.” — Voltaire
I just checked again and non existant .com addresses still resolve to Verisign. The trick is that your ISP may have blocked it. I'm on a university network that has blocked it. However, when I log into a remote machine and use lynx, non existant pages still resolve to Verisign. Also, keep in mind that this is only for .com and .net addresses.
Wh47 d1d j00 541, 31337 15n't t3h r0xor5 ne m0r3???
They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers. They then proceeded to sell them monitoring services for things such as server up-time, defacement detection, email up-time and DNS testing.
In other words, they upsold your customers without your consent. That in itself it unethical and any thought in my mind that this is a 'reputable' company would go away at that point.
You go on to describe how they DoS'd your boxes, and complained to your customers when you took action to protect your customers from the DoS attack.
If their behavior is really as you described, why are you bending over backwords to say how reputable and legitimate they are? They are neither.
This is not legal advice. Find a lawyer, ask them what to do.
It seems as though you've got a tort of negligence on your hands, insofar as they seem unaware, or oblivious to, the damages they are causing you. They do not seem, from your statements, to be wilfully causing damages, but negligence torts need not show (at least in the commonwealth) either wilfulness or intent. You need only show damages, which are an indirect consequence of their actions.
Take into account that torts are, by most accounts, very expensive, though the threat of a tort is often sufficient, or binding arbitration (though that is apparently not oft met with success), or mediation (same deal as binding arbitration). If you do have to litigate, the general rule is somewhere north of $100,000 in damages to justify the transaction cost, from what I have heard. See the first line, though - find a lawyer.
In the least you can establish damages in support of a trespass if you inform them that their actions cause damage, in which case their actions are thereafter wilful, which may make for a cleaner case. The onus in trespass is on the defendent (them) to defend against damages established, not the plaintiff (you); and whereas in negligence, the onus is on the plaintiff (you) to show damages.
Ok, so in gist, take everything I said with a grain of salt, and seek legal counsel. Your jurisdiction may have many options with respect to small claims or public dispute resolution, and I would suggest those because they are significantly cheaper.
Hope that helps.
Charge for it. Notify yer customer (by perl of course *tee hee*) that their logs are causing their account to approach its space limit. They can either move the logs, delete the logs, stop the logging software or remove the logging software. Warn them that if this is not taken care of additional hd space fees will apply.
Make sure they know that cleaning up logs should be *cough* easy and pain free!
Not quite such an open and shut case. They produce an equal number of accounts that they have with users on GeoCities that are receiving the same level of "testing", and then what do you do?
Set usage policy with a fee/penalty structure, and hold them to it. Ignorance of consequences is no excuse for filling the server with logfiles.
Charge the security consulting firm with your downtime expenses too... They may refuse to pay, but simply getting the invoice may make them think twice about doing that to you again.
There should be no reason to add 3rd party security IF your security is in place. There are a lot of ways to protect your environment that do not require outside monitoring.
Alert your users of this fact - send them all an E-mail to alert them of this scam!
You run the show -- not some 3rd party. You set the rules and the security policies. You do the monitoring internally.
I can't believe that monitoring consumed 15GB of space. There's something else going on there. I helped work on a data warehouse to capture all of Worldcoms routers data every 5 minutes -- every router's SNMP logs and for years dumped all that data into an Oracle database so we could report on it. That's a bunch of routers and a ton of data. For your company to consume that much log data in a single weekend doesn't make sense.
Block the 3rd party polling IP at the routers and do the job internally.
Banjo - The more I know about Windoze, the more I love *nix
If there was damage to your business, the simply take them to court. More than likely with that kind of leverage you can come to some sort of agreement and drop the case once they come to a new agreement or give you a settlement and you find a different company.
How do we get RTFA failures on Ask Slashdot when it's on in the page. This guy didn't ask for this company... his webhosting customers did. He's got to convince his customers that this monitoring company isn't worth their money, because it overmonitors to the point that it creates its own downtimes.
I would guess the web monitoring company also sells webhosting services
(or a partner they recommend)
The fact that they contacted all your clients should send up a red flag.
Treat it as a DDOS/Hack Attack.
Inform/educate your customers.
I am the unwilling control for my Origin.
If I understand you right:
1. You have some customers to which you sell services such as email and web space.
2. Some of these customers contracted this monitoring service to watch the servers.
3. The monitoring service caused problems with your servers.
And the answer is:
Correct your hosting contract. Your hosting contract should include provisions for how much usage is reasonable and how the situation will be handled when the customer's usage exceeds those parameters. If the customer insists on doing something stupid which brings the server to its knees, then the customer should pay you enough for you to be able to afford a seperate server for them.
If the sales force insisted that they'd lose sales by bothering the customer with such notions, now would be an excellent time to point out that they just lost sales because they didn't.
As to how much monitoring is too much, the answer is simple: anything the customer is willing to pay for is fine. Anything more is too much.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Why let a third party come in in the first place? When you can spend a bit of time proactively to make your customers happy and avoid the whole mess.
If they want to pay for a service which pings them once a second let them, just remember they pay you money to host them, not to tell them what they can and can't do. as for up your logs, obviously your own monitoring is lacking, as nagios/mon/whatever should be able to alert you when a disk gets 75% full. note: I don't think that the monitoring service is effective don't get me wrong, i'm just saying you shouldn't be butting in here. what you could do: - change your acceptable use policy - provide bandwidth caps/megabyte charging - provide a better monitoring service to your clients (so they don't need this shit) - upgrade your machines (disks are cheap) so that your servers don't fall over - upgrade your access reporting script to filter out high-user IP's or robots (a nice value add?) - educate your users on what a proper monitoring service is like.
It's a DoS attack on your systems and should be treated as such (especially as you warned them already and they ignored you).
Block them off and take them to court.
-uso.
What you hear in the ear, preach from the rooftop Matthew 10.27b
If they're letting their logs get huge before rotating them it would cause a problem every time the server tries to append data at the end of the file.
And they shouldn't be keeping the logs on the server anyway. It's static data that only they could need access to. It should be moved off site to a standard IDE harddrive for processing.
Statistical data should be created as the data comes in and not from the log files if they intend to let the customers have statistics for whatever.
As for my own site, I have Apache doing the combined log format and wrote custom software to process and analyze the data. Every month I move the log off the server and every 10 megs or so I rotate the logs and move the data into a second cumulative file that Apache doesn't work off of.
Ben
Work Safe Porn
we are authorized to do what is necesary should a client or an outside agencies' actions compromise the integrity of the network system as a whole. Let them do it and bill the customer for the overage...then offer the services they are trying to get MUCH cheaper and also ensure them that they data transfer done in that manner will NOT count against their cap....however should they choose to pursue a 3rd party solution, which is well within their rights they will have to make arrangements to stay within their alloted range and NOT impact other customers or pay the price :)
The 3rd parties behavior in this case is really rather poor
Your customers want monitoring.
Some outside firm wants to do it.
So what's the real problem here?
Costs for bandwidth and storage.
Thus the simple solution:
tell them both you simply
need your costs covered.
Everyone wins, you look like a hero,
and you save yourself from lawyers.
Good luck! -Joel
It is very important for a bigger hosting firm to have a good moniroting strategy which shows trhe external perspective.
The timing need not be more than 15 minutes in most cases. The plan should include the network, web server and applications, and possibly supporting servers such as email or DNS.
The external capabilities are critical - if you are going to do external, use a firm who has profressional managed remote stations in many places.
Tim Goeke
http://www.globalnetwatch.com
gTLDs (Generic Top-Level Domains):
ccTLDsCountry-Code Top-Level Domains:
Why are you putting up with this crap?
As several posters have already mentioned, firewall them off, and then report them to the legal authorities.
Jesus tap-dancing Christ! They are attacking your network. I feel like flaming the original poster for his incompetence. Acquire the BOFH nature. After you firewall them, file a report with the FBI's cybercrime division. Tell them you are a hosting company, and you have the IP of someone who is costing your company $BIGNUM dollars per day because they are DOS-ing your network. That should keep this "monitoring company" busy for a while, and it will teach them a lesson.
Whining about it on slashdot is the last thing you should be doing. Get a clue.
"Is this typical behavior of monitoring service companies?" - yes and no. First, I have to ask why do you give the control for an outside company - aren't security and customer monitoring important for the company ? Second ( of course this is Slashdot but.. ), %90+ of replies are offtopic so don't even bother to read the answers ( mine included, slashdot! ). This question belongs to sysadmin Q&A or so. Now the answer ( my opinon ) - fire who ever made the contract with that company ( I know, I know, someone up in foodchain ). If that is not possible read BOFH on http://www.theregister.co.uk/.
ok ok...about the only thing I find remotley factual in this article is the fact that this guy works for a 'company'...however it looks like he works for a company doing exactly the things he is asking about.
First of all, lack of any knowledge of partion or disk utilities to prevent such an occurance is unacceptable. I would not admit that in public about my company even if I used the phrase 'a company I work for', just on the off chance my negligence would be able to be tracked back to me.
Second, why are you not able to offer these services yourself? You make a claim that these people know what they are doing, so if you are at such a level to recognise what they are doing, how come you havent done it already? Did customer service become just a novelty to you? so I doubt this line very much... While I welcome anything that lets our customers use the internet effectively
Doing hosting myself, Im well aware of the tactics you speak of, being that I get bounce mail for nonexistant addresses sent to such titles as; president, ceo, owner, support, tech...and so on. And Im not sure exactly what you mean by 'choked up' your mail server. How do 40k NONEXISTANT addresses manage to slow down your mail server? Is it a 286?
The whole article just smells funny to me, as it seems like you are just pretending to care about the ISP's end and more concerned about the backlash of doing these things. What do you mean how far is to far? Again, if the people in charge cant figure these things out on their own, I would be very hesitant to admit that in a public forum.
Get your technical skills and decision making in line...THEN question how to outsource it..
C'mon! You wanted to be in this field and now you are not using your brain / talents to work this out with your co-workers? Why give up the only fun you'll have?
No wonder so many jobs are going to India.
They are doing checks once per second? Are they retarded?
Sounds like some jackass got ahold of some monitoring software and decided to make some money with it, but doesn't know much about monitoring.
Need Free Juniper/NetScreen Support? JuniperForum
If your server/system uptime is business critical then you can and should have the systems in place on your end to tell whether you're up or not. It's not in your best interests to give these people business.
I used to run a small site, www.?bm.com, which got nailed by 100s of bozos running "monitoring" packages, all of which we blocked. We had a tripwire like tool that warned us if someone was hitting the servers too hard. If you exceeded some periodic limit you got automatically blocked. Didn't matter whether you were intentionally attacking the site or not, our assumption was that you were.
It's your systems, your resources, your business, not theirs.
In any pissing contest the winner is the one who can piss the furthest. End of story.
My Blog
...if they're selling some service (any service) to your customers, then it seems to me that they should be writing the logs on their own machines rather than on yours. After all, how can your customers (now their customers) be sure that you haven't doctored the logs if they store them on machines that you control?
I agree with others: You should be selling this service to your customers, and they should go take a flying leap.
"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
They are not a reputable company. They are a bunch of retards who should be driven out of the industry with sharp sticks. More to the point, they should be reported to the FBI for conducting a malicioius attack against your network - and you have tangible damage to prove it.
Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
Here's a hint for you: Do they offer web hosting services themselves? You may have to dig real deep to find the connection, but if I had to guess, I'll bet they do. And I'll bet they offer it to your customers, based on the fact that they crashed your servers. "Your current service seems to have a lot of downtime. Perhaps you should consider moving to another host. We can make recommendations."
If you find any evidence that they offer any kind of competition to your hosting, report them to the FBI. They may well be a criminal organization engaging in a well orchestrated scam.
Or maybe they're just fucking stoopid. It's hard to tell from here.
...so restrict as you see fit.
I'm all for customers taking control of what they need to, but you should have a standard set of threshold and event-style criteria that you monitor for, and customers should have access to the logs. Not ALL the logs, mind you, but ones that you think they should be able to see.
This should also be documented in the service contract. You do have a service contract, right? Maybe you know it as a Service Level Agreement...
If all you have is a hammer, everything looks like a nail.
Sue them for attempted DoS
Hacking the Network
If the hosting company can't deal with monitoring service that their customers purchase, they need to exit the business. There's no excuse for monitoring to break a server. Exercizing a server every 1 to 5 minutes is not exactly a DoS attack.
This is easy to answer. It goes too far when the results are more than the administrator(s) can handle - such as in your case. So what if you can plug in monitoring software that spits out mountains of data? Who's going to take the time to actually look at that shit?
When monitoring software is that elaborate, it is not unreasonable to expect the software to analyze the logs, produce a simplified brief, and nuke the unneeded information to reduce disk wastage. Software for monitoring is supposed to reduce your work, not multiply it. And it should never, ever have the potential of crashing a system as this did.
Why didn't the software recognize that space was running short and turn off logging. Sending an alert of some sort would have been good, too. Isn't monitoring the state of resources important?
I'd insist the customers dump that package. It obviously has fatal flaws in its design.
-- Will program for bandwidth
Contact Yahoo's lawyers, and have them check to see if they're also being DOS'ed too. If so, well, you got a pattern of abuse and more proof of their intentions.
Of course, I AM NOT A LAWYER.
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
Amazing polarity in responses here, from "this guy's an idiot" to "this guy's a poor victim" to "this guy's a troll." Haven't quite made up my own mind yet.
i would send these guys to hell right quick. if they are so agressive about monitoring your services and then take it to the customers, you should already have your sharks on the prowl. they are not legit, they have attacked you, and are just setting you up to steal your clients. they are causing outages in the name of proving their usefulness to the clients by alerting them of outages. that sounds like fraud to me. go get em, trash em, eat their lunch and smoosh them.
"You never want a serious crisis to go to waste." - Rahm Emanuel
their set of monitoring servers filled an entire 18 gig partition full of web server logs (causing the server to crash on a weekend) and choked an email server with 40k some messages that could not be delivered
/var on a separate disk so that fat logs don't cause the whole system to choke.
I think that's called a Denial Of Service attack. Your boss was right - firewall the attacker first, negotiate with them later.
On the other hand, you're kinda foolish for not putting
One of the biggest problems with monitoring something is that you inevitably affect it, a la Heisenberg in the Physics world. The more closely you try to monitor something, the more you affect it. This is a basic principle of monitoring.
Sounds like you need to purchase more monitoring from them to monitor the free disk space on the servers.... :p
Because, as every monitoring sales monkey knows - there's no point monitoring if you're not monitoring the monitoring... the monitoring .... the monitoring ...! Etc etc ad nauseam :-)
This is one reason why you have access lists in routers, for jerks like these people who basically engage in DOS attacks under the guise of "monitoring". I'd redirect anything from their IP block, back to their IP block.
The solution to this is simple. Publish the web address of this loser monitoring company and we'll let Slashdotters "check the integrity of their system."
These people are either technically incompetent, or intentionally damaging you. THere's no reason for what they're doing - there are many more practical ways to go about such monitoring.
When it starts costing you money for their 'mistakes', I think it's then time for either them to compensate you, or for you to sue them (in the cas where they don't return compensation).
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
According to the 'Book of Slashbot', "A slashdotter must POST first. First must he post, NOT last, but FIRST. Then these STEPS must be undertaken, AFTER the FIRST POSTING: Reload to see if anyone has replied to the POST, then read the article, then reload to see if its slashdotted yet, then reload slashdot to see if a change in karma has taken place, then reply AC to an IDIOT, then RELOAD the article to check your facts or make your argument, reload slashdot for KARMA updates or replies, and reloading to see what new posts HAVE BEEN made while reloading the front page to make sure a new story hasn't been posted while reading posts, replies, journals, post histories of idiots, your own posts, your past posts, your AC posts, your AC post replies, your list of friends, foes, FREAKS, and fans, and lastly, the article."
"One is allowed to slashdot stories in the future if one is a subscriber, or reads FARK. One should also watch for 'DUPES' as they are gifts of manna FROM heaven, allowing one to slashdot sites that thought their TRIALS and tribulations were over. Slashdot these heavily, so that THEY might leave the web, leaving their knowledge ONLY in your brain, to be quoted at a later date as fact and law, flaming those who whore for karma, and those fools who have not proven thy worthyness."
Yes. We're experts here. I'd quote more from the scriptures, but I've got some sites to take down by manically refreshing.l ick**clicick**click**clickety*click**
*click**click**cliclickickety**click**clickick**c
Anyone else have their webservers HAMMERED by a plethora of grub.org spidering clients?
...
You know, the distributed spidering system that IGNORES ROBOTS.TXT and hammers the crap out of anybody that hosts a bunch of websites?
I just recently had to block them with a nifty Apache SetEnvIf, happened to notice quite a few monitoring services hitting us, and then read this article
First of all, they illegally acquired confidential information regarding your customers. This is a serious breach of faith on the part of your former employee, and an act of theft on the part of the third party.
The second issue is that the third party company is guilty of committing a DOS attack - even if it was oversight and the use of software with legitimate purpouses.
There are two ways to mitigate the situation. The slimy corporate way is to sue the piss out of the former employee and the third party. The happy way is to replicate the tools your customers use and offer them for free. Basically, just hack some code to read your weblogs and print out pretty graphs regarding uptime, access, etc. Sell these services to your customers based on the fact that it is free, it is accurate because it comes from your servers own logs, and it won't affect their web site logs the way the third party software has. You should see the use of the third party software drop off drastically.
Simply shutting off your customers who use this software may not be the best marketing ploy, but I highly recommend you keep them in the loop, and pre-empt the third party by announcing the whys and hows of your actions.
Finally, although I abhor rampant litigation, I do highly recommend you document everything, and sue the third party company for damages. As you say, you have lost reputation and uptime, and while your uptime may have cost you hundreds of dollars, your reputation may cost thousands of dollars. I'd also sue to have the third party company reimburse your customers for the software; this should get you out of dutch with your customers who will feel screwed by both the third party company, and you, in this little pissing match.
as I used to work in the ISP department of a large mobile network provider in Austria I can tell you a little bit about the habits of our company: :-))) fixed them. But the customer was only informed if either the programs he ran on his server threatened the hole ISP environment or a downtime might occur due to other problems. We did the hardware monitoring with ORCA and the service level monitoring was done with OMNIBUS/NETCOOL. While the amount of logs that were gathered is still impressive, it could be handled and did't cause any crashes or other inconveniences. I have to add that the monitoring was done from other machines, with only a small probe of the above mentioned programs installed, the logs and the monitoring software were kept on different machines.
We used to let the users do the monitoring from external servers, such as ping probes if the server is up, they could look on a web page to see their volume information and so on. Bot we didn't implement e-mail alerting and so on, the only option you had was an occasional SMS. The whole hardeware monitoring (mem usage and so on) was done by the operators inside the company, that went after problems if they occurred and (sometimes
".Sig Stealer" was here
$ units bits/second bits/day
* 86400
So you're looking at (roughly) 100K hits per day per file downloaded per site. If they're downloading 15 files per site, and you've got 100 sites on the box, then you're looking at an increase of about 120 million requests per day. My acess log has an average of 200bytes/er line, so you're now looking at 120Mrequests*200bytes/request == a sudden jump of 24gigabytes of logging per day.
Then you've got the effective mail-bombing to deal with.
The article author said that these people sounded like they know what they're doing, so that leaves (in my mind), two likely possibilities:
- They're really really good snow-job artists. They understand the terminology, but they have no real sense of methodology or purpose.
- They really do know what they're doing, and they're trashing your servers with intent.
I mean -- for crying out loud: Multiple files once per second? And just how long did it take them to inform your customers that they'd managed to crash the servers? Monitoring granularity of more than about one quarter the normal notification time is a complete waste of resources -- and that's giving them lots of leeway to waste.And Tens of thousands of undelivered emails??? If those emails didn't get delivered, then what did the company do when they didn't arrive in short order? Why didn't they stop the transmission and diagnose why the emails weren't coming thru? If the emails really are undeliverable, then how in the world did you manage to conclude that they know what they're doing?
Other notes (mostly mentioned elsewhere)
-
are you charging your customers based on their net volume? If so, have you informed your customers of what sort of costs these, uhm, people are imposing on them in addition to their monitoring fees?
- I'm guessing that your AUP includes a clause on activities that wilfully or negligently cause inappropriate server load, outages, etc. I think that this company's "services" classifies.
- I think that you had better seriously consider possibility #2 above. Meticulously document what they've done to your servers (including somehow scamming your customer list). Have that information ready to present to your customers and/or a judge. If all goes well, you won't need it, but I'm not expecting all to go well, given how they've gone so far.
One last point -- Even though you may be dealing with a company that you think has a (otherwise) good reputation, doesn't mean that you're not dealing with an inept department of an otherwise good company. Sometimes the VP Engineering puts his/her stupid cousin in some group where they're not likely to do much damage, and then finds out that the goofball has managed to get out 'in the wild' with a 'bright' idea.Sometimes boldness is in fashion. Sometimes only the brave will be bold.
From here
internetseer.com - the newest web scam/spam. Here's something a little bit interesting on the web. This company, internetseer.com, is constantly hitting my site and others ostensibly to get web uptime statistics. Seems pretty harmless, but it does tend to fill up web logs pretty quickly. I don't know why their bot is set to visit this site 20 times a day, so I ended up blocking it. Yesterday, I received an email from one of their sales reps more or less saying, "Hi, we noticed your site was down last night, for x amount of dollars we can monitor it and sell you uptime statistics!!!" Which of course can be done for free locally. This is spam, these people are spammers. May google seaches with sitecheck.internetseer.com end up here. Avoid these bandwidth wasting spammers.
I used to work for a similar type of 'monitoring' company. The CEO/CTO was completely WACK!! He would have all logs centralized on ONE server, WITHOUT a backup server in place! A lot of the things he did regarding clients was simply unspeakable, as they would figure out his scheme and dump him for someone more reliable. I got sick of this and, after recieving non-payment for it because of HIS mistakes, I left too. There are a lot of companies out there who provide such services - finding the right one, though, is certainly a PITA. Extensive background checks are a good thing, but getting bit in the arse like that is even more annoying. I sympathise and hope that they get what they deserve.
As for your 'security expert', I find it strange that your company management is still listening to him/them after the second fiasco. If you don't have the time to take on security yourself, it might be a good time for you to go out and find someone a bit more capable to handle this stuff.
IF you do have the time to handle it, then this might be a good time to ask (again). Even with very little time to work on it, you could probably do a better job than these dofuses.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I work for a large hosting company. We have a lot of customers who have monitoring companies monitor their websites (we actually use some). We obviously monitor our services ourselves, but it is not always objective doing this. Having said that, monitoring once per second is *stupid*, generally 5 minutes is appropriate and we monitor some things internally every 60 seconds. We charge for bandwidth and disk usage (including logs), so if people want to monitor every second, go for it, your credit card will get dinged next month. For a smaller provider, I can see this being a problem, I would blackhole the IP. It is a DOS attack and I'm pretty sure you would have the legal right to do that. You do have a provision in your policies that you can take necessary action to protect your network, right? We do and will use it when necessary. Right after 9/11, we had a *very* popular and large image on our servers (the "eagle", if anyone has seen it). We "chmod 0"'d it and called the customer. They didn't realize what happened (getting so many hits), understood (once we explained bandwidth charges), and where happy we did it. Monitoring every 5 minutes is reasonable and will catch almost all outages.
Block them in the firewall as you did, dont givein for them pushing customers. Putout a statement telling your customers that you blocked them and that you did it for the customers sake, couse the company abused your servers, and you do not let anyone abuse your "customers servers uptime". Even if its an monitoring company, state that this is an action taken by your own monitoring department.
You have the right to block whoever you want, its your servers.
Just my 2cents.
I can add that I would block them instantly, no descution. Abuse is Abuse where ever it may originate from.
Yes, unless your site gets knocked down (or something stupider, its mistyped) and you get verio's siteseeker page instead :) so, no 404 there !
I meant VERISIGN and not Verio
And SITEFINDER instead of seeker. dammit
Now why didn't i pressed preview ?
I remember this one time I was monitoring this umm thing and something and there was an explosion and ahh.
Now wash your hands.
All I can say is that I see this problem often enough where a security consultancy is threatened by their client because their portscan which was done during a valid Internet security test has brought down an important system. Realistically, if something like a portscan can bring down a system then that is a real problem. You know how much random garbage comes across the Internet that can cause a similar problem?!
If the monitoring is happening so as to cause a DoS, which appears to be the case, it's an availability threat to your customers and a security threat to you. Since you provided no details on the type of monitoring and specifically how they did it, it's not possible to advise specifically. For example, if they use ICMP, there is a very good possibility that you should have been dropping silently ICMP of all types coming through your gateway router. That is considered best practice for security.
Treat this like a security issue and make it go away like a security issue. That implies using technology controls and policy to clean up this mess. In your case, policy will also include a letter from your lawyer and providing your customers with the uptime data they require.
-pete.
-- You bought all that security for your network, maybe it's time you got something for free. Like the ability to test it. The OSSTMM at www.osstmm.org - Stop talking security and start doing it.
Hehe, that is United Christian Emirates
You know we are doing everything to make you
kids the best Christians in the world! In
another couple of years your gals are gonna
put the veil on too!
Yours in Jesus,
Bribe Walls
Cricket.
http://cricket.sourceforge.net/
Microsft spel chekar vor sail, worgs grate !!!
Help no one...
As to the log processing. They should offer filtering rules/tools to avoid log thrashing.
Oh and once per sec is ludicris.
and if you're feeling vengeful, be sure to let them know that is what you are doing.
there are monitoring systems you can use inhouse, if you want to put the manpower into it.
i think the suggestions to get the company lawyers involved are correct, also. knowing where you stand legally is important.
mp
"The secret to strong security: less reliance on secrets." -- Whitfield Diffie
How long is a piece of string ?
If you don't understand anything I post, please accept that I ate paste as a small boy...
(Especially if they are reading this right now.)
It really wasn't a question of not wanting to take care of it, not being able to take care of it, or not having the time to take care of it.
It is because the head of the company pays far more attention to those who tell him what he wants to hear than those who tell him the truth.
The "security expert" told him what he wanted to hear.
For example, the president of the company doesn't want anything but Microsoft and Apple OS's. He doesn't want to learn anything about UNIX, Linux, or anything else. I think that "security expert" knew absolutely nothing about UNIX and Linux.
It took me forever to finally start switching from Windows NT to OpenBSD and Linux.
By the way, my first encounter with him was when he couldn't detect the version of BIND running on a Windows NT machine. Since it wasn't running Microsofts miserable excuse for DNS software, he didn't believe it was a DNS server. So he installed Microsoft's DNS service and started it up.
At that time, I was living 600 miles away and telecommuting. The president of the company told me on the telephone that the "expert" was installing Microsoft's DNS software on the server and I couldn't convince him that was a mistake.
So I sat back and watched. It took that expert about 24 hours of very expensive billable time to find and disable the port of BIND that we had been using.
The reason he was there at all was that we were switching from one block of IP addresses to another. So, the BIND software was providing the old IP addresses for all UDP queries and the miserable Microsoft DNS software was providing the new IP addresses for all TCP queries.
I assume your TOS allows you to take any measures necessary to preserve the normal activities. Tell ALL your customers what you just told us. If I were your customer who had signed up for this "service", I would understand why they were firewalled. If I had NOT signed up for the service, I would be annoyed if you allowed this amazingly RUDE external company to crash servers and fill email spools that I was paying for.
A "Cease and Desist" letter, telling them that their product's abnormal actvity is interfering with your core business services, causing you support porblems, and is indistinguishable from a spammer crossed with a DDOS, should follow. I'd add a bill for the cost of supporting their malfunctioning: your customers may have agreed that the monitoring service was not responsible for anything, but you certainly did not.
know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers.
My take is, if they knew what they were doing, it wouldn't have come to this. Your company approached them and asked them to make adjustments. They chose to ignore you -- their customer. Someone who knows what they're doing doesn't ignore their customer.
They trolled your site and pulled information out to sell their services to your customers. Your CTO cut them off and they proceeded to beyotch to your mutual customers.
Yeah, sounds like a classy group of people.
They may otherwise be good, but technical exptertise is no grounds for arrogance.
I say find a new service and put into the contract strict guidelines as to what your company wants. I imagine whoever wrote your contract with them did this in the first place. Obviously, it wasn't strict enough. I guess they left out the part about using your system to determine who your customers are, so they could sell their services to them as well as overburdening of your resources.
If not, thats grounds for contract termination as well as a lawsuit.
Cruising the internet on my TI-99/4A @ a whopping 300 baud!
I would not admit that in public about my company even if I used the phrase 'a company I work for', just on the off chance my negligence would be able to be tracked back to me.
And it's all too easy to do. For example: Jafiwam's bookmarks on dmoz.org list Bear Hart Ltd, a hosting company. And a slow one at that. I'm not saying that's who he works for, just that jafiwam is a pretty unusual sequence of letters, and it usually seems to be associated with Stevens Point, WI.
well, it goes from the start to the '\0' :)
Free as in mason.
How about partitioning your servers properly so they don't crash when they fill the logs?
Basic sysadmin 101, people. You're going to piss off customers by doing what the parent suggests.
Please help metamoderate.
On Thu Sep 11, 2003 at 06:23:10 PM EDT we were unable to reach your website:
Fuck off, cunt!due to the following reason: Host Not Found
As of Sat Sep 13, 2003 at 11:30:47 PM EDT we are still unable to reach your website.
We discovered this error during our normal course of website content checking for one of our search engine clients.
If you would like to receive notifications like this in the future if we find pages unavailable, click here.
Click here to learn more about us.
Sincerely,
Connie Davis
InternetSeer.com
You don't have to worry about people harvesting your IP address space. Sure, it's a complete pain in the ass to set up, and it increases resources needed (abuse, etc) immensely, it's probably worth it in the long run.
You can control how many queries/day, etc.
Malicious code and DOS attacks aside: The internet is free, let's keep it that way. If you put in a law that says so and so can't use the internet for such and such a reason, what's to stop it from going overboard? I know it sucks to be you, but no one ever said you wouldn't get 40k *legitimate* email's over a weekend. That is simply the price of doing business online.
Your company is paying this other company for monitoring services. If you are not satisfied with the work that they are doing, let them (and those in your company who make these decisions) know about it. If they are unreceptive (which it sounds like they have), take your business elsewhere.
While your firewall machines may not be running Linux, the IPFilter function has the ability to limit accessibility based on frequency. For example, you can say:
"allow no more than X ping requests from Y in Z seconds"
Of course, you shouldn't have to resort to measure like this to deal with these specific people, but it does send a message.
Admin-on-the-street says "I need a job, you insensitive clod"
My beliefs do not require that you agree with them.
Note: OP means the monitoring company with "they". Pissing off your customers by telling them that they themselves are under legal investigation is a kind of SCO idea...
I suffer from attention surplus disorder.
'While I welcome anything that lets our customers use the internet effectively, their set of monitoring servers filled an entire 18 gig partition full of web server logs (causing the server to crash on a weekend)'
Had YOU had proper monitoring and processors in place YOU would have noticed the file system filling up, YOU would have taken care of the issue before it crashed the server. Sounds to me that your company needs better procedures, processes, and perhaps admins!
...it sounds like your servers are ripe for a logging based DOS. Have you thought about tweaking your logging so it rate-limits similar and identical log entries?
that when the website monitoring brings down the website, it's gone too far, heh.
This has nothing to do with website monitoring per se, but it's an issue of commercial confidentiality. It's like outsourcing a mailing list to third party company to find that the third party steals the list and uses it contrary to the agreement. You should - if you can prove it - bring an action against them for violation of confidentiality - I think you would have a strong case. As a customer, I've had this happen to me in the past (where it seemed like someone inside a larger company had sold off an internal mailing list /data to a third party).
Naturally, none of this applies if they simply walked your address space - I think they could do that legitimtaetly.
First, I suggest notifying your customers that "a monitoring services company" is using poor standard practices that resulted in what mimicked a denial of service attack on the servers that hosted their domain(s). Once identified, the offending IP addresses were blocked by firewalls to prevent similar attacks in the future. That your company contacted this service provider and advised them of their poor practices and you offered suggestions on improving their practices, which they summarily refused to even listen to. Third, if your customers are on a measured bandwidth plan, did this cause any of them to go over their alotted bandwidth? If so, advise your customers that this company's actions and poor practices will now cost them $X on their current bandwidth bill. If necessary, project it out for a full month, then tell them it _would have_ cost them $X on their current bill. If anyone gets a bill for exceeding their bandwidth, ask them if they'd like to join you if you take civil action against the "services" company. [Hmm, you might ask all customers that suffered due to the outage, not just the customers of the service company, if they'd be interested in a class action suit too.]
Now, since you notified this "services" company that their actions caused problems that worked just like a denial of service attack, and they refused to alter their behavior, then treat them like they are attacking your systems. Call the authorities. Charge them with whatever criminal counts you can. Take them to civil court, charge them for the damages the outage you suffered due to their actions caused you. Invite your customers to join your action for any bandwidth charges that resulted, if any.
You'll really need to consult an attorney about these ideas. Plus, you'll need to have very good documentation. Such as who you spoke with at the services company, date, and time, and notes on what they said.
Also, in any communication, do like you did with this posting and do not name the company. Once you file a court case, then you may name the company since then the court case will be a public document. At least, you'll be able to say your taking legal action against XYZ.
. 62,400 repetitions make one truth -- Brave New World, Aldous Huxley
You have to love a web design company whose site has UI issues. (I wasn't able to use their fancy dancy scrollbars.)
Stevens Point, home of UW-Stevens Point and Point Beer!
Monitoring sucks, respect the privacy of your users- this is a fundimental admin rule.
See, what you should have done at this point was to notify your customers first that you were blocking this company. That way, you can explain to them in advance that it's the monitoring company's fault and not yours.
As it is, I see your best option is to 1) keep blocking them until they shape up and 2) give your customers your side of the story. If this company is as sleazy as they sound, their sales drones have already sold the situation as something that's completely your fault and possibly dishonest too. (E.g. you're blocking them because they compete with your own monitoring service.) You need to make sure that your customers--all of them, not just your mutual customers--know that it's the monitoring company's fault. Reduce it down to "they cost you $BIGNUM per month in bandwidth due to their incompentence and they ignored us when we tried to talk to them about it."
This isn't a tech issue so much as a business issue. These assholes are playing hardball and you need to defend yourself.
Never underestimate the ability of senior management to believe a consultant over their own employees.
I once had a boss ask for a list of domains we were serving DNS for. So I gave it to him. Turns out what he really wanted was copies of the config files, for whatever reason. When I handed him the list, he decided that I didn't know what I was doing, so he brought in a consultant.
Said consultant then spent an hour or two going through everything, shaking his head, as all was well. "Why am I here?" he asked me. "I honestly don't know," I replied.
Then, when it was explained, by the boss, that he wanted the config files, I took five seconds, and out the printer they came.
Vintage computer games and RPG books available. Email me if you're interested.
their set of monitoring servers filled an entire 18 gig partition full of web server logs (causing the server to crash on a weekend)
I agree that these logs are excessive (does anyone actually look at them?), but a log should never crash your system.
Don't you have things like log rotate or system monitoring to keep disk usage in check? If not, then you need it!
Websites should not be monitored by anyone. However, all IT admins have a RESPONSIBILITY TO READ ALL YOUR EMAIL. It's the only way a fully-qualified, professional who gets treated as an information janitor can have any power. When sending your email, be afraid! BE VERY AFRAID!
:-)
Oh, and don't forget keylogging.
It seems to me that the monitoring company is doing a disservice to its clients (your clients) by hosing your hosting system. The clients will not be pleased to discover this, and can probably be enlisted in the effort to persuade the monitoring company to adjust its procedures. So the first question is whether these clients know what their vendor has done?
Exactly the best way to handle this is a market-relationship issue. The hottest would be to sue the monitoring company (or threaten to sue it) and then put out a press release. That should be hard on their sales, and so have a lot of leverage, but it would provoke a public fight that you should first make sure you'd win (in the public's eye).
A more calm way would be to call each of your customers and tell them what has been happening and ask them to call the monitoring company.
In the long run, of course, this needs to be incorporated into a comprehensive "how to cope with DoS attacks" strategy.
MRTG
n dex-2.html
link: http://people.ee.ethz.ch/~oetiker/webtools/mrtg/i
If you aren't automatically "rolling" your log files, then your server crashing is your own fault! At least Syslog them somewhere else!
Think about it this way. Obviously you're logging incoming connections, what they do, and probably some other stuff, right? Well then let's say this company didn't probe your servers... your servers would have eventually crashed with regular internet traffic anyway...
Get over it. Probing is a reality now. If you don't accomidate it, you will crash and burn.
That guy seems to be to naive to be from this world.
I dont know ANY unix admin running ANY server for other peoples fun for free.
There are those admins who run their servers for their own fun (unix admins playing whatever system games) and there is the largest portion:
Windows-Only-Users renting cheap servers in the internet and therefore always using unixservers.
This base alone may be responsible for some thousand dedicated serves of Quake, Halflife and many many other systems. Those thousands of thousands servers matter, not the 25 servers set up by unixadmins to join playing with unixclients.
I have seen die-hard-windows-only-users getting into linux very fast if confronted with the choice between an dedicated Halflife-Server running linux for 40/month or paying 200 for an equal windows-system.
"Life is short and in most cases it ends with death." Sir Sinclair
Eeeepp.... ! Once a **SECOND** ?! That is just plain nuts. I happen to be in the midst of setting up a monitoring system for a hosting company at this moment, and we're doing 5 min. intervals at most.
Perhaps mebbe, for uber-important services we MIGHT one day monitor faster. But that's a big mebbe. You'd have to be in a situation where that extra 4 min or so is worth the costs in load on the system.
Anyway, as long as folks are talking monitoring systems, (I here nagios mentioned a few times) we are using OpenNMS. Anyone else out there have much experience w/ it? It seems to be a major, industrial grade system (which is good fer what we do) with all the power, and complexity that implies.
-- -- The Dragon De Monsyne
I've worked with them on web server monitoring before.
The customer is actually our parent company, and has asked to get external monitoring on their behalf.
Working with alertra we set our own check intervals. We can customize just about everything from checking DNS responses, page gets and that sort of thing. We also have the ability to scrip XML gets for communicating to external web servers (something that's hard for us the check from inside, we catch about 50%, Alertra catches 100%)
I know that I do get pissed off when something breaks and alertra keeps paging until the problem is resolved, but their monitoring is very unobtrusive and effective.
Just use netcraft. :)
Firewall them off, and tell them to stop.
Then send mail to your customers explaining how their "monitoring service" was so horribly misconfigured that it was essentially more of a denial-of-service attack. Explain that you've firewalled that monitoring service off.
However, I recommend that you make it extremely clear that you're not just afraid of them monitoring -- if my hosting place blocked off monitoring, I'd figure they had something to hide and be afraid. You should specifically mention that they are welcome to use a "safe" monitoring service, perhaps recommending EasyMonitor.com (free), or Alerta (not free).
As an alternative, if your uptime is pretty good, consider paying Alerta (a _really_ popular monitoring company) to monitor your servers for you. The hosting company I'm with has Alerta monitor each of their servers for them; this saves from having the hundreds of people on each box each do monitoring. It could cut way down on the load / traffic, in addition to making yourself look good. (A company with publically-available uptime records for all of their servers shows that they're not trying to hide downtime.)
Anyway, block these people, but make sure you keep your customers well-informed of what's going on, and make it clear that any sort of _sane_ monitoring is entirely allowed.
________________________________________________
suwain_2
There are those who say that if a designer (or in this case design company) has the time to work on their own stuff then, they're not getting enough work, and must not be good enough. If that is true or not, is up for debate. This company, with a portfolio of just 3 clients, one of which is their own site, probably doesn't have that problem.
My subtext is just a figment of your imagination.
P.S. The article is the zeroth post, named after the percentage of /.ers that bother to read it before posting.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
I run a website monitoring company and used to work as a network management consultant for some years now.
In the early days it often happened, that monitoring caused more trouble than it helped to make sure, that all systems are running fine.
As for the marketing methods mentioned, I do not think, that it is the right way to do IP-scans and spam the owners, but I can say from my own experience, that it is quite hard to find people responsible for websites if you do not do it like that. At www.nmsalert.com we never used that method anyway.
As for creating failures instead of preventing them I personally would never ever ask a company to do stress tests on my production systems. I cannot see any reason at all, why anybody else should do so. For sure, if these tests are done, they dont have to repeated every few minutes (or even seconds).
We monitor usally every 5 minutes, by doing a ping, opening an smtp-connection and getting the http header. This does not cause any additional load on the systems, which would be worth mentioning.
If I would be responsible for the datacenter where the servers are hosted I would also block that company and their machines completely.
Cheers,
Markus
www.nmsalert.com
Friendship to you from mare lovers. -1 flamebait m2'ed unfair!