Slashdot Mirror


When Does Website Monitoring Go Too Far?

jafiwam asks: "Recently, the IT department of the company I work for and a 3rd party monitoring and security firm got into a pissing match about how much monitoring is too much. They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers. They then proceeded to sell them monitoring services for things such as server up-time, defacement detection, email up-time and DNS testing. While I welcome anything that lets our customers use the internet effectively, their set of monitoring servers filled an entire 18 gig partition full of web server logs (causing the server to crash on a weekend) and choked an email server with 40k some messages that could not be delivered, and they failed to properly brief the hosting customers about what would happen to their log analysis software when faced with 99% traffic from a small set of IPs. These things caused down-time, lost productivity and a damaged reputation. What is appropriate for monitoring a web site and email server? Who should be allowed to monitor? Where should the give and take lie in this situation? I am interested in finding out what admin-on-the-street has to say about this."

"Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.

Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)

Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."

71 of 259 comments (clear)

  1. How about enforcing a time-based rule? by Anonymous Coward · · Score: 5, Insightful

    They must be a way to enforce that they could check, say, only once every hour. And BTW, isn't your company missing an opportunity here? If you're already checking the servers, etc., why not make the tools available to the customers? They'll be more satisfied with the tools, and not having to pay the outside firm. You'll have more satisified customers and less churn....

    1. Re:How about enforcing a time-based rule? by joeszilagyi · · Score: 5, Insightful

      Except if you open those monitoring tools to your client base, it opens the possibility of them seeing the same info you do, which isn't always a good thing for a variety of technical reasons.

      --
      Dude, where's my packet?
    2. Re:How about enforcing a time-based rule? by toast0 · · Score: 2, Interesting

      I know this is feeding trolls, but... who is to say how bad it is for something to be disclosed?

      not to mention, it's probably already quite possible for your users to find out you were down for 30 seconds or so; even if they don't know it was cause the ceo tripped over a network cable, and knocked some network equipment down

    3. Re:How about enforcing a time-based rule? by SatanicPuppy · · Score: 4, Interesting

      Here's my funny story: I was using Perfmon (NT monitoring utility) to monitor usage on this half dead database server, trying to get some compelling figures so I could argue for a new server.

      So it's got all these options, and I wasn't paying attention, so I just said, "Monitor EVERYTHING...At 5 sedcond intervals."

      Fortunately, I'm not a complete idiot, and it only filled up the directory I'd set for it, not the whole harddrive, but it did teach me an important lesson about log files: You can get a gig of useless information in less than an hour, OR you can monitor the IMPORTANT stuff, and get a gig of useful information in 2 or 3 days.

      In case anyone is wondering, my logs proved 2 things: 1) That they needed a new database server and 2) That the people who were bitching about it being slow ALL the time, were actually only working about an hour a day.

      --
      ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
    4. Re:How about enforcing a time-based rule? by ananke · · Score: 3, Informative

      one of such monitoring tools is nagios. it allowes for multiple users, with access limited to view information only on specific hosts/host groups. it's a pain to set up initially, but in the end it works quite nicely. www.nagios.org

      --
      --- d'oh
    5. Re:How about enforcing a time-based rule? by Bios_Hakr · · Score: 4, Insightful

      UUh, maybe I'm missing something here. Why would you not want a customer to see all the data associated with his server.

      I work in a network shop that provides connectivity to remote buildings on our campus. Each building has a psuedo-network admin. Usually a second job that some paper-pusher takes to get in good with his boss. By default, the building admin has his home page set to a MRTG log showing every switch in his building. They are trained to look for network spikes on user's ports and notify us so we can disable that port, if nescessary. He can also monitor everything from fan speed to temprature setings on his router and the core router for our remote users.

      --
      I'd rather you do it wrong, than for me to have to do it at all.
    6. Re:How about enforcing a time-based rule? by LittleBigLui · · Score: 2, Insightful
      and it only filled up the directory I'd set for it


      so how big can a directory get before it is full?
      --
      Free as in mason.
    7. Re:How about enforcing a time-based rule? by poot_rootbeer · · Score: 3, Insightful

      UUh, maybe I'm missing something here. Why would you not want a customer to see all the data associated with his server.

      Don't tell me you've never gotten an irate message from some idiot out on the net who installed poorly-configured personal firewall software and says "I went to your website and it tried to hack my computer on port 80!"

      Sharing information is, in general, a Good Thing. But if they don't have an understanding of how to apply the information in proper context, it can do a lot more harm than good.

  2. The obvious answer by Exiler · · Score: 5, Insightful

    Don't give a company of strangers the key to the front door. There's no reason someone from your company wasn't there to say 'when.' As for when too much is too much, it'd be when the efficiency of your main product is impaired to the point that you lose customers or reputation.

    --
    Banaaaana!
  3. Log partitioning by Anonymous Coward · · Score: 3, Informative

    A server should not choke if the log partition is full. Is the log in a separate partition, isn't it?

    1. Re:Log partitioning by MikeFM · · Score: 5, Informative

      I'd think somebody would have noticed the high usage and firewalled off that site too. I mean jeez that must have been thousands and thousands of hits to use up that much space. I'd suspect a DoS attack if I saw that in my logs.

      I also suggest anyone running servers to have some sort of program monitoring disk usage. If the disk gets dangerously low on space it should notify staff and take action such as rotating logs. Have the server page an admin or set an alarm off (where it'll be noticed) or something. Whatever you'd do if an attempted intrusion was detected. I usually have the server send warnings at 90% and 95% and at about 97% usage it should give me a good loud yell.

      --
      At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
  4. OVERKILL, is what it is. by joeszilagyi · · Score: 4, Insightful

    Their 'harvesting' your IP block is tacky at the least. That said, the current range of InternetSeer type monitoring is flat out overkill, and doesn't even work right half the time. According to some of them, my site is constantly down, but it *never* is. I know, since I'm an access_log nerd and always play with it; people are always going through it without any large 'dead' blocks appearing. All you need is a remote monitoring system to let you know when your major ports aren't functional, and to have it mail you ONLY when it's down. These 100k emails dripping with HTML to let you know that your site is still up are a complete waste of good bandwidth. Ping your damn site on your major ports, and that's all you need.

    --
    Dude, where's my packet?
    1. Re:OVERKILL, is what it is. by joeszilagyi · · Score: 3, Insightful

      A fair point, and THAT is the point at which you ratchet up your monitoring practices to compensate for things that are being missed. It's the same as the military; carpet-bombing is often ineffective, but preciosion targeting will (almost) always get you what you want. Pings look good? Check. Things still not working? Dig deeper. More often than not, though, ping queries should be enough (assuming the network/host doesn't block ICMP or screw with it).

      --
      Dude, where's my packet?
    2. Re:OVERKILL, is what it is. by k12linux · · Score: 4, Insightful
      Ping your damn site on your major ports, and that's all you need.

      Sometimes services can lock up to the point where they are not functioning without closing down the port. Something slightly more thurough like nagios should do nicely. ie: Does a simple http request and confirms the reply is ok.

    3. Re:OVERKILL, is what it is. by cbreaker · · Score: 4, Interesting

      Sometimes you need more checks then ping.

      At one of the companies I worked for, we had a pretty large farm of web servers running, and some hefty database servers on the backend. Not to mention all the support servers; running specific tasks. Some scheduled, some triggered.

      For our web application, ping wasn't enough. Sure, the server would be running, but since the application wasn't coded in pure html, we needed to make sure it was actually working.

      We set up scripts to test the functionality of various application functions. We also had to monitor all the web servers and database servers individually. We also had to monitor the "service status" of the entire system; ie two web servers can fail and it's not an emergency - but if the application is not functioning through the load balancer, it was.

      Ping doesn't always cut it. With any somewhat complicated web application you need to monitor the functionality of the application, not just the server.

      To add, 5 minutes is a big deal. If you have a web application that's heavily used and with paying customers, it's important for you to be up and running. If something unfortunate should happen, you need to know right away. We had some of the simple checks running every 2 minutes, and some of the more intensive checks running every 3.

      Obviously, running a check every second is ridiculous, especially if it's something dumb like TLD servers. An hourly check on that would be more then enough because you can't fix it quickly anyways. Not to mention that you must be aware of the monitoring system in place and make sure your servers won't choke from it. Make sure you have enough log space. Make sure you're not affecting application performance from monitoring.

      --
      - It's not the Macs I hate. It's Digg users. -
    4. Re:OVERKILL, is what it is. by smithware · · Score: 2, Insightful

      If you're doing your own monitoring (on a small level), I've found a program that I really like, Host Monitor by ks-soft.net. It allows you to not only do simply ping testing, but also test against databases, webservers, odbc connections and much more... it's pretty nice, and comparatively inexpensive. As far as the timing of the checks, I use three levels. 1. For production severs, I perform my tests every minute or so, including http get tests, pings, database connects to make sure things are up. 2. For internal but mission-critical servers, I test every five minutes, and these are mostly just pings. 3. For secondary servers it becomes every 10 or 20 minutes. One of the nice things is that through normal daily use, the testing seems to 'normalize' itself so that everything wasn't happening at once. Testing intervals should be counted from the end of the previous test so that things don't bunch up.

  5. Confidentiality by Chester+K · · Score: 4, Insightful

    They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers.

    Sounds like you've got an open and shut legal case to recoup those costs they're causing you to incur.

    --

    NO CARRIER
    1. Re:Confidentiality by Maserati · · Score: 5, Interesting

      Firewalling them is good, your customers have no authority to allow them that kind of access to your network. Have your corporate attorney send them a polite C&D letter. By polite, just the followup contact - this time on an attorney's letterhead. Also consult the attorney for what you should/can tell your customers, then do so immediately.

      Be very clear to your customers that your objection is the nearly-criminal (it's a DOS) heavy-handedness, mind-numbingly unethical and pathetically incompetent behavior of the monitoring company. It's not unreasonable for one of your customers to retain a third party to provide professional services of this nature; by professional I mean 'do it right' not in the sense of professional as a term of law. Loading your website at regular intervals and parsing their logs for them is fine. Right now, these guys are probably reporting the outages they caused.

      Billing your clients for bandwidth used by the monitoring company they hired is not completely unreasonable. Be sure to document every cost associated with this in every way, including time reading responses to this article as 'best practices research'. I'm not kidding, if you worked late you add the pizza in or the taxi home. Every penny in fine detail. Your lawyer will be keenly intereste, so might law enforcement if the polite C&D letter didn't do it.

      Since the offered protection, aka monitoring services and then caused damage to your systems you could make a case that a protection racket is being run. If, adding in their fees for their services (paid by your customers) to the damages calculated above you have more than a certain threshold, probably US$50,000, then the FBI will be interested. Also have the monthly and annual total of your revenue from the customers either employing the monitoring service plus those affected by the damage cause (probably all of them). If things go sour with them and you do go to law enforcement, wave your revenue totals around to help get DAs and FBI interested.

      Basically, you call your lawyer and then contact your customers. Your lawyer asks them to behave themselves. Then you meet with the lawyer, discuss the response and post another Ask Slashdot.

      --
      Veteran, Bermuda Triangle Expeditionary Force, 1992-1951
    2. Re:Confidentiality by LostCluster · · Score: 4, Interesting

      Unfortnately, we're missing a key part of proving the "protection racket" scheme here, proof that the monitoring company illegally got ahold of a customer list. If this company just spread by word of mouth though the customers and advertising aimed at webmasters in general, then there's nothing illegal and they'll defend themselves by tar and feathering the webhosting operation for not being able to handle the level of trafic they promised the customers.

      The customers should have run up huge bandwidth bills by causing their traffic to suddenly multiply by thousands with the auto-checking for site defacement (trans: re-spidering their site at an insane rate), and that'd be the way to recoup costs and then come off as the good guys by waiving thousands in excess fees...

    3. Re:Confidentiality by vt0asta · · Score: 4, Informative

      IANAL, but if you'll allow me to shoot from the hip for a bit, I'll take a shot at it...

      1) Tortious interference with business relationships. The solicited the customers. They directly interfered with the business relationship by bringing the servers down by overzealous monitoring.

      2) The outage was caused by the monitoring company. If just one customer leaves to another hosting company because of outages or what not, or if that customer lost business due to downtime. The damages are realizable.

      --
      No.
    4. Re:Confidentiality by Anonymous Coward · · Score: 2, Interesting

      Ahh, yes, lawyers. Sue sue sue.

      C&D what? Block them entirely with firewalling, that's your right. But lawyering this, you're asking for trouble. The very nature of TCP protocol is that THEY ask for info, the ISP network acknowledges and then GIVES them the correlated data. Absolutely nothing illegal here. The ISP defaulted and let them in.

      As to billing your customers, how nice. The way I look at it, 2 companies screwed up and now you want the customer to pay. The ISP--hey, how about setting up the servers right so freakin logs don't crash the machine. Sending out alerts to your damn admins. Why didn't the weekend admin or admin on call at least notice this? Seems like the ISP is trying to save face and pass the buck. They were contributory to the fault and hassle that was caused.

      To the monitoring service--quit being asses to potential customers.

      As to the protection racket, nice slippery slope. The facts aren't entirely clear what caused the harm to the ISP side of things--sounds more like someone was caught sleeping and is now trying to pass the back. Really--did the third party system hammer their systems, or did the customers sign up, resulting in hammering of the systems? If the latter, it's not a protection racket. If the former, you have a chance.

      Under the circumstances, the ISP has a better chance at a tort claim than a criminal case, and the tort claim would be stretching things a bit by itself.

  6. I know what to do! by rock_climbing_guy · · Score: 4, Funny

    Let's all pitch in on a little scheme. We will each agree to buy a service plan to have one non-existant .com web site monitored. If we could get lots of people to do this, we could DDOS Verisign off the internet!

    --
    Wh47 d1d j00 541, 31337 15n't t3h r0xor5 ne m0r3???
  7. One word: by stor · · Score: 5, Informative

    Nagios.

    http://www.nagios.org/

    Cheers
    Stor

    --
    "Yeah well there's a lot of stuff that should be, but isn't"
    1. Re:One word: by dpoulson · · Score: 3, Informative
      Two words!

      Big Brother

      Both are good monitoring packages, it's up to personal preference really.

      --
      http://www.22balmoralroad.net/ http://www.tinynetworks.co.uk/
    2. Re:One word: by @madeus · · Score: 4, Informative

      Both are good monitoring packages, it's up to personal preference really.

      Actually Nagios is a lot more powerful that BB (which really doesn't do all that much), and aisde from that Big Brother is not 'free' (often people just don't bother to read the Terms and Conditions and think it's free).

      You can use BB with no charge to monitor certain systems, but if you provide certain types of services you are required to by a license, and these days most medium and large ISP's fall under this category.

      Big Brother is amazingly basic, I don't understand why people get so excited about it (I could re-write it in a day, and I'm far from a rocket scientist). Nagios, in contrast, is a full network and service monitoring system, and would have been much more useful in this instance and you could have used it to more easily identify the source of the incoming traffic.

  8. When Does Website Monitoring Go Too Far? by _Pablo · · Score: 2, Funny

    When it exceeds the point of being far enought!

    Kind of depends on how rapidly you can respond to a problem with something being monitored - obviously every second or even every minute is too rapid. Every hour sounds better.

    --
    $2B OR NOT $2B = $FF
    1. Re:When Does Website Monitoring Go Too Far? by LostCluster · · Score: 2, Interesting

      Some $9.95 a month websites don't even get a "real" user once per hour. So, for them that'd be a sudden multiple of traffic...

      What this really smells like is a webhost who oversold their server on the theory that everybody would never take their accounts to the promised limits at the same time, and then that's just what happened and the webhost got exposed as not being able to handle it...

    2. Re:When Does Website Monitoring Go Too Far? by ichimunki · · Score: 2, Insightful

      If the customer is only paying $9.95/month for the site they either have traffic limits set in terms of rate of activity (i.e. your site will never send out at a speed that would tax a 28.8 modem-- this is not a common approach, if it's used at all) or total periodic bandwidth allowance (you can't transfer more than a set GB limit a month without paying extra). Some script-allowed hosts will also set CPU limits on CGIs.

      This rate of monitoring is no way going to come in under the transfer caps at the end of the month and these discount hosting customers would get SCREWED in terms of their bill, I'd think. Or maybe they deserve those bills for being so braindead about the impact of the monitoring service on the servers and the network.

      What this really smells like is bad admin in terms of log size/rotation policy. Never once did the poster mention that there was a choke on transfer rate-- rather that the servers went down due to software crashes (running out of disk/RAM can do that, no?).

      I would lay blame on everyone involved, personally. The "monitoring company" for being a crappy service and not working with the ISP. The ISP for having bad server management policies that lead to crashing during perfectly predictable events (what happens if one of their customers gets Slashdotted? would their logs have been able to handle that, too?). And the customers who arranged for this monitoring without talking to the ISP about it first or at least properly understanding the impact it would have on the network.

      --
      I do not have a signature
  9. for smaller companies... by yet_another_user · · Score: 3, Interesting

    ...that either don't have the time / money to go after people like this, such as the webhotel I'm involved in in my sparetime, I'd recommend firewalling. Simply block all incoming connections from over zealous monitor-companies.

    Ofcourse this doesn't do anything to fix the bad reputation they may have given you by flooding your servers, but its a quick and easy antidote against future problems.

  10. hm by revmoo · · Score: 5, Interesting

    From your description, i.e. "Once per second", that is quite beyond monitoring, and that is an EXCESSIVE use of bandwidth and resources.

    Now, if you charge your customers based on gigs transferred, it seems like this would fill up their quota for the month quite quickly. What are your customers going to think when they get a large overcharge bill for the bandwidth? They signed up for the service after all.

    If you aren't hosting for money, then you probably aren't able to profit from this monitoring companies actions in the same way, so I suggest you blackhole their ip's. Downloading files from your server once per second goes way beyong monitoring, and into the realms of denial of service(It crashed your server you say).

    What I would do? Make a change to the aup for your service stating that customers that use monitoring services that abuse bandwidth will have their accounts revoked, or be charge for the excess bandwidth used. There's no reason in the world why these people need to hit your servers as often as they are.

    If you are unable to do business with your servers being hammered, then I suggest blackholing the monitoring service's IP's. It's only sensible.

    --
    I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
  11. monitoring by Feyr · · Score: 5, Informative

    we typically set our monitor software to check every 5 minutes, with one request PER SERVER not per site. if it is down it will send an email to our support address, if it is STILL down the second time around, it fires off an email to the cell phone of the on-duty admin, plus one email when it comes back up

    i've had some services set up for monitoring as low as 30 seconds, but those are specific cases.

    obviously a 1 seconds check is WAY too low, not only it's a waste of bandwidth, it's prone to false positives. what happen when you have a slight delay in one of the core routers that cause your packet to get dropped/delayed by 1000ms ?

    1. Re:monitoring by Babbster · · Score: 4, Interesting
      I may have a suspicious/cynical mind, but could the reason for the one-second intervals BE to generate false positives during the monitoring? If they (the third-party monitoring company) could generate logs claiming that the web server was down a disproportionate amount of time because of said false positives and/or the downtime generated by their own DoS-type activities, they could do things like offering alternative hosting companies (owned by the same company or just getting kickbacks) or offering [unneeded] technical support to "improve" the website to correct nonexistant issues.

      If a company did this kind of thing, even if taken to court they could produce the logs that verify the artificial downtime in order to defend themselves against accusations of lying to customers. Then, when asked if their once-per-second monitoring could have been the cause of the problem in the first place, they could make some fanciful BS claims like "a good server should be able to handle that."

      My apologies for spinning an entirely hypothetical, and possibly paranoid, scenario. This was the first thing to pop into my incredibly suspicious mind - plus, it has the makings of a good scam if it hasn't already been done. :)

  12. Bad practices all around... by Jonah+Hex · · Score: 4, Insightful
    checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.
    They obviously haven't been in the monitoring biz that long, at least not long enough to get a bill for all the bandwidth they're sucking down.
    Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)
    Sounds like your company is reasonable, and therefore expecting this possibly "fly-by-night" monitoring company to also be reasonable.
    Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers.
    I just checked out ClarkConnect's monitoring page (I use their free Linux firewall but not these pay services) and their lowest monitoring interval is 2 minutes for $45/mth, then 5 for $30/mth, 20 for $10/mth and finally 60 mins for $40/yr being the cheapest. Obviously they know such continuous monitoring justifies passing that cost along to the consumer.
    It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."
    Thier own biz practices will be the death of them, don't shed any tears over a company that makes this large of a mistake and uses dirty methods to contact customers. I wonder if your now going to have to charge your hosted sites that used the services for the excess bandwidth they used? Maybe cut them a "goodwill" deal on the excess charges?

    Jonah Hex
  13. How much is too much? by Alien+Being · · Score: 5, Interesting

    Here's a common sense reaction.

    They are in the business of measuring Net availability. They should learn to set the scale on their instruments before they connect them to the circuit. And they should back off when availability drops because they might be the cause of the drop. If their traffic represents more than about 10x that caused by an individual customer, then as a "juror" I'd think they were being irresponsible.

    You are in the business of supplying Net availability. You should install circuit breakers. Too many connection from one host/network? Start dropping packets. Too much raw incoming traffic from one source? Get on the horn quickly to the netadmin.

    Your customers don't care who's at fault, they want what they paid for. But they can't expect miracles.

    1. Re:How much is too much? by LostCluster · · Score: 2, Informative

      Of course, a webhost also needs to communicate what their customers are paying for. If you claim unlimited bandwidth for $9.95 a month, don't be surprised when somebody takes you up on it. These customers should have had some sort of bandwidth limit where the overmonitoring would cause their site to get defaced with the webhost's "This site has exceeded it's bandwidth limit, come back next month!" page or start running up a huge bill. The customers should know better not to invite such an attack on the server, and should be the ones feeling the pain. That'll put this monitoring company out of business in a hurry...

    2. Re:How much is too much? by Phroggy · · Score: 3, Insightful

      But they can't expect miracles.

      Of course they can, and do. They won't get them, but that's different. ;-)

      --
      $x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
      $x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
  14. I work in network management... by Ranger+Rick · · Score: 5, Interesting

    And I can tell you that if they're polling at 1 a second of *anything*, they don't "know what they're doing". That is complete overkill, there's no way the amount of bandwidth being used for testing is worth the 59-second jump on knowing what went wrong. Humans generally have to react to it, that kind of resolution is just crazy.

    --

    WWJD? JWRTFM!!!

    1. Re:I work in network management... by Guido+von+Guido · · Score: 2, Insightful

      Absolutely. This isn't monitoring--this is load testing.

  15. My Take on This by Bruha · · Score: 4, Insightful

    Okay so you're telling me that a 3rd party company is contacting your web customers and selling them monitoring services that you already provide and some other services that you may or may not provide. They then begin to access your system to do said monitoring but it's crashing your servers.

    Lets put it this way.

    You provide your customers a service. Part of that user agreement (This is doubly important in a shared server enviroment) that the customer cannot install any software/script/service that impacts the performance of the servers beyond what you say they can. Even the act of using 3rd party monitoring that is causing this problem is in violation of your AUP your customers are contractually binded to. Now I cant see your AUP but I hope there are provisions in there stating this.

    Now as far as the 3rd party company goes. You need to have your legal department file a cease letter to them with a explanation of the problems they're causing and until things can be worked out they are not permitted to conduct business across your network.

    You also need to notify your customers the actions you're taking on this company and why. Also pointing out your AUP/SLA's with them and the un acceptable behavior of the company that was selling them services. Tell them what you can monitor and explain what they really need.

    In the assumption of a web/email then all you need to do is monitor the ports and maybe a script that will verify the email server is accepting connections on a minute basis. That's all you need for that setup. Also if they're allowed to telnet into the box (SSH I hope) then you'd also monitor the SSH port as well to ensure they can connect to their equipment.

    If you're co-locating: Then I would suggest getting a Nagios setup running and sell some sort of monitoring to your customers. A good example would be the system that springboardhosting.com provides to their users. We use them as our colo partner and I've had no complaints. Though we only use the basic monitoring I do have advanced tools at the house and my laptop should I feel I need to watch any critical services. And I use webmin to monitor peer servers and page my phone in case there are any problems.

    You're in a pickle at the moment but I think your customers will appreciate cutting off the source of the outages. Nobody needs to know if their service is up by the second unless it's some sort of huge database application and then you'd have special provisions to monitor it and not remotely.

    That company is basically DDOS'g your servers to death. So it's basically them or you. I think the choice is simple :)

    Hope that helps.

  16. Don't let others eat off your plate. by NachoDaddy · · Score: 3, Informative

    From a business perspective, monitoring is a service *you* should offer to your customers. Since it is your network, you have the ability to provide a much more effective and accurate monitoring service, and can set the resolution of the service according to your customers needs. All the problems you describe are because they are operating from the outside. What that monitoring service is effectively doing is stealing your bandwidth, and selling to your customers. If you want to get your lawyers involved, send them a C&D since they are affecting your ability to conduct business. personally I would firewall then as the CTO has done, and offer the same service internally.

  17. I haven't been impressed with monitoring companies by eric76 · · Score: 5, Interesting

    A couple of years ago, a so-called "security expert" sold the president of my company on the idea of installing a firewall.

    To some extent, that was fine with me. I'd been arguing for that for a very long time but had gotten nowhere because the "security expert" said that firewalls weren't necessary! I guess someone finally bothered to break into his system.

    The security expert's idea was to have a third party monitoring company do it all. So I spent a couple hours on the telephone one day talking to the monitoring company's personnel about our network requirements and traffic. We went into great detail over exactly which servers had to handle which services.

    The firewall arrived and the security expert plugged it in. It didn't work at all. All it did was block everything. I was 600 miles away at the time and it took me a week to convince them to take it off.

    They decided the firewall was defective and the monitoring company set up another one. By the time it arrived, I was back in the office. The big day came and the security expert had one of his employees come out and plug it in.

    It didn't work at all.

    I caught the employee of the so-called security expert before he could leave the building and had him remove it. The idiot didn't even bother to check to see if it was working.

    After he left the building, I started looking at how he had it plugged in. He still had a cable plugged into the firewall from an internal hub.

    He had connected the untrusted side of the firewall to the internal network. I assume that the cable from the Cisco router was plugged into the trusted side of the firewall.

    But it really didn't make much difference. I also found the rule set for the firewall. The monitoring company had set it to pass nearly everything in both directions.

    The only thing they configured was to block incoming traffic containing our IP addresses. Since it was plugged in backwards, it really just stopped all traffic from going out.

    At this point, it would take a lot of convincing to get me to advocate using a monitoring company's services.

    By the way, the same so-called "security expert" declared that rules on the Cisco router to block traffic attempting to connect to port 135 and other similar ports constituted a security list and removed them.

  18. It's your own fault... by LostCluster · · Score: 4, Insightful

    Your system should have been set up to attribute the log file to the disk space of each client, causing them to eventually hit their limit and lose their abilty to log any further. No set of requests from the outside world should be able to bring down your server short of a vicious DOS attack, which clearly this wasn't. This was a an overload level of legit traffic, if your server can't handle it then you need a better server.

    You should be able to create a few new services and convince your clients that they don't need to pay a 3rd party to monitor their server, that you can tell them all they need to know, and besides that you don't go down anyway. :)

    It would have been an absoulte fiasco if one of your customers were to attract a Slashdotting...

    1. Re:It's your own fault... by sgtrock · · Score: 4, Insightful
      This was a an overload level of legit traffic, if your server can't handle it then you need a better server.


      WHAT????? What planet are you from that doing ANY kind of network monitoring once a second is considered legit traffic? No, this was either a deliberate attempt to generate a ton of false positives, or total incompetence on the part of the monitoring company.

      If I were the owner of the hosting company, the FIRST thing that I would have done was refuse all requests coming in from the monitoring company so I could get traffic flowing for all my customers. That is what they are paying for, after all.

      The second thing that I would have done would be to save off copies of all logs that might be considered relavent in a legal situation to read only media.

      The third thing that I would have done is send out an email to all affected customers explaining the reasons for the downtime incurred, what had been done to alleviate the situation for all concerned, and that further efforts were ongoing to resolve the issue permanently.

      Then, call my lawyers. Ask for a Cease and Desist order to be sent right away.

      No way do I play nice with assholes trying to put me out of business.
  19. Monitoring Report: by Snoopy77 · · Score: 2, Funny

    It seems there has been an unusual amount of downtime to your web and email servers. Probable cause: we over monitored them. Sorry.

    --
    "She's a West Texas girl, just like me" - G.W Bush Iraqis
  20. Re:Confidentiality & TOS & Abuse by vt0asta · · Score: 4, Informative
    What he said...
    Sounds like you've got an open and shut legal case to recoup those costs they're causing you to incur.

    First things first. These are your servers. Your network. I am assuming you have the standard abuse clause in your TOS. You need a lawyer.

    Unfortunately, you are in a bad situation. They apparently have more resources than you, because they can bring your setup to it's knees. Not saying it's right, not saying it's fair.

    A lookup of your TLDs each second makes sense if you are Yahoo! or Google. Their web monitoring levels don't appear to be reasonable. You already know the technical answer.

    Personally, I would be worried about them stealing your customers. I mean the argument is going to be simple from their side. They will simply say, "hey look, their stuff folded under 'normal' monitoring, we have a hosting company we can 'recommend'" or they will just have the hosting company call them up out of the blue and ask if they are "unhappy" with thier current service..."oh, it goes down a lot"..."they can't handle simple monitoring"..."gee, that's a shame"..."well, we've worked with that monitoring company before, and we have never had any problems, in fact we routinely get 5 9s"...etc

    Honestly, talk to legal, explain the potential situation, and have them make contact with the monitoring company. A couple of tortious interference this, and cease and desist that, will put the monitoring company on it's toes and maybe get them to leave your customers alone, or possible play nice with your servers. Notify your customers yourself and explain that they are being investigated by your legal team, etc.
    --
    No.
  21. Way over the line by jmitchel!jmitchel.co · · Score: 2, Interesting

    These guys don't know what they're doing if they are banging on your servers every second. It is a strategy that is bound to make any competent admin irate and probably break things. Anything more than once-a-minute is probably overkill. Once every 5 minutes is a good window for most things. Your people are quite entitled to block them at the firewall.

    Your sales people have to figure out how to appease the customers. That's their job. You are a tech and you'll just foul things up using tools like fairness and logic. I've been there.

    Lastly, if they overflowed your log partition, you aren't monitoring enough things. It isn't enough to make sure that your sites are up, you need to make sure that the disks they depend on have enough free space, that the servers they run on don't have unnacceptible load spikes, etc... Comprehensive solutions are hard, but quick-and-dirty solutions aren't. Remember though that it's hard to send pages from a dead server and design accordingly.

  22. Feel justified by mr_z_beeblebrox · · Score: 2, Insightful

    Monitoring your servers is a security function. A security company should strive to appear beyond reproach. Wether they got your customer list by looking through your ip logs or from a former employee, that is unsuitable behavior. I would contact my customers tell them that a security firm you do business with has "acquired" a customer list of yours and you are unsure of their intentions but you are sure that they acquired it dishonestly. None of your customers will hire them. The down side is, be careful not to tell your customers in a way that makes you look stupid, because you might look it.

  23. Ask for compensaton for their stupidity. by cenonce · · Score: 3, Insightful

    It seems to me that unless your company signed some kind of waiver in case their monitoring did any damage, you have a case for negligence.

    Even with a waiver, generally, you can't waive somebody's negligence. Their actions sound negligent in that they used excessive resources such that your servers crashed.

    Additionally, it sounds like there may be some form of defamation claim when they complained to your customer base about you. Though defamation claims, especially slander (spoken defamation), are thorny claims that can be hard to prove, it sounds like you may have a number of incidents that may show intentional defamation (much better when seeking damages).

    I think, at the very least, your general counsel should be asking for compensation for your downtime.

    -A

  24. This is not a reputable company by Gunzour · · Score: 3, Insightful

    They either got a hold of a customer list from a former employee or walked our IP space to find our web hosting customers. They then proceeded to sell them monitoring services for things such as server up-time, defacement detection, email up-time and DNS testing.

    In other words, they upsold your customers without your consent. That in itself it unethical and any thought in my mind that this is a 'reputable' company would go away at that point.

    You go on to describe how they DoS'd your boxes, and complained to your customers when you took action to protect your customers from the DoS attack.

    If their behavior is really as you described, why are you bending over backwords to say how reputable and legitimate they are? They are neither.

  25. Tort, on the case by debrain · · Score: 2, Informative

    This is not legal advice. Find a lawyer, ask them what to do.

    It seems as though you've got a tort of negligence on your hands, insofar as they seem unaware, or oblivious to, the damages they are causing you. They do not seem, from your statements, to be wilfully causing damages, but negligence torts need not show (at least in the commonwealth) either wilfulness or intent. You need only show damages, which are an indirect consequence of their actions.

    Take into account that torts are, by most accounts, very expensive, though the threat of a tort is often sufficient, or binding arbitration (though that is apparently not oft met with success), or mediation (same deal as binding arbitration). If you do have to litigate, the general rule is somewhere north of $100,000 in damages to justify the transaction cost, from what I have heard. See the first line, though - find a lawyer.

    In the least you can establish damages in support of a trespass if you inform them that their actions cause damage, in which case their actions are thereafter wilful, which may make for a cleaner case. The onus in trespass is on the defendent (them) to defend against damages established, not the plaintiff (you); and whereas in negligence, the onus is on the plaintiff (you) to show damages.

    Ok, so in gist, take everything I said with a grain of salt, and seek legal counsel. Your jurisdiction may have many options with respect to small claims or public dispute resolution, and I would suggest those because they are significantly cheaper.

    Hope that helps.

  26. Or better yet... by ProfessionalCookie · · Score: 4, Interesting

    Charge for it. Notify yer customer (by perl of course *tee hee*) that their logs are causing their account to approach its space limit. They can either move the logs, delete the logs, stop the logging software or remove the logging software. Warn them that if this is not taken care of additional hd space fees will apply.

    Make sure they know that cleaning up logs should be *cough* easy and pain free!

    1. Re:Or better yet... by PhilHibbs · · Score: 3, Insightful
      AIUI, the logs were the poster's internal logs, not the customers'. The third-party monitoring company was querying the servers and sending the emails, and if the first and second parties didn't have a charging agreement that covered this kind of usage, then he's in trouble.

      p.s. Why is using perl funny?

  27. Alert your community of users by BanjoBob · · Score: 2, Insightful

    There should be no reason to add 3rd party security IF your security is in place. There are a lot of ways to protect your environment that do not require outside monitoring.

    Alert your users of this fact - send them all an E-mail to alert them of this scam!

    You run the show -- not some 3rd party. You set the rules and the security policies. You do the monitoring internally.

    I can't believe that monitoring consumed 15GB of space. There's something else going on there. I helped work on a data warehouse to capture all of Worldcoms routers data every 5 minutes -- every router's SNMP logs and for years dumped all that data into an Oracle database so we could report on it. That's a bunch of routers and a ton of data. For your company to consume that much log data in a single weekend doesn't make sense.

    Block the 3rd party polling IP at the routers and do the job internally.

    --
    Banjo - The more I know about Windoze, the more I love *nix
  28. Fix the contract. by Spazmania · · Score: 4, Insightful

    If I understand you right:
    1. You have some customers to which you sell services such as email and web space.
    2. Some of these customers contracted this monitoring service to watch the servers.
    3. The monitoring service caused problems with your servers.

    And the answer is:

    Correct your hosting contract. Your hosting contract should include provisions for how much usage is reasonable and how the situation will be handled when the customer's usage exceeds those parameters. If the customer insists on doing something stupid which brings the server to its knees, then the customer should pay you enough for you to be able to afford a seperate server for them.

    If the sales force insisted that they'd lose sales by bothering the customer with such notions, now would be an excellent time to point out that they just lost sales because they didn't.

    As to how much monitoring is too much, the answer is simple: anything the customer is willing to pay for is fine. Anything more is too much.

    --
    Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
  29. DoS by dosius · · Score: 2

    It's a DoS attack on your systems and should be treated as such (especially as you warned them already and they ignored you).

    Block them off and take them to court.

    -uso.

    --
    What you hear in the ear, preach from the rooftop Matthew 10.27b
  30. Depends on how by KalvinB · · Score: 3, Informative

    If they're letting their logs get huge before rotating them it would cause a problem every time the server tries to append data at the end of the file.

    And they shouldn't be keeping the logs on the server anyway. It's static data that only they could need access to. It should be moved off site to a standard IDE harddrive for processing.

    Statistical data should be created as the data comes in and not from the log files if they intend to let the customers have statistics for whatever.

    As for my own site, I have Apache doing the combined log format and wrote custom software to process and analyze the data. Every month I move the log off the server and every 10 megs or so I rotate the logs and move the data into a second cumulative file that Apache doesn't work off of.

    Ben

  31. Monitoring Strategy by Anonymous Coward · · Score: 2, Informative

    It is very important for a bigger hosting firm to have a good moniroting strategy which shows trhe external perspective.

    The timing need not be more than 15 minutes in most cases. The plan should include the network, web server and applications, and possibly supporting servers such as email or DNS.

    The external capabilities are critical - if you are going to do external, use a firm who has profressional managed remote stations in many places.

    Tim Goeke
    http://www.globalnetwatch.com

  32. Are you kidding me? by dan14807 · · Score: 5, Insightful

    Why are you putting up with this crap?

    As several posters have already mentioned, firewall them off, and then report them to the legal authorities.

    Jesus tap-dancing Christ! They are attacking your network. I feel like flaming the original poster for his incompetence. Acquire the BOFH nature. After you firewall them, file a report with the FBI's cybercrime division. Tell them you are a hosting company, and you have the IP of someone who is costing your company $BIGNUM dollars per day because they are DOS-ing your network. That should keep this "monitoring company" busy for a while, and it will teach them a lesson.

    Whining about it on slashdot is the last thing you should be doing. Get a clue.

  33. Re:To be expected by pantherace · · Score: 2, Insightful
    When in all actuality, uptime and overall accesibility of a site are all that a lot of webmasters care about.

    Ha, tell that to all the webmasters with non-compliant HTML out there.

    I just thought of a good idea, a web page upload form or something which scans the webpages which gives a nice little dialog about a webpage being non compliant, and may not display correctly in many browsers :) Now to get ANY ISP to implement it... HA!

  34. Re:To be expected by macdaddy · · Score: 2, Insightful
    That sounds exactly like all the host-based personal firewall products on the market today. They have to tell you every little thing that's going on all the time and they absolutely MUST sensationalize EVERYTHING.

    "Oh dear God! You've been pinged! The sky is falling!! Whew. It's a damn good thing you installed our over-priced over-hyped personal firewall thingy because we just saved your ass!"

    Think I'm kidding? Don't. These ass clowns prey on guilible users that simply don't know any better. It's just like what many auto repair shops do to those people whom they don't think know jack about cars.

    The belt on your carburetor are about to break. We also had to grease you exhaust bearings and reprogram your warp convertors. That'll be $700 please.

    If only we can eliminate stupid people and those that would prey on them (including the media) the world would be a much better place.

  35. do we all have SUCKER imprinted on our foreheads? by PhreakOfTime · · Score: 4, Interesting

    ok ok...about the only thing I find remotley factual in this article is the fact that this guy works for a 'company'...however it looks like he works for a company doing exactly the things he is asking about.

    First of all, lack of any knowledge of partion or disk utilities to prevent such an occurance is unacceptable. I would not admit that in public about my company even if I used the phrase 'a company I work for', just on the off chance my negligence would be able to be tracked back to me.

    Second, why are you not able to offer these services yourself? You make a claim that these people know what they are doing, so if you are at such a level to recognise what they are doing, how come you havent done it already? Did customer service become just a novelty to you? so I doubt this line very much... While I welcome anything that lets our customers use the internet effectively

    Doing hosting myself, Im well aware of the tactics you speak of, being that I get bounce mail for nonexistant addresses sent to such titles as; president, ceo, owner, support, tech...and so on. And Im not sure exactly what you mean by 'choked up' your mail server. How do 40k NONEXISTANT addresses manage to slow down your mail server? Is it a 286?

    The whole article just smells funny to me, as it seems like you are just pretending to care about the ISP's end and more concerned about the backlash of doing these things. What do you mean how far is to far? Again, if the people in charge cant figure these things out on their own, I would be very hesitant to admit that in a public forum.

    Get your technical skills and decision making in line...THEN question how to outsource it..

  36. A couple of comments by taustin · · Score: 3, Interesting

    "Though I believe they are a reputable company, they are doing some things I do not think are good: checking for the domain names on the TLD servers once per second, downloading various files from the site once per second, and sending email to themselves once per second.

    They are not a reputable company. They are a bunch of retards who should be driven out of the industry with sharp sticks. More to the point, they should be reported to the FBI for conducting a malicioius attack against your network - and you have tangible damage to prove it.

    Our first response was to talk to them and explain what we needed them to do, including a list of IPs that we used for customers so they could adjust their monitoring to suit what we thought was reasonable. They chose to ignore the first discussion and continued to abuse the servers. After the email server required a half-day of cleanup, the CTO simply shut them off at the firewalls. Rather than using the contact information they had, they chose to complain to our mutual customers instead. (I should note we do significant monitoring of the servers ourselves, and typically know if something is wrong within minutes of the event.)

    Is this typical behavior of monitoring service companies? I know some of them are not reputable at all (due to spamming) however these guys seem to know what they are doing, and yet managed to effectively attack our mail and web servers, as well as doing some things I would not do to the TLD servers. It is hard to feel justified to shutting off someone else's cash-flow, but at the same time we need to defend servers from over zealous monitoring."


    Here's a hint for you: Do they offer web hosting services themselves? You may have to dig real deep to find the connection, but if I had to guess, I'll bet they do. And I'll bet they offer it to your customers, based on the fact that they crashed your servers. "Your current service seems to have a lot of downtime. Perhaps you should consider moving to another host. We can make recommendations."

    If you find any evidence that they offer any kind of competition to your hosting, report them to the FBI. They may well be a criminal organization engaging in a well orchestrated scam.

    Or maybe they're just fucking stoopid. It's hard to tell from here.

  37. Heisenberg and monitoring by Morty · · Score: 2, Insightful

    One of the biggest problems with monitoring something is that you inevitably affect it, a la Heisenberg in the Physics world. The more closely you try to monitor something, the more you affect it. This is a basic principle of monitoring.

  38. Slashdot the MFers by mabu · · Score: 3, Funny

    The solution to this is simple. Publish the web address of this loser monitoring company and we'll let Slashdotters "check the integrity of their system."

  39. Something smells fishy here by darkonc · · Score: 5, Insightful
    I mean jeez that must have been thousands and thousands of hits to use up that much space.

    $ units bits/second bits/day
    * 86400

    So you're looking at (roughly) 100K hits per day per file downloaded per site. If they're downloading 15 files per site, and you've got 100 sites on the box, then you're looking at an increase of about 120 million requests per day. My acess log has an average of 200bytes/er line, so you're now looking at 120Mrequests*200bytes/request == a sudden jump of 24gigabytes of logging per day.

    Then you've got the effective mail-bombing to deal with.

    The article author said that these people sounded like they know what they're doing, so that leaves (in my mind), two likely possibilities:

    1. They're really really good snow-job artists. They understand the terminology, but they have no real sense of methodology or purpose.
    2. They really do know what they're doing, and they're trashing your servers with intent.
    I mean -- for crying out loud: Multiple files once per second? And just how long did it take them to inform your customers that they'd managed to crash the servers? Monitoring granularity of more than about one quarter the normal notification time is a complete waste of resources -- and that's giving them lots of leeway to waste.

    And Tens of thousands of undelivered emails??? If those emails didn't get delivered, then what did the company do when they didn't arrive in short order? Why didn't they stop the transmission and diagnose why the emails weren't coming thru? If the emails really are undeliverable, then how in the world did you manage to conclude that they know what they're doing?

    Other notes (mostly mentioned elsewhere)

    • are you charging your customers based on their net volume? If so, have you informed your customers of what sort of costs these, uhm, people are imposing on them in addition to their monitoring fees?

    • I'm guessing that your AUP includes a clause on activities that wilfully or negligently cause inappropriate server load, outages, etc. I think that this company's "services" classifies.
    • I think that you had better seriously consider possibility #2 above. Meticulously document what they've done to your servers (including somehow scamming your customer list). Have that information ready to present to your customers and/or a judge. If all goes well, you won't need it, but I'm not expecting all to go well, given how they've gone so far.
    One last point -- Even though you may be dealing with a company that you think has a (otherwise) good reputation, doesn't mean that you're not dealing with an inept department of an otherwise good company. Sometimes the VP Engineering puts his/her stupid cousin in some group where they're not likely to do much damage, and then finds out that the goofball has managed to get out 'in the wild' with a 'bright' idea.
    --
    Sometimes boldness is in fashion. Sometimes only the brave will be bold.
  40. Website Monitoring and Your Customers by todd1000 · · Score: 2, Informative

    I work for a large hosting company. We have a lot of customers who have monitoring companies monitor their websites (we actually use some). We obviously monitor our services ourselves, but it is not always objective doing this. Having said that, monitoring once per second is *stupid*, generally 5 minutes is appropriate and we monitor some things internally every 60 seconds. We charge for bandwidth and disk usage (including logs), so if people want to monitor every second, go for it, your credit card will get dinged next month. For a smaller provider, I can see this being a problem, I would blackhole the IP. It is a DOS attack and I'm pretty sure you would have the legal right to do that. You do have a provision in your policies that you can take necessary action to protect your network, right? We do and will use it when necessary. Right after 9/11, we had a *very* popular and large image on our servers (the "eagle", if anyone has seen it). We "chmod 0"'d it and called the customer. They didn't realize what happened (getting so many hits), understood (once we explained bandwidth charges), and where happy we did it. Monitoring every 5 minutes is reasonable and will catch almost all outages.

  41. DOH ! gotta stop smoking weed .. by Tensor · · Score: 2, Funny

    I meant VERISIGN and not Verio

    And SITEFINDER instead of seeker. dammit

    Now why didn't i pressed preview ?

  42. One More Word by MoZ-RedShirt · · Score: 2, Informative
    --
    Microsft spel chekar vor sail, worgs grate !!!
  43. Lousy sysadmins by SuperBanana · · Score: 2, Insightful
    Charge for it. Notify yer customer (by perl of course *tee hee*) that their logs are causing their account to approach its space limit.

    How about partitioning your servers properly so they don't crash when they fill the logs?

    Basic sysadmin 101, people. You're going to piss off customers by doing what the parent suggests.

  44. Admin on the street says.. by gosand · · Score: 2, Funny
    I am interested in finding out what admin-on-the-street has to say about this.

    Admin-on-the-street says "I need a job, you insensitive clod"

    --

    My beliefs do not require that you agree with them.

  45. Re:Confidentiality & TOS & Abuse by orkysoft · · Score: 2, Interesting
    Notify your customers yourself and explain that they are being investigated by your legal team, etc.

    Note: OP means the monitoring company with "they". Pissing off your customers by telling them that they themselves are under legal investigation is a kind of SCO idea...

    --

    I suffer from attention surplus disorder.