Slashdot Mirror


Ask Slashdot: Remote Server Support and Monitoring Solution?

New submitter Crizzam writes I have about 500 clients which have my servers installed in their data centers as a hosted solution for time & attendance (employee attendance / vacation / etc). I want to actively monitor all the client servers from my desktop, so know when a server failure has occurred. I am thinking I need to trap SNMP data and collect it in a dashboard. I'd also like to have each client connect to my server via HTTP tunnel using something like OpenVPN. In this way I maintain a site-site tunnel open so if I need to access my server remotely, I can. Any suggestions as to the technology stack I should put together to pull off this task? I was looking at Zabbix / Nagios for SNMP monitoring and OpenVPN for the other part. What else should I include? How does one put together a good remote monitoring / access solution that clients can live with and will still allow me to offer great proactive service to my servers located on-site?

22 of 137 comments (clear)

  1. Reverse-SSH tunnel phone-home from remote device by Anonymous Coward · · Score: 2, Informative

    Set up a script to initiate a reverse-SSH tunnel from the remote device back to a monitoring server, set up no-login on the tunnel but distribute keys for the monitoring user on the remote devices.

    You should be able to passwordless login from the monitoring box over a completely secure link that doesn't require port-forwarding at the remote site.

  2. Scratch my back by Anonymous Coward · · Score: 2, Interesting

    Will you do my job if I tell you the answer? You've already gotten your start. What more do you need?

  3. I just discovered NewRelic ... by WayneDV · · Score: 4, Interesting

    Check out www.newrelic.com - even their free service tier offers great features and it's easy to deploy on all servers

    1. Re:I just discovered NewRelic ... by astro · · Score: 3, Informative

      NewRelic is pretty sweet, as the parent says, even at the free tier. They will definitely bombard your email and phone with hard-sales pitches, though, and there's a giant cost leap from free to the next tier.

  4. Re:Central server by Noah+Haders · · Score: 2

    Would this centralized server be your universal remote server?

  5. Ping? by danknight48 · · Score: 2

    For Server active status (eg: am i dead?)
    Inside a while loop or sleep() if you cant be bothered.
    for(int i=0;iMAX_SERVERS;i++)
    {
              IcmpSendEcho(..........);
    }

    For everything else monitoring related. Employ someone to make a custom monitoring application ,or, Google "server monitoring software".

    1. Re:Ping? by Enry · · Score: 3, Informative

      For some reason, disabling ping is considered a security feature, so a lot of places block it at the firewall. Cloud services (I'm looking at you, Azure) also either doesn't allow it or can't do it.

  6. Re:reverse ssh by aheath · · Score: 2

    I agree that this creates the potential for a hug security that has the potential to compromise the privacy of all of the employees at 500 companies. The consequence of this breach might be worse there is a connection between his servers and a payroll system or any point of sale system. I also wonder his clients are willing to open up the ports required to support remote access to their data centers.

  7. Openvpn and x11vnc by Wycliffe · · Score: 2

    I do something similiar. I use openvpn and x11vnc. I have a cron on each client that runs a
    small perl script that grabs the output of several programs like top, uptime, and sensors
    and then saves the results in an easy to parse file that my server periodically grabs so that
    I have stuff like cpu temperature, cpu usage, memory usage, etc...
    I also grab a screenshot of x11vnc using vnccapture.
    I also have a way to remotely activate reverse ssh if for some reason openvpn fails.
    My only problem with openvpn is key management. Creating and distributing unique keys
    to each client is kindof a pain.

  8. Re:Or you could by Idimmu+Xul · · Score: 3, Informative

    Nagios is Open Source.. GPL V2 specifically..

    --
    The problem with slashdot is that most of its users were bullied and stuffed into lockers as kids!
  9. Hopefully this goes without saying by 93+Escort+Wagon · · Score: 3, Insightful

    Make damn sure your clients are aware of exactly what you're doing. They probably don't care about the specifics (e.g. openvpn, reverse ssh); but they need to know you can remotely access the boxes.

    It's probably a good idea to have some sort of document to give them that does spell out all the specifics - something they need to acknowledge/sign, with both of you keeping copies.

    --
    #DeleteChrome
    1. Re:Hopefully this goes without saying by dskoll · · Score: 4, Informative

      Actually, the model of remotely-managed on-premise appliances is not that crazy. Assuming it's done securely, you get the best of both worlds:

      If the customer's Internet access goes down, they're not dead in the water as they would be with a cloud solution.

      If you manage everything for them, then the box is completely hands-off... just like a cloud solution.

      There's an entire business category called "Managed Service Providers" whose vendors do exactly this: Remotely manage all aspects of your IT infrastructure so you don't need to worry about anything. For mom-and-pop non-technical businesses, it's an excellent model.

    2. Re:Hopefully this goes without saying by dskoll · · Score: 2

      The fact that a well-managed cloud service is multiply-redundant is of little consolation if your crappy DSL line goes down for 6 hours and your salespeople cannot access the CRM tool.

      What's more likely to happen: the loss of access to Amazon cloud services/internet, or a local box getting cacked

      Unequivocally for us: Loss of Internet access happens far more often than a server failure.

  10. zabbix is NOT an snmp manager by TheGratefulNet · · Score: 2

    not really. snmp is an afterthought for them and its clumsy as hell to add snmp to it. I tried and gave up. instead, I picked hobbit (uhm, the new name is 'xymon').

    xymon has its quirks but it was not hard to modify to add more snmp features to and its coding was not too bad to get thru. its not written in a lot of 'strange' languages, and that's a plus, to me, too.

    personally, I usually just write snmp code fresh, from scratch, using net-snmp mgr tools. its not hard and you get just what you want and you are not muddled down in lots of 'infrastructure' that someone else thought was good but useless to you (like zabbix).

    --

    --
    "It is now safe to switch off your computer."
  11. Re:Keeping track.. by fearlezz · · Score: 2

    You'll need a means of knowing that 10.20.20.x is client x and 10.20.20.y is client y. Of course OpenVPN allows you to do this but maintaining that table by hand could be a bit of a pain.

    You mean like the common name of the ssl certificate used to connect in the first place? Combine this with a client-connect script to update dns and/or the ifconfig-pool-persist option and you've got a great solution.

    --
    .sig: No such file or directory
  12. Re:Keeping track.. by dskoll · · Score: 2

    Managing the OpenVPN connections is not that bad. You give each client its own key and certificate and you use OpenVPN's ccd/ directory to assign VPN IP addresses.

    We use the following tools to monitor our servers, but we're only monitoring about 30, not 500:

    • OpenVPN for accessing the remote servers. SSH if we need to log on to the server to do something. Some of our more important servers include built-in KVM-over-IP ability which can be very handy if the OS locks up.
    • Xymon (formerly known as Hobbit) for monitoring the health of remote servers. We include some custom Xymon plugins to monitor SNMP variables. I find Xymon much easier to configure than Nagios, though it's not quite as flexible.
    • Munin for tracking performance and ensuring we have baseline data.

    I'm not sure how well this would scale to 500 boxes, though Xymon claims to be able to monitor "lots of systems".

  13. Re:PRTG by chipperdog · · Score: 2

    NAV has very similar functionality to prtg, but is completely open source.

  14. Re:Reverse-SSH tunnel phone-home from remote devic by BitZtream · · Score: 4, Insightful

    Or, do the right thing and hire a network admin so someone with a clue is involved.

    If you have to ask this question on slashdot, you need to change the question to something appropriate. Based on exactly what was posted, he doesn't have any idea what his requirements are. He knows the conceptual goals, but not the actual goals or requirements. Unless he is trying to change careers from whatever he is to a full time network infrastructure person he is going to be wasting a lot of time getting a clue. That means time he won't be spending doing whatever his actual job is.

    He needs someone who can look at his actual setup, figure what what actually needs monitored, and knows the appropriate ways to do it.

    Short of multiple Bennett hasleton length posts, and many discussions in depth, no answer coming from slashdot or all of them combined is going to be useful.

    Everyone here posting solutions has their own, certainly incorrect idea of what he wants but no one actually knows. No one so far has even started by asking the right questions. It's the blind leading the blind at best.

    --
    Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
  15. GFI MAX by DigiShaman · · Score: 2

    Problem solved. Next topic please.

    http://www.gfimax.com/

    --
    Life is not for the lazy.
  16. Re:Reverse-SSH tunnel phone-home from remote devic by BitZtream · · Score: 2, Funny

    Just because you're unfamiliar with networking administration doesn't mean this needs to blown up into "hire a network guy". That's just ignorance and

    As someone who's been a network admin for a few years, I'm fairly confident in my statements. Do you do even minor surgery on yourself if you're not a surgeon? If you come to slashdot to ask how to do something for your business, you already fucked up and the only valid responses you should be getting from slashdot are help on finding someone who can help you. If he asked 'how do I find someone, like a consultant for a short term project, like this' that would be one thing. He didn't, he came here expecting a solution which illustrates his complete lack of understanding of the problem, THAT IS WHY he needs to hire a network guy.

    He is, by definition, ignorant, which is why he is asking for help ... clearly you are as well as your choice of words indicates. I suggest you learn what the word ignorant means before you brandish it about like an insult as you just end up insulting yourself through your own ignorance.

    (I suspect) trying to make yourself sound important on an anonymous message board.

    I have no need to make myself sound important, I certainly don't need your approval ... and if you bother to google for my nick, you'll find its not even a little difficult to link to a real name, address, and everything else. I'm not in the least bit anonymous. People have been able to recognize that nick and its association with me for 20+ years. On the other hand ... your post ... is from ... anonymous coward. Do you know the meaning of the word ironic?

    As my granddaddy used to say, if you don't know what you're talking about, it's best to not open your mouth and prove it. So no need to apologize, just take the advice and consider it a lesson learned. Best of luck.

    Your grand daddy said that too you a lot, didn't he? Did you ever wonder WHY he said it too you so much? Maybe he was trying to get some sort of point across to you ... Go look in the mirror and repeat those words until you get the point of them and who he was talking about. Hint: Its the guy in the mirror.

    You're an absolutely shitty troll. You just suck at it. Nothing you've said did anything other than show how stupid YOU are, not me.

    --
    Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
  17. Re:Keeping track.. by dbraden · · Score: 2

    There's no need to install Ansible on the remote systems, only on the machine running the playbooks. All Ansible activity is run over SSH and has no remote dependencies.

  18. Re:Reverse-SSH tunnel phone-home from remote devic by pspahn · · Score: 2

    You sound like a Windows admin for a gov't entity.

    You spend a lot of energy telling people they do it wrong without having any real insight or advice on how to do it correctly.

    A blanket statement like this shows your cluelessness and shear ignorance.

    What does his knowledge of a specific cutting tool have to do with anything?

    --
    Someone flopped a steamer in the gene pool.