Slashdot Mirror

← Back to Stories (view on slashdot.org)

Preventing My Hosting Provider From Rooting My Server?

Posted by Soulskill on from the booby-traps dept.

hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?" Read on for a few more details of hacker's situation. "With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.

What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."

15 of 539 comments (clear)

  1. If they do this.. by sopssa · · Score: 5, Insightful

    .. just switch providers. I'm sure there are companies that treat you better.

    1. Re:If they do this.. by drinkypoo · · Score: 5, Informative

      Second this. Isn't it an adage that someone who has access to the hardware has already won? Secure some solid evidence and publicize it on your way off the host.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    2. Re:If they do this.. by jcr · · Score: 5, Insightful

      First, check your contract and make double sure that you didn't give them permission for this, and if not, go ahead and file charges.

      -jcr

      --
      The only title of honor that a tyrant can grant is "Enemy of the State."
    3. Re:If they do this.. by JeffSh · · Score: 5, Interesting

      I might ask for more evidence that the provider actually rooted the server before pronouncing judgment. I'm not saying that the person posing the question is lying, but simply because I don't have enough evidence either way.

      Highly intelligent people tend towards a sometimes unreasonable paranoia and sometimes make conclusions (i.e. my server was rooted to look at the logs) that are not exactly true.

      That said, I don't know either way really. It could be argued one way or another. If I were a provider, I might even insist upon the ability to access systems running on my network simply because of liability concerns as the provider. I as the provider can't be allowing untoward activity on my network.

      That all said, and without actually proclaiming judgment one way or another, in the end if you're not happy with your provider for any reason, whether reasonable or not, you should just leave them and find a new one.

    4. Re:If they do this.. by DamonHD · · Score: 5, Informative

      Bogons, UK

      GetNetworks/JavaServletHosting, US

      WebVisions, AsiaPac (currently India and Australia)

      Rgds

      Damon

      --
      http://m.earth.org.uk/
    5. Re:If they do this.. by dave562 · · Score: 5, Insightful

      As a network admin, I've run across "I know what I'm doing" people in the past. FWIW, I'm often times that guy when I'm calling tech support. It's one part ego, one big part actually knowing what I'm doing. I don't want to go through tech support 101 with some monkey on the phone when I know what the issue is.

      Having said that, there have been times when I thought I knew what the issue was, but it turned out to be something else. I think that a hosting provider wanting access to log files is perfectly reasonable. They aren't arbitrarily asking for the files. The questioner states that he is having problems and he asked them to sort it out. Tech support 101 says to look at the log files. The questioner doesn't make it clear whether or not he offered to give them the log files.

      Is the hosting provider a bit off base? Yes and no. Yes, it's kind of lame that they are rooting boxes. On the other hand, the questioner might be more problems than he is worth from their point of view. If I were in the same situation, I'd just change providers and find one who will put into writing that they won't root my box (good luck with that).

      (Car Analogy) - It's like leasing a car with a repair warranty and wanting to do your own repairs. You diagnose the cause of the problem and take the car to the mechanic. You ask the mechanic to fix your car under warranty and he asks you for your keys. You refuse to give him the keys.

      It seems to me that if a person can't fix a problem on their own, and that person then asks for help fixing the problem, they need to give up some control to the person they have asked for help from. Unless a person selects a hosting provider with an SLA that will give them physical access to their hardware on a 24/7 basis, that person is going to have to make some accomodation (like providing access to log files) when the hosting provider needs to get involved with troubleshooting.

  2. Just.. by roblarky · · Score: 5, Funny

    Be sure to stun them as soon as they start casting it.

  3. This is very simple by rgigger · · Score: 5, Interesting

    1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.

    2. Figure out how they broke in. If they broke in then someone else likely could too.

    I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.

  4. Other side by Spazmania · · Score: 5, Interesting

    On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.

    They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."

    --
    Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
  5. Stop being a douche by jascat · · Score: 5, Insightful

    As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.

    1. Re:Stop being a douche by Sargonas · · Score: 5, Insightful

      Agreed! What you are asking and what you are wanting are an unreasonable combination. Take a step back off your sysadmin high horse ( I am allowed to use that term, since I too was once on one) and look at it from their point of view. You are sending them WTF tickets and at the same time refusing to "help them help you". Honestly, what do you expect?!? Agreed they should not be rebooting your box to get access without first warning you, but at the same time you are demanding a response asap and then withholding critical info from them. What do you expect them to do? As the above poster said, either create a limited account for them with only log file access, or else man up and just give them a full login. I will bet all the money I have made in my previous career as a sysadmin for several large companies and hosting companies that in your hosting terms it clearly states they own the system, hardware and software, and that you have no inherent right to deny them access. (unless we are talking about a co-located server you personally own, but since you did not state that I can only assume we are not.) In short, you are being a jerk. Get over yourself and either A: work with them to help you, B: diagnose your own damn problems and stop asking them to without giving them the help they need, c: change hosts to someone who more suits your needs, d: colo you own box in an IBX and handle all the work yourself.

    2. Re:Stop being a douche by ShinmaWa · · Score: 5, Insightful

      You say this

      I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.

      ....however, from another post you let the truth slip out

      they moved my drive to a different chassis, with completely different hardware, and are asking for the root password so they can reconfigure everything to coincide with that hardware change (...LATER...) When they migrated it from Savvis to some datacenter in Dallas 2 months ago.....

      So you openly admit the machine IS NOT YOURS. You are essentially keeping them from their own machine, which I find unethical. I can't blame them for taking matters into their own hand and rebooting the system into single-user mode and locking you out until you play nice.

      Stop being a jerk and cooperate with the owners of the machine you are renting or take your data elsewhere.

      --
      The /. Effect: Thousands of users simultaneously accessing a site to not read its content.
  6. Tough question... by couchslug · · Score: 5, Funny

    "How do I turn a whore into a housewife?"

    Some things are only solved by replacment.

    --
    "This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
  7. Re:How do they Root your Box? by hacker · · Score: 5, Informative

    "How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access?"

    They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.

  8. you might be our customer by Eil · · Score: 5, Insightful

    Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:

    1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.

    2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.

    You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.

    The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.