Slashdot Mirror
Preventing My Hosting Provider From Rooting My Server?
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
.. just switch providers. I'm sure there are companies that treat you better.
Sounds like this is in your hosting contract. Either switch, or if your that concerned, host it yourself, not in a data center. Every data center is going to say "Prove it" if you try to pin an issue on them.
look for a pre-authorized ssh key in ~/.ssh/authorized_keys or something similar, remove it.
As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.
You call yourself "hacker" but you don't already know how to do this.
If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.
As others have pointed out
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
Hello, I work for a very similar company that provides support. How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access? That would surprise me greatly because no one in this field would care enough to go through the trouble of a sophisticated hack of your server, and besides, if they could do it, so can anyone else. Because of the hazy situation here, I'm going to assume that you are running this "server" as a VPS as opposed to a dedicated server plan. If that's the case, then they can easily log into your root account because your server is already run under VMWare. Chances are they asked you for your password in order to bypass looking up the vzid of your container. After that, it's typical procedure to restart the container if you're eating up massive resources. That will usually clear out the http/svn/mysql connections that are eating away at your container, and likely the entire VPS node. Also, I'm pretty sure that they do retain the legal right for such procedures for the purpose of cleaning up your VPS in order to keep it from taking down the entire node. Because they can gain root access on your server, VMWare would just eat up more resources, and probably not fix the overall problem at all. It may keep them from viewing your files, but they'll still restart the container when they check top and see it at a load of 50 or something. So the next time that your 'server' goes down, ask them if they can tell you exactly wtf happened, and provide some examples so that you can show that you know enough about it to handle a mildly complex answer. For instance, ask them, "Why did you restart my server, was the load too high? Is there any way you can help me identify what was causing the server load?", or at the very least optimize PHP and MySQL in your scripts. If you don't like them logging into you VPS without permission, you really need to be upgrading to an approximately $300/month actual dedicated server. You may need to anyway, considering that load is most likely the reason that they restart your container. Regards, A Pissy Tech Support Lacky
Just stop Spamming, and they'll stop rooting you. And don't ask us how to prevent it, because they have physical access. You're hosed. Stop spamming.
Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:
1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.
2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.
You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.
The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.
If they wanted to retain access after you've changed the password, they could have easy enough.
Your MTA melting due to incoming connections is not the fault of their network. It's your box. Fix it, or get someone else to, or don't run an MTA (srsly, SENDMAIL? The 90s called, they want their line noise configuration back). If the connections never transfer any data, maybe SYN cookies would help? (is there a full TCP handshake?) Did you get a new IP when you moved?
And $35 isn't that much to pay. Surely you're paying several times that per month for the hosting, and if not, their margins are thin enough that you can't expect them to jump through whatever hoops your paranoia requires.
Pretend that something especially witty is here. Thanks.
First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.
There's basically three plans we have.
#1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.
#2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.
#3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.
So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
Since you are not disclosing the ISP name so we can examine their TOS or contracts to see who's really being the jerk here and learn enough actually help you, pay the $35/day just to recover/delete your data if you need to and find another host that suits you.
Otherwise STFU; I'm beginning to understand how your ISP feels.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
Then pay the mind-numbing fee of $35 and get on with your life dude.
I know exactly why your hosting provider needs your root password - that's because it's absolutely impossible to tell whats wrong with your server without a valid login, preferably root. If your machines aren't showing orange hardware failure lights, and you have no proof or data on a networking outage, then it's 90% sure to be an issue with the software on your machine. Since it's the most likely problem, it's unreasonable to expect your hosting provider to immediately spend a lot of time investigating the last 10%. You have two options (three actually): 1) Provide a root login 2) fix it yourself (this may require going to the datacenter in person) 3) see if they can work with an account with limited priviledges (it must be able to read logs and see all processes at the least). You also might want to try posting on serverfault - I can't comment on the technical end as you've supplied no detail Actually the support staff would probably be happiest if you fixed it yourself. In addition, have you considered that they may have brought down your machine or machines, to run memtest86+ or the like? Are you *sure* they rooted it? My only advice is to see if they'll accept a limited account (that can go through logs and see all the running processes).
Enable SELinux in your server. Then disallow root from doing anything but looking at the logs. (Also, create a new, suitably enpowered, account for running your server). Then they can have root access all they want and not be able to mess with your server.
Don't try to out wierd me, three-eyes. I get stranger things than you, free with my breakfast cereal. --Zaphod Beeblebr
"Hey landlord my heat is broken for the third time since you changed out the external heat pump unit. I think that's broken."
"Ok, can I come over today and fix it? I'll need you to leave all your safes unlocked and open, and you cannot be present while I'm there."
"Nope, never, sorry. Im giving you my notice and suing you for no heat"..
FTFY.
This "hacker" guy is actually causing a bit of a stir on the Drobo forums, accusing support left-right-and-centre of destroying his data. Only a couple of days before he started screaming bloody murder he was posting questions about "tuning" his filesystems with tune2fs.
Shame the Drobo forums are for customers only, but he's a bit of a tit. I wouldn't believe a single word he says about the ISP.
Based on your previous replies, you have said that the server is not yours.
It really doesn't matter WHO the server belongs to after that, it simply is not yours.
Whether they rent it, re-sell it or whatever, it still is not yours.
Honestly, I don't care how "clued" you are, they are not wrong in asking for the root password to diagnose a problem which you claim is happening with their hardware. ( I say "their" since they are in a contract with someone else over this server and you are not in that contract). If you feel they are that inept, you should have kept detailed notes and asked to speak to management to voice concern about their previous ineptness to see if a more senior technician can work on the issue.
Keep in mind that a good business would at least want to try to see if there is a problem with the machine in question so that they can they replace it with those they are renting from. At the company where I previously worked, they rented their machines for a period of time and that worked out better than buying new machines every few years. If anything went seriously wrong during that period, it was a matter of shipping the machine back and getting a new one at the same rental fee.
Now, as to them locking you out and all that, I'd have to see YOUR contract with them to know what is right and wrong regardless of how inept you think they are.
If your contract allows this behavior, then you really have no room to complain.
If they hardware was determined to be the issue, who knows if they had the exact hardware to stuck you back on (since they rent the hardware). Its not clear and I honestly do not feel like reading through more replies here.
It sounds like you made things harder on yourself than needed. But you chose to pay the 35/day KVM switch and fixed things yourself (good for you). BUT, that was YOUR CHOICE in not giving up your password.
I also question WHY they would try to hide their tracks on rooting your box as they did. If its in their contract, so what? Hiding it makes it suspicious.
At any rate, the short version is what I said in the title.
You need to get a machine and colo it. Get the necessary equipment as has been previously stated and at that point, you have legal recourse. As it stands, I don't know what re-course you have as that depends on your contract with them.
Example: As a company I worked for, NOT providing the requested information and/or logs was reason enough to close an open trouble ticket. We normally gave our best effort since some situations existed where people genuinely could not do so (security clearances, etc). Once we hit a point where that info was non-optional, they customer had a choice to make and that was do what they had to in order to get the logs or close the ticket.
Now is the time for you to continue to make your choices.
* Abide by their rules and fess up the password (pursuing through management as needed)
* pay KVM charges as needed to avoid giving the password
* change providers that might more suit your whims (good luck on that)
* COLO