Slashdot Mirror
Preventing My Hosting Provider From Rooting My Server?
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
.. just switch providers. I'm sure there are companies that treat you better.
As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.
If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.
As others have pointed out
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:
1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.
2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.
You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.
The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.
First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.
There's basically three plans we have.
#1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.
#2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.
#3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.
So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
Since you are not disclosing the ISP name so we can examine their TOS or contracts to see who's really being the jerk here and learn enough actually help you, pay the $35/day just to recover/delete your data if you need to and find another host that suits you.
Otherwise STFU; I'm beginning to understand how your ISP feels.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
Then pay the mind-numbing fee of $35 and get on with your life dude.