Slashdot Mirror
Preventing My Hosting Provider From Rooting My Server?
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
.. just switch providers. I'm sure there are companies that treat you better.
Be sure to stun them as soon as they start casting it.
1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.
2. Figure out how they broke in. If they broke in then someone else likely could too.
I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.
Depending on where the center is located, and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.
I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.
On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.
They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.
You also have zero chance with litigation, unless you've somehow gotten them to sign something saying they specifically won't muck around in your server.
I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?
I don't have too much experience in this arena but once I was running a few units and got a rack mounted sun box to play with. Thing didn't have video IIRC and it was all done via suns various terminal connections. Once I got the box set up on the rack (in a room I didnt have normal access to), I ran the terminal cable to a linux webserver that I ran on the same rack.
One day, the sun stopped responding over its ethernet connection I thought I was screwed until I remembered that cable...sshed into the other box, brought up the terminal cable and I was soon at sun's management console that let me figure out what was going on.
I would assume any reasonable host would be willing to get you a similar sort of hookup.
Bottles.
Are you co-locating a machine you own outright, or do you have a "dedicated hosting" package with the company? I was a system admin at a web hosting company for a long while, and on our dedicated packages if a customer took root access they had to inform us if they changed the root password. We also kept root ssh keys to all of the servers just in case someone wanted to try and be a dick about it. The logic is the machine is actually our property and the customer is renting its use, just as most apartment complexes will keep master keys to the units.
However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.
My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.
If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.
93rd rule of Slashdot: No matter how obvious my sarcasm is, my comment will be taken seriously by someone.
"How do I turn a whore into a housewife?"
Some things are only solved by replacment.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
The logs should tell you why the machine crashed.
How busy was the server?
There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).
They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.
Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:
1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.
2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.
You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.
The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.
I host with Softlayer.net (dedicated boxes) and I had the same mysterious issues, server going offline and coming back on. I have a different approach. I trust the techs of the company I'm hosting with so I don't mind giving up root access to chase this problem down. What I do after that is change the root pass again and I'm done. What I'm finding is when the OS and logs come back clean, the problem is mostly likely tied to a DC router issue (a bug or misconfiguration). That's exactly what the excellent techs at SL found. They even filed an RFO (reason for outage) report several days later explaining the problem in detail. So, just like everyone here says, get with a good hosting company and put some trust in the support staff. I used to think that all these companies were about the same level of service if your on a dedicated but, I soon found out you really do get what you pay for.
First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.
There's basically three plans we have.
#1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.
#2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.
#3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.
So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go