Slashdot Mirror
Preventing My Hosting Provider From Rooting My Server?
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
.. just switch providers. I'm sure there are companies that treat you better.
Be sure to stun them as soon as they start casting it.
chmod 744 /var/log (modify the directory name as needed so that it points to where your logs reside)
and they will be able to look at your logs without root password. If this is not enough for them, remember that internet is full of service provider that are eager to host you for the same money (if not less)...
1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.
2. Figure out how they broke in. If they broke in then someone else likely could too.
I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.
look for a pre-authorized ssh key in ~/.ssh/authorized_keys or something similar, remove it.
Depending on where the center is located, and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.
I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.
On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.
They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.
You also have zero chance with litigation, unless you've somehow gotten them to sign something saying they specifically won't muck around in your server.
I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?
I don't have too much experience in this arena but once I was running a few units and got a rack mounted sun box to play with. Thing didn't have video IIRC and it was all done via suns various terminal connections. Once I got the box set up on the rack (in a room I didnt have normal access to), I ran the terminal cable to a linux webserver that I ran on the same rack.
One day, the sun stopped responding over its ethernet connection I thought I was screwed until I remembered that cable...sshed into the other box, brought up the terminal cable and I was soon at sun's management console that let me figure out what was going on.
I would assume any reasonable host would be willing to get you a similar sort of hookup.
Bottles.
Password on GRUB will not protect against physical access to a machine. Maybe the best thing you can do is to encrypt the disks. And for now on try to get servers with Drac http://en.wikipedia.org/wiki/Dell_DRAC or something similar installed. Through Drac's remote console you can remotely access the computer during boot process as if you were sitting at the local console.
If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.
As others have pointed out
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
Are you co-locating a machine you own outright, or do you have a "dedicated hosting" package with the company? I was a system admin at a web hosting company for a long while, and on our dedicated packages if a customer took root access they had to inform us if they changed the root password. We also kept root ssh keys to all of the servers just in case someone wanted to try and be a dick about it. The logic is the machine is actually our property and the customer is renting its use, just as most apartment complexes will keep master keys to the units.
However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.
My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.
If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.
93rd rule of Slashdot: No matter how obvious my sarcasm is, my comment will be taken seriously by someone.
"How do I turn a whore into a housewife?"
Some things are only solved by replacment.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
The logs should tell you why the machine crashed.
How busy was the server?
There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).
Hello, I work for a very similar company that provides support. How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access? That would surprise me greatly because no one in this field would care enough to go through the trouble of a sophisticated hack of your server, and besides, if they could do it, so can anyone else. Because of the hazy situation here, I'm going to assume that you are running this "server" as a VPS as opposed to a dedicated server plan. If that's the case, then they can easily log into your root account because your server is already run under VMWare. Chances are they asked you for your password in order to bypass looking up the vzid of your container. After that, it's typical procedure to restart the container if you're eating up massive resources. That will usually clear out the http/svn/mysql connections that are eating away at your container, and likely the entire VPS node. Also, I'm pretty sure that they do retain the legal right for such procedures for the purpose of cleaning up your VPS in order to keep it from taking down the entire node. Because they can gain root access on your server, VMWare would just eat up more resources, and probably not fix the overall problem at all. It may keep them from viewing your files, but they'll still restart the container when they check top and see it at a load of 50 or something. So the next time that your 'server' goes down, ask them if they can tell you exactly wtf happened, and provide some examples so that you can show that you know enough about it to handle a mildly complex answer. For instance, ask them, "Why did you restart my server, was the load too high? Is there any way you can help me identify what was causing the server load?", or at the very least optimize PHP and MySQL in your scripts. If you don't like them logging into you VPS without permission, you really need to be upgrading to an approximately $300/month actual dedicated server. You may need to anyway, considering that load is most likely the reason that they restart your container. Regards, A Pissy Tech Support Lacky
Just stop Spamming, and they'll stop rooting you. And don't ask us how to prevent it, because they have physical access. You're hosed. Stop spamming.
Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:
1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.
2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.
You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.
The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.
I host with Softlayer.net (dedicated boxes) and I had the same mysterious issues, server going offline and coming back on. I have a different approach. I trust the techs of the company I'm hosting with so I don't mind giving up root access to chase this problem down. What I do after that is change the root pass again and I'm done. What I'm finding is when the OS and logs come back clean, the problem is mostly likely tied to a DC router issue (a bug or misconfiguration). That's exactly what the excellent techs at SL found. They even filed an RFO (reason for outage) report several days later explaining the problem in detail. So, just like everyone here says, get with a good hosting company and put some trust in the support staff. I used to think that all these companies were about the same level of service if your on a dedicated but, I soon found out you really do get what you pay for.
First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.
There's basically three plans we have.
#1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.
#2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.
#3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.
So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
Since you are not disclosing the ISP name so we can examine their TOS or contracts to see who's really being the jerk here and learn enough actually help you, pay the $35/day just to recover/delete your data if you need to and find another host that suits you.
Otherwise STFU; I'm beginning to understand how your ISP feels.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
It's obvioius David's provider for gnu-designs.com is Layered Tech. In my opinion he'd be WAY better off going to another provider; Layered Tech hosts spammers, malware purveyors and all sorts of net scum. We have LT firewalled for quite a while now. In the past they never respond to abuse complaints so we got tired of their crap and just completely blocked them. Move on to someone else, even AT&T would be preferable to LayeredTech.
Buddy of mine had a box at ovh and he found ssh keys stored in the "/root/.ssh" which can be setup to allow log in without need of the password, he found stored ssh keys in there from them and log's showing someone from the datacenter going in there and poking around. you should check in there to see if there are keys in there and delete them and change all your passwords.
I know exactly why your hosting provider needs your root password - that's because it's absolutely impossible to tell whats wrong with your server without a valid login, preferably root. If your machines aren't showing orange hardware failure lights, and you have no proof or data on a networking outage, then it's 90% sure to be an issue with the software on your machine. Since it's the most likely problem, it's unreasonable to expect your hosting provider to immediately spend a lot of time investigating the last 10%. You have two options (three actually): 1) Provide a root login 2) fix it yourself (this may require going to the datacenter in person) 3) see if they can work with an account with limited priviledges (it must be able to read logs and see all processes at the least). You also might want to try posting on serverfault - I can't comment on the technical end as you've supplied no detail Actually the support staff would probably be happiest if you fixed it yourself. In addition, have you considered that they may have brought down your machine or machines, to run memtest86+ or the like? Are you *sure* they rooted it? My only advice is to see if they'll accept a limited account (that can go through logs and see all the running processes).
Enable SELinux in your server. Then disallow root from doing anything but looking at the logs. (Also, create a new, suitably enpowered, account for running your server). Then they can have root access all they want and not be able to mess with your server.
Don't try to out wierd me, three-eyes. I get stranger things than you, free with my breakfast cereal. --Zaphod Beeblebr
The fact it's a Celeron isn't the issue - the rest of the machine is substandard, commodity parts, shoved in consumer cases and crammed onto a breadrack. I knew before I worked at The Planet that this wasn't industry standard, and it's still not - the standard is to use full size server racks with 1U or greater servers, 1U switches, 1U networked power supplies (instead of a serial port hack that flips the power jumper on the motherboard - which, albeit a cool hack, is a Bad Idea), and hot & cold aisles. I'm not talking about zip tying cables in place - I'm talking about zip tying a 24 port switch and a series of $7 Wal-Mart power strips to the underside of a bread rack so you can literally fit as much CPU per square foot as possible - reliability be damned.
Either way, the relevance to the conversation was that we were told to root a customer's box if they had a hardware complaint and wouldn't give us the root password to make sure it wasn't the software, which resulted in quite a few customers getting emails from Frank Castle and forfeiting their fees and server lease. It's just bad business, in my opinion, and it's why I left The Planet after 6 months.
Truck driver, plumber, Linux systems engineer.