Slashdot Mirror
Preventing My Hosting Provider From Rooting My Server?
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
Second this. Isn't it an adage that someone who has access to the hardware has already won? Secure some solid evidence and publicize it on your way off the host.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Depending on where the center is located, and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.
I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.
I don't have too much experience in this arena but once I was running a few units and got a rack mounted sun box to play with. Thing didn't have video IIRC and it was all done via suns various terminal connections. Once I got the box set up on the rack (in a room I didnt have normal access to), I ran the terminal cable to a linux webserver that I ran on the same rack.
One day, the sun stopped responding over its ethernet connection I thought I was screwed until I remembered that cable...sshed into the other box, brought up the terminal cable and I was soon at sun's management console that let me figure out what was going on.
I would assume any reasonable host would be willing to get you a similar sort of hookup.
Bottles.
Are you co-locating a machine you own outright, or do you have a "dedicated hosting" package with the company? I was a system admin at a web hosting company for a long while, and on our dedicated packages if a customer took root access they had to inform us if they changed the root password. We also kept root ssh keys to all of the servers just in case someone wanted to try and be a dick about it. The logic is the machine is actually our property and the customer is renting its use, just as most apartment complexes will keep master keys to the units.
However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.
My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.
If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.
93rd rule of Slashdot: No matter how obvious my sarcasm is, my comment will be taken seriously by someone.
Bogons, UK
GetNetworks/JavaServletHosting, US
WebVisions, AsiaPac (currently India and Australia)
Rgds
Damon
http://m.earth.org.uk/
The logs should tell you why the machine crashed.
How busy was the server?
There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).
They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.
I host with Softlayer.net (dedicated boxes) and I had the same mysterious issues, server going offline and coming back on. I have a different approach. I trust the techs of the company I'm hosting with so I don't mind giving up root access to chase this problem down. What I do after that is change the root pass again and I'm done. What I'm finding is when the OS and logs come back clean, the problem is mostly likely tied to a DC router issue (a bug or misconfiguration). That's exactly what the excellent techs at SL found. They even filed an RFO (reason for outage) report several days later explaining the problem in detail. So, just like everyone here says, get with a good hosting company and put some trust in the support staff. I used to think that all these companies were about the same level of service if your on a dedicated but, I soon found out you really do get what you pay for.
I used to lease a dedicated box, and over the years, I was faced with this decision to switch to another provider on 4 separate occasions. A similar situation, they weren't always asking for the root password, but in each instance, there were hardware problems crashing the box, and they would play ring around the rosies fixing it, and my family's business was losing business and credibility. I understand the problem, for $200/mo. for a dedicated box, a company can't afford to have a gaggle of techs so they can provide 4 hour response time, and have hot spare boxes ready to roll into place.
We decided we could no longer employ "hosting provider roulette" as part of a reasonable business plan.
I found a data center not exactly close to home but within a reasonable distance, near Downtown L.A., that had a reasonable colocation rate. We put together a 1U box, and put it in the rack. For $125/mo (~$40/mo. less than we were paying for an inferior dedicated box) our down time has all but disappeared. The thing is, whenever the down time was because of the hardware, I was able to drive down there and swap stuff around, including swapping in a tower for a time while I had to send our server out for repair. Our down time profile changed from multi-week periods of unreliable service to brief windows of usually less than an hour though one time about 4 hours while I had to drive around town rounding up some new drives once.
Another thing we got out of this move was the ability to configure our box as we pleased. We upgraded out box to an 8 core box with 24GB of RAM and a 1.3TB RAID 10 array. Leasing a box like that is cost prohibitive. And the time to do this was minimal, I just ordered the parts from Newegg, built it, burned it in, and went down to perform the swap. They didn't quibble about me having two machines hooked up for a day while I made the swap.
The "company" that runs the data center is actually a few companies sharing a space, and they help each other out covering tech support at night. They are all 100% top-notch geeks, who understand the problems a web admin faces, and they are very accommodating. They will put an IP KVM on the box or even wheel up a head, plug it in, and tell you what the screen is saying, even help diagnose, all for no additional charge. You can hire them to be a monkey by the hour, if needed, or just go there 24x7x365 on a moment's notice, to access the data center, which is secured, has halon, backup chillers, redundant power and backbone feeds, UPS, diesel generator, etc. all the amenities. I get nothing from them except goodwill for my recommendation. I can tell you I have never once in the 6 years I have colocated a box with time, have I ever considered moving. For anything. Not even the cloud could beckon me away. If anyone is interested: http://colocation.la/ also http://serverlogistics.com/ if you are interested in shared or dedicated hosting.
cat
I definitely agree. The local staff at my colos are happy to do simple tasks while acting as my eyes and performing keyboard instructions on my behalf (if it's critical) or even simply exchanging a dvdr in a backup burner, otherwise they need to (and would) stay away. But those are my boxes in a rack and any network outages could be confirmed by the datacenter's logging and equipment.
I get the impression that OP doesn't have his own equipment in a rented rack, otherwise hardware would be solely on OP's shoulders. If you are using their equipment, I don't feel that it's unreasonable to ask you for logs to diagnose, however they should have gone about it legitimately with you sharing it to them.
Screw this paranoia about encryption, The Man isn't gonna come after your FOSS site and it just adds additional complexity that needs to be troubleshooted when things go south. If your sites are so heavily trafficked, buy your own box to eliminate one of the things you are blaming on the provider and move over to a provider who will not fuck with your box on a whim and respects you.
If your hosting provider wants the log files, they don't need root, just a copy of the files. Give them a user-level login, and put a copy of the files where that user can see them.
The outage already happened, right? They don't need the current logs as they happen, just the logs for the outage period.
"with their freedom lost all virtue lose" - Milton
I _DO_ work at a hosting provider, and unfortunately root access is often required to repair the steaming piles of crap customers often leave behind.
That may be a symptom of the type of customer we attract, but I don't think this is unusual. The submitter is an exception, most people who get them have no business operating a server.
For the submitter: get an internet KVM and use LUKS to encrypt. You'll need the KVM to remotely type your passphrase. They can still get at it if they really wanted to - but you aren't going to be worth the effort.
Hell if you are where I think you are, you better check your boot scripts, I think you'll find openvt opening a terminal where you may not expect.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...