Preventing My Hosting Provider From Rooting My Server?

hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?" Read on for a few more details of hacker's situation. "With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.

What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."

Comments

Re:If they do this..

[Slashdot Parent]

HA HA HA EC2 uptime SLA HA HA HA

I'm pretty sure that an EC2 instance can just go away at any time... it's not supposed to be a permanent VM that lasts for ever.

Any single machine can "just go away at any time". How does that make EC2 any different?

What EC2 does offer you is the ability to relaunch your instance (in another data center, if desired, in case yours went offline due to natural disaster, fire, or something). If you use EC2 correctly, you should have some pretty robust uptime, for the price.

Re:If they do this..

[Slashdot Parent]

Not even the cloud could beckon me away.

When one of my virtual instances goes down, a replacement instance is launched automatically. If I didn't check my email, I'd never even know it happened.

But don't let the cloud beckon you away. Not when there's LA traffic to drive through... ;)

Re:If they do this..

[Slashdot Parent]

Relaunch ... with a new disk loaded from S3. So relaunch from a backup that you had to make yourself.

It was definitely like that a few years ago, but now things are totally different.

Read up on EBS. It's persistent storage that you can attach to an EC2 instance. As of a few months ago, you can even have EBS-backed images. Worth a read, even if you don't plan to ever use it.

My reading of the EC2 terms suggested that they could make your VM go away for maintenance reasons, whereas "normal" VPS hosting will try to keep your data around etc.

In practice, you'll usually get an email a few days to a week before your instance gets terminated. And with EBS, you should be able to relaunch from where you left off. Not from the point of your last backup.

Re:If they do this..

[Slashdot Parent]

I knew about EBS, didn't realise they did EBS backed images. It's good to know they warn you that your VM is going away (I didn't know that; is it advertised at all?).

It isn't advertised because they can't guarantee you'll get a warning. Basically, if they can run a host machine in a degraded state and send out an email warning, they will do that. However, if a machine simply fails, they can't exactly send out a warning. :)

If you run an application on EC2, you still need to architect your application so that an individual instance can terminate with no notice.

But isn't that always the case? I mean, my physical machines don't usually give me any warning before a crash. What do you do when one of your physical servers fails?

But still, if I want a stable box I'd rather get one that stayed up without me having to check my email to make sure it wasn't about to get shut down.

I'm not sure I know what these "stable boxes" are. Any server can fail. And what's your plan when (not if) one of those "stable boxes" fails?

And there are lots of VPS providers who'll sell you one of those for less than what EC2 charge.

Comparing EC2 prices to another VPS provider doesn't make sense, because EC2 is not a VPS provider. That's like saying zipcar is ripping you off when they charge $7/hr for a car because it'd be cheaper to lease the car for 3 years from the dealership.

Re:If they do this..

[Slashdot Parent]

The original article was someone asking about problems with their dedicated server. What I guess I'm getting at is that EC2 isn't a dedicated server / VPS provider, and therefore not suitable for their use case. Which you agree with :-)

Well, now I'm beginning to wonder if EC2 might fit their use case. The OP was asking for a provider that wouldn't root his instances, and as Amazon points out in their forums all the time, they won't access customer instances for any reason short of a search warrant. So if you want them to root your box. If you give them a key. They still won't touch your instance.

As far as pricing goes, if you know that you are basically running a server 24/7, you'd use their newish "reserved instances". Now I don't know his server specs, but let's just assume an EC2 "large" instance, which is probably pretty good for his workload of 300 websites, some code hosting, and a few other things.

If he were to purchase a large reserved instance and run it 24/7, it would cost him $1400 + $0.12/hr or $1400+3153.6=4553.6 for 3 years, or $126.48/mo+bandwidth.

Now what would a similarly-speced dedicated server run him from a reputable provider? Basically, a Quad Xeon with 7.5GB of RAM and 850GB local storage. I looked at some of the name-brand folks, and was coming up with some really, really high quotes.

Yes, I know servers will fail. But as long as the disks still spin and haven't had crap written over them, then random VPS/dedicated server provider will be able to resurrect your box without you having to lift a finger. Which is a lot less work for you to do. I still keep backups myself :-)

Well, look at the time The Planet caught fire vs. the time that EC2 got struck by lightning. I forget how long The Planet customers were down for, but if you were an EC2 customer, you could have just relaunched and you'd have been fine. People who really engineered their apps correctly wouldn't have even noticed the lightning strike.

Provided you're using EBS correctly, as you pointed out before. :)