Slashdot Mirror
Preventing My Hosting Provider From Rooting My Server?
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
.. just switch providers. I'm sure there are companies that treat you better.
It's not possible given the constraints. While you can make some attempts at doing this sort of thing, at the end of the day, physical access is root access.
You can protect the data with dm-crypt, and just hide most of your system behind the encrypted partitions. But even then, you can't stop them from screwing up your server, rebooting it, messing around, etc.
Be sure to stun them as soon as they start casting it.
I'd be curious which company this is.
I had a bad experience with SoftLayer in which they screwed up the hardware and cost me $20K at the end of the day and they wouldn't cooperate on getting everything resolved.
XEN FTMFW.
http://www.howtoforge.com/creating-a-fully-encrypted-para-virtualized-xen-guest-system-using-debian-lenny
Sounds like this is in your hosting contract. Either switch, or if your that concerned, host it yourself, not in a data center. Every data center is going to say "Prove it" if you try to pin an issue on them.
Just send them a copy of your logs. It's crazy that they ask and root your server but if it helps until you can get out of the relationship send them the logs.
Alternatively give them access to the logs via splunk.
You need a Linux BIOS with minimal networking/ssh and password protection and a linux install that has no options for booting to any sort of prompt without a password. The BIOS-level stuff is required to prevent anyone with console access from easily booting an alternate kernel/environment. This still would NOT prevent someone from yanking out the drives, plugging them into another system, mounting the file systems, and "rooting" your system that way.
chmod 744 /var/log (modify the directory name as needed so that it points to where your logs reside)
and they will be able to look at your logs without root password. If this is not enough for them, remember that internet is full of service provider that are eager to host you for the same money (if not less)...
1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.
2. Figure out how they broke in. If they broke in then someone else likely could too.
I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.
The only reason you wouldn't be able to remotely enter in a boot-time decryption password, is if you don't have any remote management capabilities on this server. If this is the case, you should get just better hardware.
Causation can cause correlation
Host it yourself?
look for a pre-authorized ssh key in ~/.ssh/authorized_keys or something similar, remove it.
Or a remote access card. or the IBM machines, they are called RSA cards, on the dell machines they are called a DRAC. There is an equivalent for HP called iLO, and every other large brand. I also know that supermicro sells them for some of their server boards too. I think these may be generic: http://www.ami.com/serviceprocessors/
In order to achieve this it would require the server to process encrypted data - without needing to know what that data is or even why it's doing it.
You send it encrypted data - it process it without decrypting it and returns the still encrypted result. Thus preserving your security.
As I understand it (from a slashdot article I now cannot find) IBM are working on this but there is as yet no solution.
Depending on where the center is located, and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.
I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.
Change ISPs. My colo company specifically states in our contract that they will not touch my server (physically or remotely) without my prior consent. Once they had to rearrange the rack to fit a new server in, and they called me to ask if I wanted to be present or to move mine myself.
On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.
They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
You could always not encrypt your drive, but keep GRUB from doing anything but booting regularly and keep the BIOS from doing anything but booting from hard drive (without a password). This will still allow them to root your box, but they'll have to grab your hard drives first.
As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.
You could create a number of LVM partitions and encrypt them, then mount them once the machine boots, but I'm not sure if that will fully prevent them from rooting your machine.
You also have zero chance with litigation, unless you've somehow gotten them to sign something saying they specifically won't muck around in your server.
I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?
You call yourself "hacker" but you don't already know how to do this.
Run ESXi as the host OS and give all available resources to the VM. It's nice, because when the VM crashes, I can still see what's on the console, rather than calling the NOC monkeys and trying to decipher what they're telling me is on the console.
and stick a BSD firewall in front of it. Good grief. The main reason servers get compromised is because the server has been left vulnerable. If you can run a server then you can harden it. Do a search on "server hardening rules linux|redhat|win|mac|etc" and you should find some good stuff.
http://www.freebsd.org/doc/en/books/handbook/firewalls.html
If you don't feel comfortable with this then talk to a techie friend or hire a computer science/engineering student to do it for you. ....and change providers.
gott im himmel
Password on GRUB will not protect against physical access to a machine. Maybe the best thing you can do is to encrypt the disks. And for now on try to get servers with Drac http://en.wikipedia.org/wiki/Dell_DRAC or something similar installed. Through Drac's remote console you can remotely access the computer during boot process as if you were sitting at the local console.
Among the many choices you have, you can install a remote monitoring/administration card.
But that's really using technology to solve the wrong problem. The problem is your ISP.
Fire your ISP. You already have two very good reasons for doing so. First, they
should simply ask for the logs, not demand entry into the system. Second, for taking
down your server, breaking into it (what if you had data on there you didn't want
unauthorized people to see?) without your express, positive, verified consent.
Using technology to solve a problem is a fine thing. However, the problem you are
reporting isn't technical.
Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves.
Get a hosting company that allows you access to a remote access card. Dell servers use a 'DRAC' Dell Remote access card. IBM and stuff have their own.
"...They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs.."
What does the word "rooted" mean in this case? They did not have the root password so what is the posted referring to here?
Disclaimer: I am no computer geek.
This is SOP at The Planet - which hosts on the cheapest commodity hardware they can hack together. MiniATX with Celeron procs, all stacked together on a bread rack. The switches are zip-tied to the racks, as are the power strips.
My NDA has long since expired - I'm open to answering questions via email if anyone has them.
Truck driver, plumber, Linux systems engineer.
If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.
As others have pointed out
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
Sue them. And switch to a different company.
You see? You see? Your stupid minds! Stupid! Stupid!
Are you co-locating a machine you own outright, or do you have a "dedicated hosting" package with the company? I was a system admin at a web hosting company for a long while, and on our dedicated packages if a customer took root access they had to inform us if they changed the root password. We also kept root ssh keys to all of the servers just in case someone wanted to try and be a dick about it. The logic is the machine is actually our property and the customer is renting its use, just as most apartment complexes will keep master keys to the units.
However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.
Why did you not just give them access to the logfiles. Just setup a new apache on port 8080, do a few symlinks to bring the log files into the default html folder, update config to follow symlinks, add a .htaccess file and you are done. Should not take more then 20 minutes.
How exactly did you expect them to help you, if you are the only customer with problems, and you don't give them any access to your log files.
And it's time to change hosting provider if they really did the "bringing it down and poking around through my logs" part, because there is no reason to bring the server down to look at the log files. They could just copy them.
My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.
If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.
93rd rule of Slashdot: No matter how obvious my sarcasm is, my comment will be taken seriously by someone.
First: switch providers. Do not put up with this behaviour.
The only thing you can do otherwise it use encrypted filesystems for your data (you don't need to encrypt *everything* including the root filesystem, just main data store(s) like /home & any databases & sensitive logs stored elsewhere and temporary storage areas) without storing any trace of the keys on the server or anywhere else accessible by the server. Have the server request (or otherwise wait for) the keys to be provided by you before it will mount the protected filesystems.
The major problem with this arrangement is of course the fact that if the machine does down unexpectedly overnight (power+USP failure, other hardware issues, service provider interference, ...) you will either need to be disturbed so you can provide the keys or your services will be offline until you get up and notice the pending key request.
This won't stop them trying to root the machine by rebooting it and accessing the discs from cd-booted linux setup, but it will stop them succeeding unless that can convince you that an outage is a "normal" freak occurance and the server is requesting decryption keys as expected, rather than them hoping you'll provide the keys to their setup so it can ready the encrypted volumes.
But still: move provider. Really. Implement the above (and/or other protections) at your new provider for the sake of paranoia by all means, but definitely don't hang around.
"How do I turn a whore into a housewife?"
Some things are only solved by replacment.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
The logs should tell you why the machine crashed.
How busy was the server?
There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).
Simple:
1. Require all passwords to be Yubikey OTP passwords on any login prompt.
2. Refuse access, and only give them the logs manually.
3. When they shut your server down and open it up to yank the drive, hit 'em with a breach of lawsuit.
4. ????
5. PROFIT!
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
That's all you need.
First, if you don't have physical access to the server, you always password-protect grub, encrypt important directories, and have an access card + watchdog connected.
With the passprotected GRUB they won't be able to just pass init=/bin/bash or similar to bypass your login.
The encrypted directories will keep your data safe in case the worst happens and someone boots your machine through a USB/CD/etc. If you provided your own hardware for this, or can access the server locally at any time, go there and do this:
- Disable booting from devices other than your main drive, disable legacy usb support, only enable the SATA/SCSI/IDE port you are using.
- Password protect the BIOS
- Use a glue gun to cover the Motherboard's internal battery and the Jumpers to clear CMOS ( So they can't just erase that and access your bios again )
- Put some warranty-style stickers so you know if someone opens your case.
Your access card and watchdog will let you do everything you need yourself.
Finally, get an collocation provider that doesn't suck.
Also, post that ISP's name so we never do business with them.
WTF am I doing replying to an AC at 5 A.M on a Friday night?
How about:
a) when they ask for root, change it to something innocuous, let them log in and do their stuff, then change it back.
b) change hosting providers.
c) host it yourself. Get a business class internet connection to your office/home/etc, rig up a closet with power and A/C. And if you need "five nines", get a second power provider and a UPS, and a second internet connection.
Ultimately, physical access is everything.
And yeah, why are you asking for help if you're not going to let them help you? Are you just seeking to prove that they're to blame? That seems like a waste of time given option b). Do you need to have the system locked down so that absolutely no one else can gain root access? That leaves you with option c).
Did you read the fine print? Basically, you don't "have" anything. You are using their server(s), at their location, and your access is dependent solely upon their discretion. It's always been a known fact that if a third party has access to your physical, or in this case, virtual server, they OWN you. No amount of encryption/passwords/prayer will improve your privacy. It doesn't matter what you do, since they have physical access to the servers, you'll never be secure, you'll never have exclusive access to your "server", and you'll always wonder what they are doing.
Face it, to get what you want, you're going to need to setup your own server(s), behind your own firewall(s), and get a good nights sleep. You'll still have concerns to worry about, but you'll be the one in the driver's seat.
Collect all of the evidence you have, find a decent lawyer to represent you and file suit against them.
If you have concrete proof that they are logging into your server machine without permission, that's a serious criminal offence.
http://wiki.fukt.bsnet.se/wiki/Mandos
Seems to do what you want, but requires another server somewhere on the internet in case of a reboot.
Surely you dream - any boot media will bypass a Grub password.
That goes for anything else - physical posession is everything.
Several ways: /var/logs to new NFS (may need some scripts to check to see if mounted if not, then change the links?)
- Set up a NFS mount, redirect
- logd can write to remote servers... (not all processes use logd so this may/may not be good solution)
Then if the ISP insists, give access to that... or a copy of that...
Or just send ISP copies for requested period in question... (if not over https, then they probably could monitor everything on there side anyway)
I run several systems using vmware or one of the alternatives; all the guest OSes run encrypted partitions (requiring password to boot). Generally, it makes it difficult even with physical access to gain access to the files. Similarly, I prefer Truecrypt on Windows (* cringe *) machines. In either case, all the machine's issue/login screen clearly state access to the machines are restricted to the rightful owner and any other access is a violation. I've never had a problem with any colo facility before; while I've been asked on various occasions for the root password, I've provided the vm server's root password once (when a hardware / power supply failure occurred). Obviously, the password was changed as soon as the boot occurred and I compared md5 hashes of all files on disk (for trojans). Whenever outages occurred, I'd bz2 a few log files and send them over; I've received valid responses such as this failed or that failed or "we pulled the wrong cord". However, I've also received lies such as "our accountant didn't send uunet their check this month" but then they've come clean. However, never did they bring down a working machine to mess with it. My advise, regardless of what you choose for file encryption, I'd suggest you immediately leave that 4th rate provider.
Get a dedicated server running the latest centos or ubuntu server release. Use Xen to run your various applications in dedicated virtual machines. You can encrypt entire domains in a number of ways both internal and external. A dedicated test domain can be set up for your hosting provider to access, etc.
Oh shit! I forgot to click "Post Anonymously"...
Hello, I work for a very similar company that provides support. How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access? That would surprise me greatly because no one in this field would care enough to go through the trouble of a sophisticated hack of your server, and besides, if they could do it, so can anyone else. Because of the hazy situation here, I'm going to assume that you are running this "server" as a VPS as opposed to a dedicated server plan. If that's the case, then they can easily log into your root account because your server is already run under VMWare. Chances are they asked you for your password in order to bypass looking up the vzid of your container. After that, it's typical procedure to restart the container if you're eating up massive resources. That will usually clear out the http/svn/mysql connections that are eating away at your container, and likely the entire VPS node. Also, I'm pretty sure that they do retain the legal right for such procedures for the purpose of cleaning up your VPS in order to keep it from taking down the entire node. Because they can gain root access on your server, VMWare would just eat up more resources, and probably not fix the overall problem at all. It may keep them from viewing your files, but they'll still restart the container when they check top and see it at a load of 50 or something. So the next time that your 'server' goes down, ask them if they can tell you exactly wtf happened, and provide some examples so that you can show that you know enough about it to handle a mildly complex answer. For instance, ask them, "Why did you restart my server, was the load too high? Is there any way you can help me identify what was causing the server load?", or at the very least optimize PHP and MySQL in your scripts. If you don't like them logging into you VPS without permission, you really need to be upgrading to an approximately $300/month actual dedicated server. You may need to anyway, considering that load is most likely the reason that they restart your container. Regards, A Pissy Tech Support Lacky
Just stop Spamming, and they'll stop rooting you. And don't ask us how to prevent it, because they have physical access. You're hosed. Stop spamming.
It sounds like you have a "Managed Server" type of plan with your hosting provider. With a managed plan, a provider has some legal obligations (despite customer instructions) to maintain the host. Go find an "Unmanaged" hosting provider, or colo your own equipment.
What part of password protect the bios, put warranty stickers on the case, disable booting from other media, hotglue the internal battery and clear cmos jumpers and use ENCRYPTED partitions you didn't understand?
WTF am I doing replying to an AC at 5 A.M on a Friday night?
Don't ever go by web sites that rank the top 10 providers. Those are all paid placements.
Sometimes good providers turn bad. Forums provide the most up to date info.
I've found this site to provide useful info: http://www.webhostingtalk.com/
Just go with the opinions of those with lots of posts that don't appear to be promoting a single agenda.
Perhaps you were rooted by someone else and your service provider is trying to help you
figure this out.
If there was an outage on their side, it might be quite obvious. Maybe you need to ask more
questions and work with them to solve your problem.
---
Happy Festivus
Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:
1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.
2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.
You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.
The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.
That can all be undone.
What part of 'physical posession is everything' do you not understand?
There are two ways that I can think of to prevent this. 1) Take a bike lock and wrap it around your server, chaining it closed and to everything nearby. They won't be able to root it easily, because the drive is locked in. 2) Go to the datacenter and pick up your server. Hide it in another companies racks, somewhere inconspicuous, and hope they don't follow any wires.
No wonder my midiclorians are going down! I can barely make sense of the world now...
We searched hard for pr0n in it in a moment of desperation, your server was taking too many hits... we only found OSS projects in it, what a waste of time!
I host with Softlayer.net (dedicated boxes) and I had the same mysterious issues, server going offline and coming back on. I have a different approach. I trust the techs of the company I'm hosting with so I don't mind giving up root access to chase this problem down. What I do after that is change the root pass again and I'm done. What I'm finding is when the OS and logs come back clean, the problem is mostly likely tied to a DC router issue (a bug or misconfiguration). That's exactly what the excellent techs at SL found. They even filed an RFO (reason for outage) report several days later explaining the problem in detail. So, just like everyone here says, get with a good hosting company and put some trust in the support staff. I used to think that all these companies were about the same level of service if your on a dedicated but, I soon found out you really do get what you pay for.
First and foremost, I agree with the suggestion that most people made... Consider changing providers. That being said, there are two very easy ways around giving someone your root password:
- Create a support account with sudo privileges
- Have your syslog re-direct the output to a remote server/logging facility
Both of those solutions are quite easy to implement. Based on your claim that you log/graph anything that comes in/out of your server, such trivial tasks should be second nature to you.
It's their equipment. Forgetting about ethics and due process and legal agreements for the moment, the very fact that it
is their equipment and they have physical access, you cannot secure your platform against them.
All your talk about encryption is moot. Learn that fact now.
Your solutions are either to
good reason to suspect this, hence have accessed your platform on multiple occasions; (option - work with them or fix your own problem)
caters to folks with these kinds of needs. That either means using a co-location provider and ponying up for rack space,
or locating it in your home/office.
The OP sounds like one of the thousands of self-important pricks I've spoken to in the 6 years I've spent in hosting. Nobody gives a shit about your server, or your projects. They're just doing their job, and faced with a "fuck you you're not getting my password" I've personally reset passwords to 50+ character passwords once I'm done.
Don't like it? Either build your own damn datacenter, or find a provider to sell you power, ping, and pipe on the 97 and manage a server you built yourself. If you own the machine, you can do whatever asinine, paranoid, double-secret encryption scheme you want.
Of course, if the machine is going down "mysteriously" and you need these "tech monkeys" to look at your logs, I highly doubt you're enough of an admin to handle coloing your own servers.
First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.
There's basically three plans we have.
#1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.
#2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.
#3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.
So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go
You cannot withhold anything from someone that has physical access to your machine. Anything they want to take, they can.
If you don't trust them with the root password, you shouldn't trust them with physical access.
http://en.wikipedia.org/wiki/Hardware_security_module
You can encrypt pretty much everything without having to manually type in the key at boot time. The module should be tied into a tamper resistant server, so, case is opened (or even light enters), keys are blown, filesystems are junked and computer is dead. (You'll be seeing this kind of stuff in cars/Apple systems soon.)
e.g.
http://en.wikipedia.org/wiki/IBM_4764
Price, $8k (not including system integration)... Oh wait, you mean your data isn't worth it?
Deleted
If you don't want to provide them with the root password, you could create a temporary user account that can sudo to root w/o the password.
You need your own hardware, remote KVM access, and to set a BIOS password. There are ways to break BIOS passwords, but your average "I'm smarter than _you_" hosting employee will not know how.
Set the boot loader password.
Set the boot order to exclude anything but what you want to permit to boot.
Set the BIOS password.
Note that if it's a virtualized server, you're pretty much out of luck. But those steps will help prevent problems with boot manipulation.
Reply that you have security and NDA agreements that prevent you from giving them root access; then shovel copies of the logs out to them.
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
imminent, theres no other way about it. Even a damn locksmith needs your permission before breaking into your place.
Be prepared to switch providers and file criminal charges against them under the Computer Fraud and Abuse act.
As for mitigating the chance of root compromise:
http://wiki.fukt.bsnet.se/wiki/Mandos will solve the local rooting problem, you will need to run an encrypted root partition
LUKS seems to be good for that...
anon
2. If they want to see the logs, it seems like a no brainer to make a logview account for that purpose. unless you dont want them to see the logs.
3. They've violated your trust, it's time to move on. Sue if they violated their contract, and you have enough proof/money.
4. If your not ok with them looking at your logs, what level of extra outage are you willing to sacrifice for that privacy?
Storm
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
Since you are not disclosing the ISP name so we can examine their TOS or contracts to see who's really being the jerk here and learn enough actually help you, pay the $35/day just to recover/delete your data if you need to and find another host that suits you.
Otherwise STFU; I'm beginning to understand how your ISP feels.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
The logic is the machine is actually our property and the customer is renting its use, just as most apartment complexes will keep master keys to the units.
While I suspect that "hacker" is using someone else's physical hardware (and is clearly not telling the full story), you don't have a right to access a machine just because you own it. Among other things, you're wrong about apartment units, and it's a really shit parallel example.
There is no reason in a non-managed hosting environment for a hosting company to require, demand, or force access the operating system of machine. It's quite simply none of your goddamn business. You provide a connection, power, security, and environment. They provide you with money. You don't have a right to be nosy.
Futhermore, maintaining root password lists and ssh root keys (which I doubt have a password, or have a commonly known password, or have a password on a list somewhere lots of people have access to) sets you up for a giant clusterfuck (namely, someone executing a script that goes and deletes a whole bunch of shit on ALL your customer's machines, or worse, starts intercepting customer's customer data.)
As to "Hacker" (jesus christ, seriously?): if you actually own the box you're running on, and just renting space- find a new hosting company. Don't bother trying to get police involved- they won't give a damn, or understand. And as to your crashing problems: let me guess, if you own the hardware: you assembled something yourself, right?
If none of their other customers in the same rack are crashing from power fluctuations, uh...maybe it's your hardware or an OS problem?
Please help metamoderate.
It's not trespassing if you give somebody written permission to be on your land. It's not trespass on a computer system if you give them permission to be on your computer system.
"He who would learn astronomy, and other recondite arts, let him go elsewhere. " -- John Calvin, commenting on Genesis 1
It's obvioius David's provider for gnu-designs.com is Layered Tech. In my opinion he'd be WAY better off going to another provider; Layered Tech hosts spammers, malware purveyors and all sorts of net scum. We have LT firewalled for quite a while now. In the past they never respond to abuse complaints so we got tired of their crap and just completely blocked them. Move on to someone else, even AT&T would be preferable to LayeredTech.
If they dont have your root account, and they wont play nice, perhaps you could take your offsite backup and walk away?
Oh Rly? You can undone RSA4096 with just physical possession?
Look, the ISP is not going to bother trying to break encryption on his machine, and it's not going to bother removing all of those protection layers. And if they do, he can find out and sue the hell out of them. This is not about the CIA trying to get his data, it is about some stupid BOFH that insists on logging into his server. What I proposed is more than enough prevention.
WTF am I doing replying to an AC at 5 A.M on a Friday night?
So the obvious answer is to get her to accept the exclusive offer.
Even if some white-hat hacker breaks into your system, its still cost time and money to verify nothing was damaged (it could even cost the time to re-install or re-image the drives). In anycase, its not free.
At the minimum, if you send them a bill and they refuse to pay it, you can then decide to pursue if further or just write it off your taxes. Having an invoice show up is one way to get someones attention.
Aren't most virtual now anyway ( jails, etc ) and you are 'rooted' by the nature of how things work? Its THEIR hardware, their datacenter remember... ( unless you have a special relationship and you bought your own box )
If you don't like the way things are done, change providers or host it yourself.
---- Booth was a patriot ----
Buddy of mine had a box at ovh and he found ssh keys stored in the "/root/.ssh" which can be setup to allow log in without need of the password, he found stored ssh keys in there from them and log's showing someone from the datacenter going in there and poking around. you should check in there to see if there are keys in there and delete them and change all your passwords.
Soooo, they rooted your server? You're pwned. You need a better setup:
(ISP lame NW) -- (HW FW, commercial) -- (OpenBSD box with nothing but a FW) -- (Your LEET NW)
Good luck rooting that!
Did you buy managed service? Let them manage your system or else find out what's causing the problem yourself and report it. If you think you are better able to manage the system than they are, examine the logs yourself when the system is up and figure out what happened. You may have to boost your logging level and install/enable some admin tools, but if they think they can determine the problem by looking at the logs, you should be able to do it also.
"The mind works quicker than you think!"
Root is not required. Create a user for them with Read-Only on the log files... And have them give you a list of the files they want to see. If something need to be done in your system, they tell you. From that point on, it's your responsibility anyway.
or is it their machine that you are hosting on?
If you are buying a hosting service from them on their hardware then I don't think it's unreasonable for them to want access to the machine as their responsibility includes the machine (and possibly the OS).
If you actually own the machine then it's a different story as their responsibility ends at the network port.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
Maybe the law is still a bit backwards, but you must agree that that’s a very fitting analogy.
I’d sue them, encrypt the hard disk (with booting a tiny system that then asks for the password via a ssh terminal connection), and move to another provider. If in any way possible all at the same time. ^^
WTF. Just seriously WTF.
You could even sue them for lost profits because of the server being down. Or for hacking into it, for “rooting” it. I bet a lawyer could come up with a package that would blast their sorry asses to the gamma quadrant and back. ^^
Any sufficiently advanced intelligence is indistinguishable from stupidity.
I could be way off base here, but is it possible to create an account that only has access to the types of information your provider wants? That way, they could access your log files without you giving them rights you don't want them to have.
I don't know if this is technically possible in the situation, but it seems like a good solution if it is.
When the axe came to the forest, the trees said, "Look out - the handle was once one of us."
your provider's behavior is not typical at all. I strongly suggest you find a new hosting provider. I've been at 3 different places and none of them behaved this way. When the network goes out or the power goes out the hosting provider has their own logs, which they would rather consult than my system's nearly useless logs.
As for an encrypted distro, OpenBSD can do what you suggested out of the box. But it is also pretty easy to just install any old distro (ubuntu or whatever) and have a minimal system on a root filesystem. and all your important stuff on a different encrypted partition which cannot be mounted until you log in and enter the key(s). generally just have a shell script that mounts them all and starts apache and other services instead of starting them automatically. (but make sure they are still stopped automatically on shutdown)
“Common sense is not so common.” — Voltaire
Got your personal files too?
Why am I the one posting this... (sighs) anyway this is all 20-20 hindsight but let it be a lesson to always follow the rules of security (Yes it's on technet, yes MS should follow their own rules). Failure to do so will result in this in this case you failed on.
Regardless, you should get your own hardware in a co-lo you've background checked and have references for. Once you've done that you should do what every respectable admin has been doing for years, turn off direct root access, Set up Public Key Authentication, and for the love of all that is secure turn off password auth for SSH. If you do that then any unauthorized access to your box is in violation of 18 U.S.C. 1030 and is punishable under the law.
Any and all content posted above may be ignored, considered irrelevant, or otherwise dismissed.
Yes, I know you've had a zillion "Set up a user with log access". I also saw the comment where you're currently locked out, so I support the suggestion that you tell your host to FOAD. But if you're in a situation like this there's nothing stopping you renaming the root account and creating a new "root" with a UID other than 0 and giving that *some* rights. You can even say with a straight face that they've got the "root" password. I tend not to bother on my home boxes (as often as not OS X these days anyway), but when I was adminning Windows I'd tend to rename the guest account to administrator and have another name for the real admin account. Similar tricks have been done on Linux boxes.
Don't know exactly what you're using your server for, but we host various clients with a top-tier hosting provider. One of the things I really like about this provider is that they will proactively help us with problems on our server. Part of what we pay them for is to alert us to problems and their indepth knowledge of the OS and sortware. They deal with the server OS and software (apache, mysql, etc.) everyday and have certified experts on staff. I consider it an advantage to host with a provider that can provide good advice and expert knowledge. I really doubt that they have a great (or any) interest in looking at our data or seeing who is browsing our sites.
Perhaps you are being a bit paranoid -- or over estimating the interest of your provider in whatever you do. Out of the couple of bucks you pay your hosting provider each month they don't have enough money to dig into your software.
If you really can't trust anyone to host your site, go with raw rack space and provide your own server.
It's not that hard of a problem to solve.
Whores sell pussy to many. Housewives to none.
Fixed that for you.
Both yum and deb based distributions have the ability to bootstrap the whole system under a mount point other than root. This is for the benefit of their installer, as you can imagine. Simply apt-get/yum install the one package you need, say apache httpd, and the package management figures out all the dependencies. After installation, you chroot to the mount point (don't forget to mount /proc and /sys there too) and run the service you want.
Instructions on how to build Amazon EC2 AMI is very similar to this, so you might find that helpful.
Of course, for the purpose of chroot, you don't need to install any new kernel. If you already know about cryptsetup and LUKS, you can then mount an encrypted disk image, install the packages, and chroot into it for the service to run.
After saying all this, I think you really should switch provider, given how unhappy you are with them. Even if you manage to get the whole Slashdot to side with you, your provider will not likely change the way they do things.
I once had a signature.
Depends on who is doing the possessing:
If a Federal LEO, intel agency, or a well funded (and clued) forensics organization has my server, all bets are off. They would have dumped the RAM, fetched the BitLocker and TrueCrypt keys, dumped the hard disks via a hardware write blocker to a .vmdk file, and would be sifting through the results with every tool known to man.
If it is an ISP, then BitLocker + TPM is more than enough to keep the data on the machine out of their hands. Especially if one uses virtual machines with the disk files in TrueCrypt partitions. This case, they wouldn't just have to bypass BitLocker + the TPM, but ninja-install a keylogger. Even with a keylogger, they would still have to figure out what keyfile(s) I use. Unless they have a RAM dumping device, the only real avenue of attack they have is remotely rooting the host OS, or attacking the client operating systems on the VM.
I've been running my own servers for over 20 years and have never looked back. How can you really trust that your machines are not being compromised by multiple third parties?
Well, Id run from than fast anyway. Also if it's not in your agreement, might see a lawyer on them going into your server anyway.
"Computers are a lot like Air Conditioners" "They both work great until you start opening Windows"
They illegally enter the users computer and cause problems? They are in as much shit as Garry Mackinnon. Report them to the FBI.
I know exactly why your hosting provider needs your root password - that's because it's absolutely impossible to tell whats wrong with your server without a valid login, preferably root. If your machines aren't showing orange hardware failure lights, and you have no proof or data on a networking outage, then it's 90% sure to be an issue with the software on your machine. Since it's the most likely problem, it's unreasonable to expect your hosting provider to immediately spend a lot of time investigating the last 10%. You have two options (three actually): 1) Provide a root login 2) fix it yourself (this may require going to the datacenter in person) 3) see if they can work with an account with limited priviledges (it must be able to read logs and see all processes at the least). You also might want to try posting on serverfault - I can't comment on the technical end as you've supplied no detail Actually the support staff would probably be happiest if you fixed it yourself. In addition, have you considered that they may have brought down your machine or machines, to run memtest86+ or the like? Are you *sure* they rooted it? My only advice is to see if they'll accept a limited account (that can go through logs and see all the running processes).
Enable SELinux in your server. Then disallow root from doing anything but looking at the logs. (Also, create a new, suitably enpowered, account for running your server). Then they can have root access all they want and not be able to mess with your server.
Don't try to out wierd me, three-eyes. I get stranger things than you, free with my breakfast cereal. --Zaphod Beeblebr
Chiming in late as usual :)
But from what i read the data center is in dallas?
I'm Curious who this is im thinking the Planet?
First of all, IANASA but i play one on tv (not really).
It seems to me both sides are doing it wrong. I understand you can only put so much information on Ask Slashdot before getting ignored by the tl;dr crowd so I might be wrong as well. Anyways, here it goes:
1) You said "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites...".
Too many eggs in one basket in my opinion. No wonder you are pissed about outages. It sounds like you have one beefy server capable of running multiple virtualized OS instances (encrypted nonetheless). So why not just cancel your dedicated service and get VPS from multiple providers? That way you would have enough redundancy (e-mail, dns, rsync, HA/Varnish, whatever), none of the headaches of hardware/virtualization and save money while at it. Managing wouldn't be harder considering your proposition of vmware/uml. Also easier to move from your server bit by bit. Start with DNS, then e-mail, then web, then... You can take your time as it won't cost you much upfront (zero?) or monthly (sub $50) and gives you the ability to scale up or down the resources to acomodate the workload and cost.
2) You said "When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system"
Do you have the option to call the provider? If not, you are doing it wrong. From my experience, over 80% of all ticket answers are canned responses. If possible, try calling (if you didn't yet) and get someone that has the power to fix the issue to talk to you. When you reach a dead end, ask to be transfered to the legal dept to discuss your contract/sla/whatever. If they don't have one, ask where you should send the legal papers. If everything fails, try the BBB.
Just my humble opinion. Excuse my poor english as it's not my first language.
Use this howto.
if you are having those issues its time to move on. Most likely you are on a large broadcast domain with hundreds of machines at worst and 50- 60 at best. most likely some fool is broadcasting your ip space and taking you down.
move on to someone else. I was a neteng who inherited a network like that and bought my time to get out. it was an unmitigated disaster.
Rootkit yourself and see what they are doing. It is easy enough and if you really want to be sure that you know what is going on without physical access to the hardware, I think that it is the only way. Read your contract first, but it should be perfectly legal.
I thought Linux was a secure server platform. And these guys are rooting it in a few minutes?
Sound to me like the poster uses a virtual shared server, ie. The hosting company provides a virtual root environment, this would be an explanation why they can "root" the server, ie. just access the file system directly from the hypervisor. I am sure this kind of service does not include uptime guarantees, it operates most likely under "Best Effort" promise. Which means they do not need to guarantee either uptime or availability. Their monitoring system and logs do not detect any error, so they want to check the posters systems logs - a reasonable request as they are trying to help him. Monitoring is a difficult task - what to you measure, where is your sensor. Was the outage on your own internet connection rather than on the server? "Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. " It sounds very suspicious, like the admin interface causes interruption in the VM. Something like this happened to my VMWare Servers totally unexplainable, but reproducably.
I've almost finished writing a HOWTO for full disk encryption on a Linode.
It'll be on their wiki in a few days time.
We provided "fully managed" hosting. Customers could take root only after agreeing to specific terms, and we mostly just left them alone. I agree that poking around someone else's shit isn't cool, and I didn't like doing it even when asked.
There's something that I don't understand here. If you don't trust the hosting company you are working with, why don't you host with them in the first place? If you believe that when they access your server, there is a risk that they will do something else than trying to solve the issue, then don't host with them. You should be able to trust your web hosting company to the point that you can give them the root on your server. Otherwise, you have made a bad choice, end of the story.
Ultimately, if you really want to keep everything private to the last point, then there's only one way. Get a server from a company that offers an integrated KVM over IP on the dedicated server, and encrypt your partitions. That way, even if they access physically the server, they wont have the passphrase needed to boot it. Of course, the drawback is that if the server reboots for whatever reason (power outage, etc.) then you will have to login in the KVM to type the passphrase, which can be annoying. Hummm... maybe you'll tell me they can try that USB boot so they can read the memory and hack access to the passphrase in the RAM after a quick reboot... But well, I suppose most host would give-up seeing that your partitions are encrypted.
But again, the best solution here is still to be with a trust worthy host were you do not fear to give the root, IMHO.
Now, you just wrote about your host trying to hide the fact that they tried to force themselves in your server. This is clearly being dishonest, and there is a big issue here. That means you CANNOT trust them to tell the truth, and proves that you have to move away. There's no way you should stay with liars.
Just my 2 cents...
I doubt that the law would agree with you. The contract you agree to when you rent a dedicated server likely covers their asses pretty well and may even have an arbitration clause in it.
It is also unlikely you can claim much damages anyhow; just how much money do you expect to make vs. the amount spent on litigation?
shouldn't have made your root password the same as your password for email and/or banking. Too bad, looks like you'll have to learn a new password to use for everything!
I worked for a large ISP that did both managed and colo hosting. The big secret is that I already have root on the boxes we manage, and I don't want root on yours. If you want to send in part of a log file, I'm happy to take a look, but I wouldn't login to your box on a bet.
When your box craps out and you loose [insert valuable service/data here], you send in the lawyers and my boss comes by to ask what happened, one of the biggest thrills of my life is being able to say "Beats me, we don't have a login for that box."
And if we're managing it, you don't have root, and your box won't crap out because we've got 500 others just like it that are working just fine and yours is nothing special.
My provider (OVH) has root access to my dedicated box via ssh.
Preconfigured linux images have ssh keyfiles (google: authorized_keys2) set up. This is also public information. This is also easy to disable.
I'ts hard to believe any provider would go through the trouble checking logs the hard way if client doesn't want to give root when asked.
It seems like hes paying a lot for managed service and is getting angry when they try to manage him. Just go do business with RapidXen or some other VPS/dedi provider that isn't a bunch of dicks.
Disclaimer: I am a RapidXen customer, and am quite happy with their service.
Patrick "Diablo-D3" McFarland || http://AdTerrasPerAspera.com
ok..lets say you rent an apartment..and the of you your apartment is leaking...and you are not home...the owner of the property can go to your apartment and fix it and leave a note just as a COURTESY..and you can't do jack about it....they can't take anything..but they can even break the door and walk in...sorry if I had to dummyfy the issue so you understand...
---
Cloud Computing Feed @ Feed Distiller
This "hacker" guy is actually causing a bit of a stir on the Drobo forums, accusing support left-right-and-centre of destroying his data. Only a couple of days before he started screaming bloody murder he was posting questions about "tuning" his filesystems with tune2fs.
Shame the Drobo forums are for customers only, but he's a bit of a tit. I wouldn't believe a single word he says about the ISP.
How much are you paying for this server you are talking about???
From what I gather you are renting a machine at their Data Center/Co-Location (or whatever buzz-word you're used to using for a space designed to house servers). Honestly I don't see an issue with them rooting their own machine to debug a problem you keep complaining about.
If you rented rack space at a Data Center and put in your own machines; then you might have something to complain about. In which case full disk encryption with a hardened kernel will keep all but the most determined people away from your data.
Based on your previous replies, you have said that the server is not yours.
It really doesn't matter WHO the server belongs to after that, it simply is not yours.
Whether they rent it, re-sell it or whatever, it still is not yours.
Honestly, I don't care how "clued" you are, they are not wrong in asking for the root password to diagnose a problem which you claim is happening with their hardware. ( I say "their" since they are in a contract with someone else over this server and you are not in that contract). If you feel they are that inept, you should have kept detailed notes and asked to speak to management to voice concern about their previous ineptness to see if a more senior technician can work on the issue.
Keep in mind that a good business would at least want to try to see if there is a problem with the machine in question so that they can they replace it with those they are renting from. At the company where I previously worked, they rented their machines for a period of time and that worked out better than buying new machines every few years. If anything went seriously wrong during that period, it was a matter of shipping the machine back and getting a new one at the same rental fee.
Now, as to them locking you out and all that, I'd have to see YOUR contract with them to know what is right and wrong regardless of how inept you think they are.
If your contract allows this behavior, then you really have no room to complain.
If they hardware was determined to be the issue, who knows if they had the exact hardware to stuck you back on (since they rent the hardware). Its not clear and I honestly do not feel like reading through more replies here.
It sounds like you made things harder on yourself than needed. But you chose to pay the 35/day KVM switch and fixed things yourself (good for you). BUT, that was YOUR CHOICE in not giving up your password.
I also question WHY they would try to hide their tracks on rooting your box as they did. If its in their contract, so what? Hiding it makes it suspicious.
At any rate, the short version is what I said in the title.
You need to get a machine and colo it. Get the necessary equipment as has been previously stated and at that point, you have legal recourse. As it stands, I don't know what re-course you have as that depends on your contract with them.
Example: As a company I worked for, NOT providing the requested information and/or logs was reason enough to close an open trouble ticket. We normally gave our best effort since some situations existed where people genuinely could not do so (security clearances, etc). Once we hit a point where that info was non-optional, they customer had a choice to make and that was do what they had to in order to get the logs or close the ticket.
Now is the time for you to continue to make your choices.
* Abide by their rules and fess up the password (pursuing through management as needed)
* pay KVM charges as needed to avoid giving the password
* change providers that might more suit your whims (good luck on that)
* COLO
If you are renting a dedicated server, the server is not actually owned by you. The provider technically has the right to do anything. Although he is tied to your SLA, meaning any downtime longer than X will automatically result in a Y credit if you ask for it. They have charts.
although you can sometime ask them to put a flag on you account saying "don't touch the server unless absolutely necessary", not all provider or agent will follow it. They will still mostly investigate if you open a few tickets complaining about non-reproducible issues. They will look into their logs, because customer's satisfaction is important, then seeing nothing, they investigate the server for any signs of hardware failure. Your failure to provide a root password led to their forced investigation. It is most likely in your contract. They do it to protect themselves ( The SLA again ).
If you have a collocation server however, different story altogether: By all means sue their ***. If it is in your contract, switch provider. No questions.
Excuse the ad hominem, but WHAT THE FUCK?
The first time this happens, you verify the backups, get another box somewhere else, migrate services, cancel the old contract and file criminal charges, in this order. Optionally, cry as loud as you can about it.
Sure, the police might not do anything about it, but you need to try, at least.
If you log and monitor as much as you claim, I really do wonder why you did not realize this yourself...
I cannot believe you have a server colocated in a 3rd party data center without true full remote access eg something like HPs integrated lights out management - ILO. That would give you complete KVM, hardware reset/power on/off, hdd/floppy/usb/cdrom redirection, with all the encryption you need not to mention hardware logging/alert features so you know if someone has physically tampered with the server.
Well worth the modest ILO license fee, the only time anyone needs to touch the server is when some hardware is actually broken.
OS fault? No problem boot from a cdrom repair disk 1000 miles away.
I work for a hosting provider and the way you reset the password is by booting from a utility CD that resets it. There are most likely other similar options that allow you to see the root password.
While I understand this guys concern, how to you troubleshoot an issue without root access to the box. Frequently customers assume that this was a network issue, when it was really something on their server that consumed all available resources that caused their server to stop responding to new requests. I've lost count of the number of people who've complained about their performance only to find that it was their code or database connections that were never closed. My suggestion is to create an account with root privileges, disable it, and only enable it when you require assistance. There is no excuse for resenting a customer's password without their consent. If a customer asks for help but doesn't provide credentials, we request them and take no action until they provide unless the server is actually down hard.
You dont need to crack someones root password to get root access to a linux machine. You simply boot it in single user mode unless you have a password on grub but hosting providers never set one on installs.
Look at Echomail v American Express and IBM (2006) out of MA, and look at whether your jurisdiction would allow for you to file a replevin action to get at the data.
http://geektechnique.org/projectlab/84/openbsd-encrypted-fileserver-howto
I stopped using hosting providers 10 years ago. There's no f*cking way, I'll ever do that again. THERE IS NO SUBSTITUTE for physical access to your own machine.
I say things which affects my Karma negatively. (and I don't care) For instance; All religion is false.
Basically no, you can't keep them out. Get a better host, who respects your privacy.
There's a downside: most diagnostics aren't possible without root, and it's quite likely that most of your small outages they really don't know about. If you seal them out of root, you're on your own up to the ethernet plug.
Some hosts will be happy to be told to GTFO the box, because it means less work for them. Be aware that if you tell a host to GTFO the box, they're not likely to stand behind you if the FBI says you have kiddie porn or warez or so on.
StoneCypher is Full of BS
So let me get this straight...
I am going to assume a lot of information here and infer quite a bit as well.
You submitted a ticket to your hosting provider and told them that you were having problems. Since you have a server that is unmanaged by said company, they wanted access to your logs to see what was going on. They asked for the root password since it has access to logs. Instead of being nice and creating an account that had access to the logs, you threw a fit make the crew that work their not willing to help due to a long rant that may or may not have been made.
There was a hardware problem that required the motherboard to be replaced and since in most servers (DELL/HP) the hosting provider uses the on board NIC, your MAC ADDRESS changed.
Again, when they asked for the root password to update the MAC ADDRESS, you flat out refused. They offered you a KVM rental and you expected it to be free. Since they had already given you the option of a free quick and painless repair, they were not about to bend to your puny whim and allow your extremely long ass ticket to go unnoticed. Therefore, they refused to give you anything free from this point on.
Furthermore, since your 'migration' that supposedly was fubared by this hosting providers technicians REQUIRED a root password to update their MANDATORY IP CHANGE, and you flat out refused or failed to respond for probably 10 days or more in which case they probably had no choice but to pull your server off the rack and move it.
My suggestion? Leave that host and do it fast because I am 100% sure that the staff there does not want to deal with you and your less than $150 a month fee to them is just not worth the tickets you probably submit every two weeks for something that you flat out refuse to let them help troubleshoot.
If I were that company and I worked there, I would do everything in my power to have your server not renew on its next payment cycle.
Oh yeah, I forgot....
Tell everyone how it worked once you possibly agreed to pay for that KVM that you wanted for free.
I believe that it is possible that your server is online and has possibly been online for a couple of days now....
And you are still bitching about it like you don't have access. I hope I never find out that you have a server hosted at the company I work for because I will make sure that management see's this post and the defamation of character to said company and point out that we don't need anyone here that acts like a little girl who was told she can't play on the swing.
Have your logs set in a password protected web space
I'm not really sure what O/S your running - but I know on BSD copying logs to a readable format on a web page is fairly straight forward
Make sure you put them all there and turn on verbose logging ( if you have the disk space)
oh you want to see the logs here's the address
there is no need to give your provider root access to your machine
if this isn't acceptable maybe give them only access to read
but to give them the root password Not happening most I would agree to is read only access thought a limited account
no matter how good it is, it is human nature always wants to make things better