Handing Over Root Passwords to Clients and Contractors?

waa asks: "I have a client who's system I remotely administer. This particular machine has been up performing its various duties 'problem-free' for 4 months (since last kernel patch/fix). The client has, on-site, a consultant who pretends to know things he certainly does not know; Linux systems administration for one, and they now have requested the root password. Since it is their system, I'd imagine they have every right to the root account, however I know for sure that as soon as this is handed over, things will start to mysteriously malfunction, and I will get an emergency call to get them back in service (or worse, I will be blamed; ie: back-stabbed). I'd rather not have to troubleshoot and fix a completely preventable, and possibly complex problem. What are peoples' experiences regarding this situation? How have you handled it? Is some form of 'release from responsibility' contract in order? I need some advice soon" In situations like this, communication with the client is important. If you ever run into a situation like this, talking to the client and informing them of the potential problems is always a good idea. If any problems happen afterward. Start documenting them, and pass them back to the actual client if things start to become a problem. Anyone else care to weigh in?

Comments

Tripwire + backup...

[Bryan Andersen]

First backup the hole system as is and keep a copy for yourself. If that isn't possible, backup the configuration files and any data files you can fit.

Tripwire is your friend. Run tripwire on all files (even ones known to change). Save the tripwire file on both the system and keep a backup copy. When you get a support call you can use this to check what the guys have changed. For the most part you can run tripwire without checking checksums, just length, data, perms, etc. This will give you a list of new, changed, and deleted files. Not doing the chacksums lessens some of the utility of tripwire, but it gets you a list fast as tripwire dosen't have to read the file in.

Security tool can be used for your bennifit. They aren't just for security. When administering systems for developers who have root I always use tripwire on their systems. Often it tells me what they are changing so I can keep on top of their needs.

tiered support

[gascsd]

Not sure about in private environments, but in academia-land, we (previous position) setup a tiered support level.

Tier 1: only we (comp supp group) have the root pw. for nt, we would also be the only ones with the local admin password

Tier 2: root password is shared. on nt, local admin password is shared, or they (customer) has the local admin password and we still have domain admin rights; all 9x machines and macs

Tier 3: we have no root/admin access at all.

Depending on the tier determined when we would get to them. If a grad student installs some software and hoses the machine, but it's tier 3, it may be a few weeks until we get to it. tier 1 machines we'd get to that day, or the next. tier 2 was generally in a few days.

Like I said ... that's in academia-land, where customers aren't paying (at least here). Tell your client, "if he fubars the machine, don't look at me" ... whether they go with that or not, who knows. doesn't hurt to ask

Give them what they asked for...

[Steev]

Change the name of the user with UID 0, and create a user account called "root".
Give them the password :)

Re:Give them what they asked for...

[Yottabyte84]

Yeah, that would work, but what happens if he figures it out?

Re:Give them what they asked for...

[melonpool]

That's when you'd want to accuse them of anything and everything imaginable. :P Make sure you press that they "broke the root account!" From what he said already, they might buy it.

Re:Give them what they asked for...

[Yottabyte84]

Ah, great!

Well, here's one arragement I've seen (GM and EDS)

[Surak]

GM hires EDS to do all their sys admin stuff. They wrote it into the contract that GM doesn't have the root password, only EDS. However, some machines that are not EDS maintained only GM has the password. There may be other machines where GM and EDS share the password.

In any case, anytime only EDS has the password, these are the machines that get maintained in the most timely manner. Machines where EDS does not have the root password, or where the root password is shared don't get as much service. So it is to GM's advantage to have most of their machines be purely EDS managed, as those are the machines where the serivce level agreements are pretty much guaranteed.

Set up a good jail...

[tre]

Aside from the legal implications involved in "making a backup of the system", it woul dbe a good idea to have everything you consider important backed up for safe keeping now. Especially configs, dbs, etc...

Also, you shouldn't have a problem setting a fake root via chroot and jail. From what you've mentioned about the experience lvel of the thier local tech, he/she/it probably won't know the difference...

Finally, in a perfect situation you wouldn't have to keep clients that you wouldn't believe you for problems that they may have caused, but you've made the right step by recognize the chance of it happening. I would recommend alerting them to the implications of having shared admins on production machines... It leads to more technological and social subversion, and by explaining the downfalls of and your concerns, you may find they rethink their decision...

Psuedo Root

[cybermage]

Express your concern about stability to your client. Find out from your client what they want to use the password for. Explain the difference between routine administration (e.g. adding users) and server maintenance (e.g. kernel patches) and suggest that you set-up psuedo root access for routine tasks and then backup the files the psuedo account can change.

It is not uncommon to provide clients with a 'root' account that lets them configure the services they use (e.g. POP3, Samba) without giving full access to things like 'make'. Understanding how to do this is left as an exercise for the reader.

How about logrotate, kerberos, CVS, groups

[QuietRiot]

I've been working as a student systems administrator for a computer lab at Cornell for the last 3 years on linux and NT systems. We've got lots of students with root access to some machines quite critical to the operation of the lab. We have had, on many occasions, situations where a lack of communication and multiple people working on the same project/server/whathaveyou, shall I say, stepped on eachother's toes. I'm not sure if we ever implemented any of these, but I've done some thinking about it! (especially after a day's worth of work on a config file gets blown away by a vi edit from another guy with root who doesn't know what he's doing....) Anyway, here's a few things you could do...

1) Set up a cron job to backup configuration files
You could either write a simple script to tar up /etc/* or whatever other files your clients might want to fowl up. Or you could use the logrotate program. It's a flexible solution to manage log files and could be used to tar up and compress periodically the files you'd like to backup should anything get messed up.

2) Use the "wheel" group
to allow only certain users to perform administrative functions on files you chown root.wheel. You could also do this for a "web" group (#chgrp -R web /home/www) or anything else to give access to certain files for certain people. Instead of "giving them root," ask them which users want to have control over which sets of specific files on the system, make groups for each set of files, and add the users to the group that need access. This would allow you to track down more easily who modified a certain file, since only a few people may have had access to it. (easier finger pointing basically). This is much better than knowing that a bunch of people could have done it, and getting blamed for it yourself.

3) Setup sudo
Edit /etc/sudoers

4) Use process accounting

5) Set up a simple kerberos5 installation
This allows you to give root [or any other administrative-type account function] access to certain users via a ".k5login" file. This allows you [root] to simply add users who are allowed root access [to kerberized services] by putting their principal [system user account name] in root's .k5login file. You needn't give them the root password. They can just login as themselves and they will be allowed to use their own tickets to access kerberized services that require a "root" principal. All access is logged so that backtracing who did what and when would be possible. Note: This may be an unnecessarially complex method to implement. It can however be used for a wide variety of services and is considered quite secure and flexible, particularly for large networks. It won't work for general file access, but if you can kerberize some of the services you'd like to administer, this would help immensely.

6) Tell them to make backups before they change anything or implement a CVS system
on all config files that anyone would want to changeCVSing config files would force documentation and allow you to revert to any previous incarnation of your system if something somehow "broke."

Hope this helps!

Hate to say it, but get a lawyer

[unitron]

That consultant has to do something to justify the expense of keeping him around. He's probably also on a power trip. He's gonna show 'em how wonderful he is by saving them the cost of keeping you around. He's also going to want to make it look like he saved them from your ineptitude just in the nick of time.

You need, notarized and in writing, an admission from them that everything is working fine right now, another stipulation of all the things that can go wrong when running as root, and that they fully understand and accept those dangers, and you need a lawyer to make sure it's worded so that it'll stand up in court.

You're in the crosshairs of fate. Time to shop for Kevlar.

Root Isn't Takin' a Turn With Your Sister

[InitZero]

The system belongs to the customer. It's just root. Give it to them. If it breaks, you fix it. If they break it, you fix it. That's what you get paid for.

How many people do stupid things to their cars? Have you ever heard a mechanic consider not giving a customer his keys after a repair has been made? Of course not. The mechanic would go out of business. Your job is to fix computers. Do it.

To protect your reputation, however, make sure there are no direct root logons. By making folks logon as themselves first then su'ing to root, you know whose pecker tracks are all over the issue.

InitZero

Re:Root Isn't Takin' a Turn With Your Sister

[penguinboy]

If it breaks, you fix it. If they break it, you fix it. That's what you get paid for.

Part of the point is that the consultant may royally screw things up that wouldn't go wrong all by themselves and that it's really annnoying to have to deal with that crap. Pay is nice, but not as much so when you tear your hair in the process of getting it.

Re:Root Isn't Takin' a Turn With Your Sister

[mjoconnor81]

yes, you may get payed for fixing it, but i find that if i set up a server, network, or workstation and have done a good job, i don't like to see it not working. I already put the time and effort into optimizing it, to allow somone else to destroy 'my system' is not something i would be very willing to let happen. just because you get paid to go back and fix it doesn't mean that you should encourage an opportunity to grant somone access that would allow them to destroy your work.

on the other hand, i have been in the other tech's position, the only difference is that i knew what i was doing, and the vendor that installed the machine would not give out the root password. i just accpeted it, and call them every time there is a problem. if they arn't going to make it easy for me, than i'm not going to make it easy for them.

Re:Root Isn't Takin' a Turn With Your Sister

[Chanc_Gorkon]

You have good points and bad points here. Sometimes we hate having to get a hold of the consultant for stuff we already know how to do. If you as a consultant have a problem doing your job (fixing and setting things up for the customer, whether the customer is at fault or not is your job) then you should not be a consultant. How many office idiots screw up their Win98 machines? Is it a pain? Yes but if everything worked all of the time, then we would not need you (think about it....if a system never went down, always worked....the need would not be perceived, even though you are doing or had done an excellent job, because nothing ever goes down!). Good sysadmins are an insurance policy. They assure that things are backed up, setup correctly and all of that PM(Preventative Maintence) stuff that the user never see's gets done. This way you/they can actually add more things and do more stuff since your not busy putting fires out all freakin day.

We have a Mainframe Systems consultant who maintains the system for us. We also have the head tech support person who heads up tech support on mainframe mostly, but can do some server things and definitely work with a DB. They both have all the passwords and access needed. If one screws something up and the other fixes it, they generally will figure out that they weren't supposed to do whatever they did. Also, even though you have a root password on a machine does not mean you own it. The user/institution does own the machine and depending on what level of support you are to give (from just handholding to doing everything except the actually work it gets used for) depends on whether the password is given out. That's not decided by you. If you have it in a contract and the institution signs it, well then it's in stone that they don't get root. Expect to be called for silly things at all hours of the day (especially if your the only sysadmin with the password). Controlled root access can be useful. Especially if it's just to add a user or clear a print queue....do you REALLY want to be called at 4 am for that?

Sudo?

[staplin]

Find out if sudo will be sufficient. Sudo can be setup to allow any given user to have "rootly powers", except that everything is logged.

Check out the related Ask Slashdot Keeping Audit Trail of Activities from Root Login?.

Clients!

[annielaurie]

I'd certainly get busy and cobble together a "System As Built" document. Describe the server, its configuration, and its functions as completely and concisely as you can. Think of it as a "snapshot" of the system on the day you last had control of it. Try not to leave anything out. Deliver it to your client with a memo saying you can't be responsible for any changes made from this point forward. Remind him of your bill rate without making any further comments.

Tedious? You bet! But you may find such an as-built document goes a long way towards covering your posterior if and when it's left flapping in the breeze.

Then go ahead and hand over the access with a clear mind. Relax and consider that you'll probably be called in to restore order.

Consultants do this quite a bit.

Anne

BSD "script" command?

[PatJensen]

I'm surprised no one has suggested this, but when I perform auditing on various shell users, I found the best way to keep session logs is to use the "script" command. It should be available on your system. You can make it "exec script" so that if the user attempts to kill it or exit script, it logs them out. Or put script into a wrapper, and use it as their default login shell.

If you do use it for root, make sure it is available in single user mode, or on your primary disk so that you are able to access your system if it goes down. Just a little warning. You can include the script command in your shell rc files in /etc, or place it directly on the user, and have it save the script to /var/log/scripts/(user).(date). This will show everything in the session, including VT100 interactive applications, like IRC and such.

Great way to have time stamped logs, and it's even great to know what changes you made at midnight the night before, in case you forget the next day. Look into it!

Hope this helps,
Pat

My view.

[LWolenczak]

My view is sure, give them the root password, but if they want the root password, then they can read root's email.

I find that when we *DO* turn over root/administrator password to customers, that the person its turned over to, has previous experence with unix type boxen. Mostly AS/400 programers oddly........

Run for cover!!!

[kko]

At my workplace, I have a lot of trouble with clients who think they know what they're doing, but always end up screwing everything. I usually get the blame for everything that happens(I hate this job sometimes), but keeping some logs and some system documentation has saved my butt more than a few times. Luckily for me, my clients have no root access to my Linux systems (or else I'd already be dead), but your case is a very difficult one.
Now, I know you get paid to fix computers, and it _is_ your job to fix this machine in case it breaks, but what about the blame factor?
This "consultant" could very well place the blame on you for anything that happens to the system while he is logged on as root, and this may jeopardize your job...
So keep logs of everything, create some system documentation, backup, get a lawyer, revise your contract... do everything in your reach to protect yourself from a sneaky "consultant"...

If it was mine I would want the root password

[innocent_white_lamb]

Looking at this from the point of view of the "customer", I think it would definitely be in one's best interest to possess the root password even if you didn't plan to ever use it.

Without it, you are "hostage" to whoever set the thing up for you. What if, for example, there is some sort of a dispute and the customer wants to lock the consultant out of his property (including the computer system). Or fire the consultant (with or without cause) and hire someone else. How can you do that if you don't have the ability to change the root password?

In the final analysis, the computer is the customer's property and should not be held hostage by the consultant.

Re:Well, here's one arragement I've seen (GM and E

[liquidcarpet]

This also the way EDS can ensure that their client has trouble changing their outsourced resources. As a consultant, I try to let my work speak for myself, I don't want to get in the habit of trapping clients.

I never voted for Ross Perot, why should he have root?

If they really want root...

[cr0sh]

...let them have it.

But let them have it in a proper way.

First off a disclaimer - I still consider myself a relative Linux newbie, and not a sysadmin - I have a Linux box here at home (what I am using right now), and a Linux based firewall/router I built, plus I do a bit of my own admin on these boxes, but take what I say with a grain of salt, because I might be wrong.

Basically, what you want to do is have them do the whole contract thing, where the contract states that they know the system is running properly right now, and that you are turning root rights over to them in full, and that anything that happens after the fact is their responsibility, and that should they wish to use your services in the future, you will bill at your normal rate x (1.5, 2.0, 3.0?). Plus, the contract should state that they acknowledge that they are the _only_ ones with root access, and they know it.

Then, with the head of IT, or whoever understands enough and is "at-the-top" IT wise, show them that you are removing all (or as many as possible) root level account logins, and show that there are only a few remaining (or "root" alone). Then log in as root, and have the "head dude" change the password on each and every one of them - while you are outside the room. Tell him to pick good passwords (or obscure ones to write down in a log or something - whatever _they_ want - so that you can't guess them). If he is really willing, tell him to try to let you log in, if you can't, then it shows that you _really_ don't know the root passwords (ok, kinda juvenile, but this company sounds that way).

Then ask for your check and leave - hopefully you won't hear back from them, and if you do, it is with the knowledge that while it will be hell to fix whatever they screwed up, at least you will be getting your what it is worth...