Slashdot Mirror
Linux in a Business - Got Root?
greenBeard asks: "I work for a government contractor, and have recently convinced them to purchase a Beowulf cluster, and start moving their numeric modelers from Sun to Linux. Like most historically UNIX shops, they don't allow users even low-level SUDO access, to do silly things like change file permissions or ownerships, in a tracked environment. I am an ex-*NIX admin myself ,so I understand their perspective and wish to keep control over the environment, but as a user, I'm frustrated by having to frequently call the help-desk just to get a file ownership changed or a specific package installed. If you're an admin, do you allow your users basic SUDO rights like chmod, cp, mv, etc (assuming all SUDO commands are logged to a remote system)? If no, why don't you? If you allow root access to your knowledgeable users (ie developers with Linux experience), what do you do to keep them 'in line'?"
Much as I hate to break it to you - this is SoP.
End users Do. Not. Get. Root. Even allowing SUDO access to change file permissions, copy, or even move files is just asking for trouble.
Installing software or libraries? Hell no. Not on a live system.
If they have a development-type machine at there desk, that's one thing (just don't call for support if you break the damned thing). Even then, my preference is that they have limited access.
On large, shared systems, users get as much, and as little, access to do their jobs as necessary, and absolutely no more than that. I have to keep the system up for other users, I can't have power-user #1 screwing things up by changing permissions on something they really shouldn't be touching (let's take the compiler for example...)
A little knowledge makes one dangerous, and I'd just as soon noone other than those paid to admin the machines have access.
Ever heard of setuid root?
ACL's are quite nice and so are different levels of security.
http://saveie6.com/
Sarbanes and Oxley. I don't know you, you don't need that access, we have a process in place and I am not signing off on you. Follow the procedure or go somewhere else to work.
Strange women lying in ponds distributing swords is no basis for a system of government.
And even allowing chmod, mv, etc via sudo can be dangerous. Someone accidentally issuing a "sudo chmod 777 -R / ", having meant to type "./" for everything below their current directory, isn't going to be good for your system health and is going to be somewhat of a pain to recover from, even if you do know who screwed things up.
You are going to get a bunch of responses. most of them from people that will say something like.. "NO." "NOBODY GETS ROOT, PERIOD".
Well, in an ideal world, it would be that way. We would setup systems for people to use and they could just use them without root privledge. Unfortunatley we know that isn't possible if you want your users to actually be productive and get things done.
I work for a large software company. Trust me you'd know the name of it if I could tell you. We use linux on the desktop, as well as the servers. We also have some Microsoft servers that are either for legacy purposes (havent been updated yet) or for testing applications against MS environments. Anyway...
All my users have laptops with Linux on it. They all have the root password to their individual laptops. Many of the also have a server at their desk for their own testing purposes. They have root to that.
However, the "real" servers that are accessed by someone that isn't themselves, the users do not get the root password, ever.
I look at it this way. If you bomb your laptop or your test server, either you can fix it, or you can call me and I'll walk you through fixing it, fix it, or just give you a new clean configuration.
If you bomb my server, I'm going to make sure you never have access to anything, ever.
Don't Tread on Me
Developers with Linux experience are a LOT more dangerous than developers without linux experience. My experience has been (100% of the time) when I give "experienced developers" access to commands like 'chmod', I find all kinds of files mode 777 (among a list of about 10,000 random, stupid things developers do) because, well, I've heard pretty much every excuse you can imagine.
The problem is that as soon as people outside of the core sysadmin team have access to critical system commands (cp, chown, chmod) the integrity of the box is left to chance. There's always the possibility someone is going to do something outside the policy. Sysadmins make it their job to know and understand the impact of every change to a box. Developers tend to make changes in order to get their stuff to work, regardless of the consequence (hey, each group is just trying to do their job, which is "make it work!!" -- I'm not defending either side).
My rule of thumb:
- Developers get root in their dev environments.
- Sysadmins get root in the production environments (developers shouldn't even have user-level logins to these machines.) If your company is releasing software (even for internal use) properly, the IT group will be managing the code as a product, using developers as a help desk rather than letting them manage the applications directly.
Stick to this and everyone will be happy.
-- People who hate Windows use Linux. People who love UNIX use BSD.
I come from the other side of the fence. I am a developer of multiplayer servers. For my part, I couldn't do my job without root access. I need to do things like set the date and time on the machines, install to /bin, upgrade compilers, etc. If I had to ask the helpdesk every time I needed root, they'd just set up right outside my cube.
On my Windoze machine, OTOH, I have no need for system level permissions, and I don't ask for them. I can install software, but so can all the other developers (and, I think, anyone in the company). All I use that machine for is e-mail and testing client connectivity to my servers, when I'm not using my Linux test client.
Some people need root and some don't. Don't make blanket policies unless you're prepared to make exceptions. Oh, and, for everyone's sake, if you do restrict access, please, please make sure that at least one person who can change things is available 24/7. I can guarantee you that Peterson up in Accounting is going to have a system crash that requires help when trying to get the year-end reports out at 2:30 A.M. before the big board meeting at 9:00.
... I'm special and rules don't apply to me.
How can I convince others of this?
The surprise isn't how often we make bad choices; the surprise is how seldom they defeat us.
This sounds to me kind of like the situation in a university Unix network. I'm not entirely sure I understand what you necessarily need that wouldn't be available (though I would like to know, to get a better understanding of the question). Certainly, at the university I attended, we didn't have sudo access, but we were able to develop some rather powerful applications.
/usr/local/whatever, and tell them how to set up their LD_LIBRARY_PATH to link against it, and they're good to go.
I can see an adjustment period of a couple of months, where applications you regularly use aren't available, so you ask for them to be installed. After that, assuming they don't see the general need for an application (or they don't want to have to officially support it), you could theoretically install applications under your home directory. (I was thrilled when I became a grad student, and got 100MB of disk quota, so I could compile and run Blackbox as my window manager instead of the crappy twm we were generally stuck with. In fact, I made it globally executable, so my friends could use it as their window manager. In fact, I received a phonecall once from one of the admins, asking me what this spinning "blackbox" process was running on one of undergrad servers, since I was the only grad student or professor (and therefore in the phone directory) who also ran it.)
These days, as part of my regular job, I am one of the unofficial sysadmins of a Beowulf cluster (largely because I'm the one of the only ones who have developed MPI applications that run on it). I get the odd request from other users who want me to hook them up with some library or such. I compile and install it under
Again, I have to ask what you need that requires root or sudo access, that can't be solved by the rare admin call or installing under $HOME. (I really don't mean this in an insulting way. I do want to know. The story post is a little brief.)
(Disclosure: I work for Sun and work with Linux since 1994).
Why would you move the modelers to Linux from Solaris? There is no real advantage....
Sure a Beowulf cluster is a nice piece of hardware, but hardware can only compensate a bit for programmer productivity... If their code is written using MPI or OpenMP or some other standard clustering environment then there shouldn't be a need to move the developers, should there? Just recompile and go.
It is really much more efficient to shove faster hardware under a programmer then to force the programmer to adapt to a different programming environment. Programming for a cluster is hard enough without having to take into account the details of the operating system, forcing them from Solaris to Linux might improve the execution part (on a side note, have you considered Sun's clustering tools?). But it *will* set them back in productivity while they move to different compilers and adapt the execution of the program to the Beowulf environment.
In my opinion you have forced your customer to make a move on questionable grounds.
Now to the matter of security. As you are aware, Solaris has the highest level rating for security. Secure Solaris is the defacto operating system at a number of government agencies. Linux cannot hold a candle to the multiple access levels of the Secure Solaris operating system. You state that you are frustrated at needing the helpdesk for file permission changes. What is your point? Are you using the fact that YOU don't like the limitations to change a customer from Solaris to Linux? Or are you complaining that the customer's environment did not deploy secure solaris with its multiple access layers? In Secure Solaris there is no need to muck with sudo. Each file can be managed properly from a security point of view (come to think of it, much of that can be done with Linux too).
Before I answer your question, let me state that I understand your point of view. When I joined the navy as a UNIX project manager, the admins gave me absolutely no rights whatsoever on the production systems. Their reasoning: '.. he can do things I don't understand, can't control or prevent.' There will always be a tension between the lockdown desired by the admins to keep their environment safe and secure and the users who want total freedom....
In my mind there is NO good reason to give ANY user root access in a secure environment. Period. If you have frustrated in the past by having to interface with the helpdesk, then the helpdesk needs to be improved. At the same time, I assume, any user has full access to their files.
You mention that you have convinced modelers to move to a Beowulf environment, then why the issue anyway. If they run cluster code then they run as user. All the need are basic user access rights, nothing more...
Maybe I don't understand your point....
I also work for a defense contractor and adhere to strict security rules. We have a fairly simple means of controlling our developers who need root access. We buy their systems using their bosses' overhead or project charge numbers, place them on a monitored, isolated subnet and, when they hose the system, all time expenditures are billed to one of the previously mentioned charge numbers. At my billing rate, it doesn't take too many incidents for them to feel serious heat or be canned. Either way, they do not touch production machines or cause problems that cannot be quickly isolated by disconnecting the subnet.
It works for us.
I've been both a developer and an administrator.
/usr. root access as we know it is a luxury that should be reserved for true administrative duties, unless absolutely positively necessary.
As a general policy, if a developer needs root access, they need to prove to me as an administrator that they actually do need root access. I'm not going to give root access (sudo, su -, or access to privileged accounts), even on a development box, to someone that needs occasional chmod privileges. More often than not, the people who are begging for root access are those that have been so spoiled by coding on their own Linux boxes that they lose sight of all the best practices that contribute to good code. They want foolish things like directories with 777 privileges so they can drop temp files when there are 30 better ways to do it. root is not a cure all... just because you're used to it on your own machine doesn't mean it's appropriate for coding in a multi-user environment developing customer-facing applications.
In the end, there are very few specialized applications that actually require root access to work. I will concede that sometimes root access is necessary but it needs to be treated on a case-by-case basis. I'm of the belief that a properly written application should be written such that it can be run with the least amount of privileges, and can be installed anywhere... not just
Another sysadmin who is going to tell you that I don't give out root or sudo access to users. Most users who think they know enough, or even DO know enough, really know enough to make big problems. They invariably never check with me before making a change, or tell me that they made a change, or even admit to having made a change when they inevitably screw something up.
I make them come to me for everything. But not directly. That's what the ticketing system is for. The ticketing system justifies my existence, keeps any requests from slipping through the cracks, and helps to keep track of ad-hoc changes made to any given system.
Many times end users think they need root for something when they don't. For example, there might be some niche tool that they need installed on a system. Or do they? If the one user is the only one that is going to use it, I advise him to do something like "./configure --prefix=~" to build apps to install in his home directory. You don't need root to install apps anymore. Besides, if you want an app installed for everyone to have access to, sysadmin should be doing that anyway.
It might be a pain in the ass to make you go to the sysadmin for everything, but in the long run it will keep things running smoothly and perhaps force you to be a little more disciplined in your work.
Why in the world do your users need root access? On Windows it makes sense; all too many poorly written programs refuse to install or run unless they can run roughshod over the entire system. But this is Unix. It's a rare piece of software that can't be installed and run as a user. Most can even be installed as a user but made available to others. Yeah, it's a bit more frustrating that you can't just install the latest RPM, but if you're skilled enough to install an RPM, you can probably manage "./configure && make --prefix=~/mybin && make install". Changing file ownership? Again, why do you want that? If you're sharing files with other users, get a group set up and chgrp the files appropriately. If you have lots of complex sharing needs, set up one of the Access Control List options.
Ultimately users shouldn't need root. Professionally I development clustering software for Linux and other Unix systems. I regularly install new applications I'd like to use in my home directory. Our administrators set us up with a good ACL system (courtesy of AFS). I do the cast majority of my work as my own account. The only time I need it is to test root-specific aspects of our software (if launched as root, it runs jobs as the user who submitted it). I can't remember the last time I switched to root, probably a month or so ago.
Unless you've got a damn good reason, your administrators are right to withhold root access from you. Your desires aren't good enough.
Search 2010 Gen Con events
I don't really get the original premise. Nobody needs to be root to run chmod, cp, mv, etc on their own files. The only command mentioned one might need root for is chown. Which would make me ask the question, why do you need to change file ownerships so often?
It would take a hard-core serious business case to convince me to grant someone root access, even sudo-limited root access to a production system. The fact that I might have a "log" of whatever broken thing they did to take a business critical machine down is fairly irrelevant to me. My job is to make sure that doesn't happen in the first place.
...and I do NOT WANT ROOT.
I have root on my workstation (cold dead hands and all), but not on a single server--not even a dev server.
sudo on things like mv and chmod gets you a root shell on the box fer chrissakes, why not just put the root password on a sticky on the rack?
When something goes wrong, I don't want to hear, "Maybe the dev did it." I didn't do it--no access. When we go to prod on something, I don't want to hear the admins complaining they don't know how to promote the app because some ass developer did it manually in dev instead of creating a proper install.
If you need root to chmod something, then your admin hasn't set up the box properly. Either he doesn't know what he's doing, or you haven't told him properly what sort of environment you need. Either get a better admin, or write up a clear description of all the functionality you require. Either way, you don't need root.
Of course, the smaller the business, the more likely an admin is a dev and vice versa. In that case, all bets are off.
Basically you need to have your entire filesystem layout setup properly, with "project" areas where each "project" has its own directory tree with setgid for the project's group on all the main directory and sub directories. Each major "project" would have a group setup for it. Then all file permissions would be covered by anyone in the group, or possibly a "project's lead" who keeps track of all the groups and knows what permissions should be set to different areas (i.e. for data sharing between projects etc.).
Once the infrastructure is in place, the worst thing that happens is that a person is not a member of the "group" and just needs a helpdesk call/form to gain group access ("ok'ed" by a lead member of the "group"). Basically something that can happen in 5-10 minutes time if implemented properly. With the setgid, all new files created in the areas will always be owned by the proper group, which has full access to chmod/chown those files (assuming someone doesn't do "chmod 700") but even then, cron jobs can be setup to run every hour or so that do a "chmod -R 770 /" to any/all project areas (with the cron job removed if you need to lock the area down to no access).
This is how it should be done, no sudo needed. All the work is in the preperation, with true processes needing to be setup and implemented (basically a form/forms for creation of a "new group" (which includes group ownership as well as a box to transfer "ownership" to another person), another form/forms for requesting new data areas (with what group owns the area), and finally a form/forms for adding/removing members to/from the group which gets signed off by the current group owner). Optionally another form for "locking" a data area to keep all access out. Then it simply needs to go to the IT staff which then simply reads down the "process" document and verifies the data in/on the form and either creates a new directory (setting the setgid bit and setting proper group ownership), adds/remove a user to a group, creates a new group, or moves a user to the first name in the group file (for easy tracking of the group owner or updates a seperate documents with this information).
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
I'll give you that you need root to set the date and time, but your system should do that for you with NTP, so it's not that you don't need root for that, but that you shouldn't have to do it...
/bin unless you're the system administrator. If you develop software that can only be run out of a particular directory, please post your name and address in a response to this comment so that those of us who have been forced to use such software in the past can come beat the living snot out of you.
But for your other examples... You do not need root to upgrade your compilers. You do not need to install things in
If you're a developer, you should probably have your own machine, which you would have root access to, but save for setting the time, you don't need root for any of the tasks you've described. Your license to call yourself knowledgeable is hereby revoked.
Even if you do have root, unless you expect the users of your software to have root access as well, you shouldn't be using your root access, or you'll end up wondering why your users have problems that you don't see on your development system.
There are some interesting 'privledge escalation' things that can happen on machines 'owned' by the user on a big network, though.
The one I experienced firsthand was a Windows NT machine that was my desktop that I ('naturally') had full admin access to. This was a machine on a large corporate network that was very diverse (there were Solaris, OS/2 Warp, Netware, and Windows NT servers on the network). I discovered, quite by accident actually, that if I ran the POSIX Interix (now SFU) shell on my NT workstation (something the company had bought for me, and I had installed myself,) that I could create any account I wanted on my local machine, and it would allow me, using that account name, to access shares on the network, doing whatever I wanted to files my username 'owned'. I am talking about the network that a company that makes implantable medical devices kept their work on. I suspect the 'defect' had something to do with NIS and 'travelling profiles' in Solaris, and the security system not being equipped to deal with other Unix-like hosts on the network that weren't secured. Incidentally, I didn't discover the problem by 'poking around where I wasn't supposed to be,' I simply noticed I was suddenly able to do things to files I normally had access to without entering my UNIX password as required in the past. Something clicked in my head, so I created a local account on the NT box that matched an important person's UID on the Unix system... yep, I had all his permissions.
Delete test account. Never touch again. Too scared to mention it to anybody. It's been enough years now that I can even mention it in public. I hope they've secured things a bit better now, because these days there are unsecured Unixy systems all over the place.
resigned
Hell, at time I think I shouldn't even give the users a keyboard and monitor. It's not a question of if the users will screw something up, but when. They are ALWAYS doing things they shouldn't. Thus the less they can do, the better.
The worst are the "I'm a sysadmin" types. For every one I meet that actually has the experience to make them a competent sysadmin, there are 50 that know just enough to be dangerous, but think they know it all.
For example some time ago I decided to roll Firefox out to the educational labs and make it the default browser. All other considerations aside, it's minority status in the browser market makes it far less of a target. Well a couple days later I get some guy in who's bitching about Firefox being installed in "his domain" and he wants it removed. Upon further questioning, it becomes clear he believes that programs are installed in user accounts. I cannot seem to convince him that the program is a local installation on every system and no, I'm not removing it.
Now for Windows systems, the damage someone can do is somewhat limited since all software installs are on the local system. However the UNIX systems all run off a central server. Like hell we are giving anyone anything but read access to that. All the time people want things installed or modified for their particular project. Quite often, they have no idea what they are asking, and what they want done would completely break the app, or worse.
I agree that access should be as limited as allows you to get the job done. Now, in some cases that needs to be total access. Fine, you get a system that's seperate and you assume responsibility for it. If you are doing something such that you need system access, you'd better have the knowledge to fix what you break. In other cases, come to us, that's what we are paid for.
We even operate that way internal to our group. I don't just go and change shit in DNS. Not because I don't know how to, not because I don't have the root password, but because it's not my area. Better I should ask the guy who is supposed to do it. That way, there's less chance somerthing that gets broken.
I think the problem is that some users have a real inflated sense of self importance and entitlement. They think that their project is real, real important, more important than everyone else's. Thus they don't have the time to wait to have the admins do things, they want to just be able to do them themselves. If it messes something up, well then the amdins can fix it. Of course people like that are also the most likely to do something that will break things for others.
The more shared the resource, the more you have to be strict with the access. Even on user desktops, limited access needs to be the rule. Support can't spend hours and hours fixing problems caused by users that don't know what they are doing. It's just not cost effective.
If you truly have the need and knowledge to run your own system, then fine, take it up with management. However part of that understanding has to be you can't bother the support team if you hose things. If you aren't good enough to admin the thing yourself, you probably ought not have admin permissions.
At MIT, all Athena workstations have the same root password, which is freely distributed. Any user can get the root password by typing in tellme root at a console. There are thousands of these machines scattered all around campus, with many different hardware configurations. This has been policy for many years now. Sometimes you get stupid kids doing stuff like this, but other than that, it seems to have been working...
Take a lesson from this guy - he's smarter than he looks. When you find a security hole like this, do NOT report it unless you can do it anonymously. If you can't report it anonymously, then just sit on the knowlege until the end of time. This is your job, your life, and your paycheck. We've all read the stories about how the person who reports a security hole gets criminally prosecuted for "hacking". You might be a smart person, but everyone around you is a blathering moron. That is a FACT. That blathering moron isn't going to say "thanks for pointing out this embarassing security hole that my ass was hanging out of." The blathering moron is going to try to cover his ass by blaming somebody else, and the easiest somebody is YOU. That way he takes care of the problem and gets brownie points for uncovering a dangerous "hacker" within the company.
Next thing you know you're getting arrested by a nice FBI agent named Bob, and then getting cornholed for days in the local jail waiting for a judge to set bail. It's not worth it.
Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
Just logging the sudo commands isn't going to give you nearly the auditing ability I suspect you're looking for, and giving them any kind of root-level access to the filesystem is game over.
Figure that any chmod u+s is suspicious and will get caught?
Figure you'd notice their subsequent use of whatever new sudo permissions they just gave themselves?
And, look at that, suddenly their UID is 0.
The list goes on...
I have a somewhat balanced view of this, as I work for a University and have a variety of different interactions with Solaris and Linux. What follows is a few notes on Linux vs. Solaris and Access Rights across different categories of system
Firstly, our Production MIS Systems:
Nextly, our Development MIS Systems
Thirdly, Academic Development Systems
Read a few "Ask Slashdot" questions, and you'll understand. Ask Slashdot can almost always be summed up: I'm old and cynical, and longer surprised by clueless fanboys with an exaggerated opinion of their own experience, intelligence, and skill.
I will never understand, however, how any of them manage to find jobs.
As both the primary web developer and web admin, I probably qualify as a "special case" because I'm both end user and quasi-sysadm. The sysadms take care of the O/S, standard software, primary user accounts, etc., and I handle server software configs, user support on the development system, etc. I do have sudo privs for chmod, chgrp, chown, and so forth to give users ownership of their stuff, as well necessary sudo privs to manage certain daemons. However, end users do not touch my production box, and have zero special privs on my development box. My "regular user" log-in has no sudo privs - I'm a Jane Schmuck just like the rest of them.
Nah, that's just a standard limitation of NFS. There is no security in NFS; the unofficial expansion of the acronym is No Fucking Security. The server trusts the client is providing a valid userid. You spoofed the userid and NFS has no way to detect that because the server assumes the client always tells the truth.
Some environments implement netgroups to limit the opportunity for attack. The server checks incoming client connections against this list; clients on the list are assumed to be properly secured so nobody using the machine can spoof a userid. This is not very effective either because spoofing a client IP address is almost as trivial as spoofing a userid.
What you found was simply standard practise for NFS, as frightening as that might be.
On a production box, the admins have access to sudo, and root itself is locked down except for scheduled maintenance/upgrades or emergencies. No paperwork, no root.
As a developer with over 15 years *nix experience, I have never had root access to a box unless I was doing an install, except for my own desktop workstation. In the case of my desktop, the only reason developers had root was so we could kill rogue services during debug sessions gone bad.
Under no circumstances do I agree with any user installing additional software on a box. If it's needed, it gets approved and installed for everyone who needs the functionality, not by rogue users.
I do not fail; I succeed at finding out what does not work.
To address the article's question, groups solve more than just file permissions; consider an environment in which users in the admin group have the ability to do things (via sudo) as the admin user, who owns /usr/local and all of its children. This lets priviledged users install things, but prevents them from accidentally messing with them (the admin group should not have write access to /usr/local, so sudo is required).
A more restricted implementation would chown /usr/local/stow to the admin user and grant the admin group sudo access as the admin user plus sudo access to the stow command (or perhaps a shell script that ensures items are stowed to /usr/local).
Of course, /usr/local is only one potential target. Perhaps your environment is better suited for /arch/beta or /opt. Also note that this idea is easily abstracted and applicable to other tasks.
Use my userscript to add story images to Slashdot. There's no going back.
If you can chmod or cp as root, you can do anything else as root seconds later.
We only give somebody root ability to do something if it's essential to their job, and a team reviews any new application of that to ensure it doesn't facilitate unwanted privilege escalation.
Their basic access to a system at all is reviewed quarterly by their manager, and if he doesn't take action to change the default answer to "yes, they still need this access", they get deleted.
Show me a publicly-traded company that's not acting like that, and I'll show you the next Enron.