Slashdot Mirror
Ask Slashdot: Linux Login and Resource Management In a Computer Lab?
New submitter rongten (756490) writes I am managing a computer lab composed of various kinds of Linux workstations, from small desktops to powerful workstations with plenty of RAM and cores. The users' $HOME is NFS mounted, and they either access via console (no user switch allowed), ssh or x2go. In the past, the powerful workstations were reserved to certain power users, but now even "regular" students may need to have access to high memory machines for some tasks. Is there a sort of resource management that would allow the following tasks? To forbid a same user to log graphically more than once (like UserLock); to limit the amount of ssh sessions (i.e. no user using distcc and spamming the rest of the machines, or even worse, running in parallel); to give priority to the console user (i.e. automatically renicing remote users jobs and restricting their memory usage); and to avoid swapping and waiting (i.e. all the users trying to log into the latest and greatest machine, so have a limited amount of logins proportional to the capacity of the machine). The system being put in place uses Fedora 20, and LDAP PAM authentication; it is Puppet-managed, and NFS based. In the past I tried to achieve similar functionality via cron jobs, login scripts, ssh and nx management, and queuing system — but it is not an elegant solution, and it is hacked a lot. Since I think these requirements should be pretty standard for a computer lab, I am surprised to see that I cannot find something already written for it. Do you know of a similar system, preferably open source? A commercial solution could be acceptable as well.
I would do it up A Clockwork Orange style.
The original BOFH stories are a good guide: http://bofh.ntk.net/BOFH/
Either give it away or get top dollar, but never sell yourself cheap.
Try ulimit. It helps a lot keeping things under control.
hilarious
I believe you can do at least some of that with systemd user sessions and resource restrictions
http://0pointer.de/blog/projects/resources.html
User sessions are currently kind of beta-ish but they're getting better / more useful... I already launch emacs and a MIDI synth through it on login, and it works wonderfully (ironically, though, PulseAudio, the other Lennart project that got a lot of flak, doesn't launch through this mechanism yet).
Trust your users.
Is this 1988? The easiest/cheapest solution is spend a couple bucks on decent machines.
You are probably using NFSv3 (instead of NFSv4 or some other protocol) which does not have encryption nor authentication.
Your user's accounts can be hijacked via ssh key login.
See the YouTube video: Hack SSH : Default configuration of NFS server
Maybe they have an EDU license? http://www-03.ibm.com/systems/...
-- How many sigs are as useless as this one?
Seems like you are trying to work out a solution to a problem you don't have yet. Maybe first see if users are just willing to play nice. Get a powerful system and let them have at it. That's what we do. I work for an engineering college and we have a fairly large Linux server that is for instructional use. Students can log in and run the provided programs. Our resource management? None, unless the system is getting hit hard, in which case we will see what is happening and maybe manually nice something or talk to a user. We basically never have to. People use it to do their assignments and go about their business.
Hardware is fairly cheap, so you can throw a lot of power at the problem. Get a system with a decent amount of cores and RAM and you'll probably find out that it is fine.
Now, if things become a repeated problem then sure, look at a technical solution. However don't go getting all draconian without a reason. You may just be wasting your time and resources.
Some of what you're asking for are ulimit settings - total number of processes, for example. That's pam_limits. Some could also be handled with pam_tally2. Or, since you're already using LDAP, you could use a simple web-based reservation system which specifies allowed login hosts in the LDAP server for however long someone wants to "check out" a machine; that's how I've done it when I've needed to control access to cluster resources.
When you talk about controlling other resources beyond logins, it's generally better to handle it at the application level rather than the OS level if you can. But using ulimits (and again, this can be integrated into LDAP pretty easily), you can restrict resources and apply process priority (ionice and nice are your friend) based on membership in a specific group or another LDAP attribute.
You could, for example, create a "highpower" group per set of machines / per machine (highpower_serverA) and add users to that group based on a checkout system, then define limits on the number of processes they can use, amount of memory they can use, total CPU time they can use, etc in limits.conf based on being in that group or not being in that group.
I'll send you my bill tomorrow.
See the subject, this is the modern Linux component that deals with the area you're trying to set policy around.
Have these problems actually been happening a lot?
When I first started to help manage a computer lab, I was concerned users would behave really badly and do horrible things. The truth is, very few users did, and we just talked to those users and told them how to behave.
If you get the occasional repeatedly defiant user, locking out their account can be the final solution. But most people (at least at our site) aren't jerks and listen. Most "bad things" are due more to incompetence than malice, and educating students is easy.
Also, as someone with experience in these matters, allow me to recommend AGAINST Fedora for production systems. I like to call Fedora the self-breaking distro; updates break things CONSTANTLY. You're much better off running Ubuntu (even non-LTS is more stable than Fedora) or the RHEL clones like CentOS or Scientific Linux.
Don't try to find a solution for a problem that doesn't exist.
If the users need more power, you buy more power, you don't limit them.
If you're giving your users access to the machines, they should be able to use them. And if you can't trust them to use them responsibly, don't give them access.
If it were me, I'd secure the boxes normally, set up some resource usage rules (guidelines?) and see what happens. If problems happen often, then maybe look into something automated to enforce the rules, but if not, then you're done.
As for renicing stuff done by remote users, I'm not sure this is a good idea, but if you want to do it you can renice sshd itself, and to be thorough you can also renice crond (if you give them access to cron/at.) But do keep in mind that nice (and ionice) can't do magic with an overloaded system -- they help, but they don't do magic.
As for commercial systems, I haven't really seen this as being a big problem outside academia. Multiuser *nix systems where different people are competing for resources is kind of rare in the commercial sector, as it seems like the trends lately are to have enough hardware, often dedicated, and to enforce limits through voluntary compliance (and have their boss talk to them if it's still a problem.)
That "have their boss talk to them" bit may not work so well for students, but still, I would wait for a problem to appear before I put too much effort into solving it.
Instead, put your efforts into proper sysadmin stuff -- stay up to date on patches, look for problems (especially security ones), make sure backups work, help users with problems, etc. If there's any troublemakers, talk to them, and if they don't shape up after a few warnings, kick them out. (And make sure the policies permit that!)
You can enforce limits on specific users through pam and sshd_config and some other mechanisms, but I'd suggest leaving that for later. Anything you do that will limit what people can do will eventually keep them from doing what they legitimately need to be doing.
$ man limits.conf
That sounds like a lot of overhead for a problem that seems unlikely. I've used lots of multi-user linux boxes over the years and never noticed that a few bad users ruined the experience for everybody else. If it's really an issue, think of it instead as a learning opportunity - post concise instructions on proper lab utilization and how to use top, etc to check if somebody else is the reason why the machine you are using is slow. Then let users police each other.
If you don't strictly need to enforce but monitoring (and occasional in-person admonisment) is enough, then you can start with system accounting and perhaps user class limiting on single machines and aggregate the statistics for your perusal. FreeBSD, for one, comes with both standard available.
Just go old school. Disable remote access entirely. Only have 1 machine for every 10 users, and make them show up and camp at a desk while they do their work. Nobody EVER wasted compute resources when I was in school, because odds were good you spent 3-6 hours napping on the lab floor while you waited for a machine to free up. If anyone was wasting time/resources, odds are good they'd have been found lynched and dangling from the rafters in the morning.
Also, it was uphill, BOTH ways, barefoot and in the snow.
If your users can't play nice together, the solution isn't to treat the place like a prison with automated systems enforcing a hard and fast set of rules.
The solution is for users to create their own enforcement. If some guy tries to take all the resources across your network with distcc, then the people affected should be able to notice that and tell the guy to knock that the fuck off.
In other words, give the users the freedom to break stuff, but also the knowledge to find out who'd breaking their stuff. It'll serve them far better than creating a walled garden where someone else has the responsibility to enforce social rules.
Slashdot and reddit work this way. Neither go around trying to enforce how people behave, they give the users the power to do that themself.
AccountKiller
Giving people limited logins to the powerful stuff means that some who need them will run out and others who have zero reason to have it will hog it. You let them manage themselves, the powerful machines should require you to get up and give them access, you then put monitors on them, if a less powerful machine would suffice then you boot them off and send them to it. If you really want to have some chrome then add a usage histogram maker on a user basis, machine basis, program basis etc. That way you can justify whatever you want with those numbers.
I would write my own with LDAP and some custom code that will manage ulimit and other tools to manage resources. It's a piece of cake.
Back when I worked in schools, one of our techs setup LTSP with NFS-mounted homedirs.
I mentioned that perhaps IP-based host authorization wasn't exactly a secure way of doing things, especially when it applied to both students and teachers/admin-staff.
I was told that it wouldn't be an issue, and that files were perfectly safe.
So some time goes by and a demo is scheduled for the system. My compatriot logs in and... he gets a hot-pink desktop with My Little Pony wallpaper theme. Unfortunately that didn't dissuade him from going with NFS, and they rolled it out anyways: "kids will never figure that out"
One thing that shouldn't be underestimated is the ability of a user (especially a young user) with *lots* of free time on his/her hands to figure out ways to game the system...
Easy solution:
Put all of your systems in to one big active/active server cluster. Then everyone is sharing all the resources evenly by default.
Here is a Fedora resource:
http://clusterlabs.org/doc/en-...
If you really want to have some fun you should try to create a Plan9 cluster. This is a transparent cluster OS that was designed for the purpose of resource sharing.
http://plan9.bell-labs.com/pla...
Put the user's mobile phone number into an ldap field that is easily accessible for any user (id username). Then let people sort those organizational things out themselves. If it's public who causes a problem, then people become less egoistic. The distcc guy may nice its processes himself etc.
NX? But you are using x2go? THAT is not NX. Contact the experts I.E NoMachine http://nomachine.com/. Only the real authors of probably the most amazing remote access and management tool can you help you there.
Since you are on Fedora already, I'd recommend FreeIPA. It'll give you more than your LDAP+PAM for centralized authentication and authorization, like Host-based Access Control, centralized sudoers policy, DNS, etc.
However, it wouldn't accomplish any of the tasks you specifically asked for out-of-the-box. I was thinking you could write some of these tasks as FreeIPA plugins.
"Love heals scars love left." -- Henry Rollins
It's as if I hear a thousand university students, seeing this post and the responses, all logging in to run the most resource intensive things they can find just to be pricks. The server was silenced.
I mean, if I had limits on how many systems I could connect to and use at once I would never have passed two of my courses.
One was a neural networking course which involved programming a computational model and then running 100,000 iterations of the model and analyzing the results. We had been given 6 weeks for it because it was going to take at least 1 week or so to run, but I could not get my model to work for the life of me, and working with the professor finally got it working the night before the results were due. He looked at me and said, I should ask for an extension, and I looked at him and said, I think I'm fine. He then gave me the are you nuts look.
I wrote a script that split up the iterations and output between approx 350 systems with the more powerful ones getting higher iteration counts than the older ones. When I handed in my report of the results to the professor he looked at them and was in disbelief that I was able to do it in time and asked me over when class was done, where I showed him all the code that spread the workload across all the systems (it helped that I was working fulltime managing a small beowulf cluster at that point so I had some experience using distributed computing resources). I got quite a few extra credit points for ingenuity on that...
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
What about scalable cloud instances that students pay for out of their tuition fees? That way if they want to use 32GB of ram and 12 cores for their hello world.c program, they can do so without affecting other users, but they have to pay?
This Sig does not Exist.
As much as I would like to see Linux displace Windows in these kinds of environments, there really aren't any systems that give you the same kind of management functionality as Userlock, or even Active Directory and Group Policy. It's possible of course, but only if you have the time, skill, and manpower to rig something together yourself. I'm sure I'll get flamed for saying this, but the Linux desktop has a long way to go before it can even hope to be a viable alternative to Windows in the enterprise. Even then it will not be possible unless that particular segment unifies around a specific distro. Not saying I like it; just being realistic. It certainly doesn't stop me from going all Linux at home, but it makes it an unthinkable idea to try and sell to management. Hopefully this changes in the future, but it's a long way down the road.
When I used to work for a university (mid-1990s), our department's sysdmin had gotten in trouble at the engineering school because he had written a script that would log into every machine multiple times until all ttys were exhausted ... so he could run his ray-tracing jobs undisturbed. I heard he got away with it for quite some time before one of their sysadmins came in early and realized something wasn't right.
They told him not to do it, but instead of banning him, they put him to work ... he wrote some pretty impressive software to make it easier for us to manage users, and a menu system for the non-technical users (a gopher-like interface that'd run elm / pine / news / lynx / gopher / etc.)
Build it, and they will come^Hplain.
The only way I see this happening is if you totally migrate your lab to something like Amazon AWS/EC2, and link each user to an individual account with specific bandwidth and storage (GRATIS) quotas.
For one, processing power wont be an issue since that's on Amazon's side, and it's virtually unlimited. Now, everyone will have a decent amount of the other resources for whatever they need, as long as quotas stay inside each user's scope (for which their free quota should have been well defined).
A user abuses his quota? No problem - Students get overcharged on their tuition fees, or reflected on their grades. Same for employees/researchers, in their salaries OR performance reviews. Is it a public community lab like, say, a library? Restrict access based on fair usage, maintaining an external log of who is where and when. Hell, if anything like this is politically unfeasible, just warn your users, at least you will know individually who is doing what without the heavy-lifting that is required to analyze it manually.
Everybody will be self-educated on how to use the system. On the long-run, the community will educate itself with no need for personal bad experiences. Much like using a printing quota, or the water/electricity bill.
All resources are very similar when it comes to management, so the principle of fair-use with retroactive consequence will always be the best bet.
We had a similar issue with our engineers. We had login servers which worked great as they were poorly advertised and woefully underused, but once we had a system in place for them to make efficient use of them, they started to randomly crash. Most times it was due to them trying to submit a job to our compute farm and end up running it on the login servers, but sometimes it was malicious and a deliberate attempt to get a few extra CPU cycles at the expense of others. For us, the solution was rolling our own virtual desktop farm. We used KVM for the hypervisor, python for the back end control, and php for the front end web interface. We used Active Directory for authentication and rights management. That way we could control precisely how much resources each engineer had rights to.
As you are working at a school, it is not without reason to believe that you can use the students to help develop a system to manage the virtual instances. With a bit of forethought and a limit to the specifications, you can have a simple VDI broker developed and tested in a month. And if you avoid my mistake and use the libvirt API, you will even have the ability to easily expand the system to using linux containers.
Feed the need: Digitaladdiction.net
http://en.wikipedia.org/wiki/Cgroups
To paraphrase Syndrome: When everyone's impacted by everyone's compile, no-one is.
Also, find me something other than a full kernel compile that takes measurable amounts of time on a real machine.
I want to delete my account but Slashdot doesn't allow it.
I'm amazed at how much effort is placed on limiting researches use of computers.
Post a short, general list of rules in several obvious places. Make them reasonable enough to cover most possible user needs but flexible enough to cover things that you haven't thought of yet. Any user who is stupid enough to break the rules by running fork bombs, torrents, mining, hiding stashes of lemur porn or anything else which a child of six could tell you was a bad idea, will have their accounts disabled as soon as they are discovered.
If they have a good excuse for abusing the systems then discuss it with them, suggest alternatives to running rendering jobs on the lab servers and keeping passwords on sticky notes or whatever else it is that they are doing wrong and then restore their access, trusting that they will know better. If you do it right, they may even decide that it is better to ask for permission than forgiveness next time.
If they don't, send a memo to their department head briefly outlining what they did, how it was detected, what action you have taken, and that you won't be reversing this decision until you see a presidential pardon come down from an appropriately high authority. It doesn't matter if they have Really Important Work which needs to be done by the end of the week or not, just cut them off until the proper User Apology and Restoration procedure has been completed.
There you go. This solution is licensed under the WTFPL which is compatible with the Open Source Definition and the Debian Free Software Guidelines so you can use it any way you want. You can even supply your own LART and display it prominently by the door of your office if that helps get the message across.
The 1970's called, they want their userspace problems back:
http://www.cmu.edu/computing/c...
An enigma, wrapped in a riddle, shrouded in bacon and cheese
cgroups (abbreviated from control groups) is a Linux kernel feature to limit, account, and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups.
Use Condor, it will do what you want after you configure it properly:
http://en.wikipedia.org/wiki/Condor_cycle_scavenger
...it's called Windows...and is easy to do there. Should try it sometime.
Buy one beefy pc and a ton of these - http://www.ncomputing.com/products/lseries/overview
Run user sessions on linux containers (docker is getting momentum, may be the right option) that you can limit on the resources that they can use, while being far more efficient than VMs for that. Just a word of caution, they aren't as secure as VMs, they may be present or future vulnerabilities that may let hostile students to break their limits and/or access the main system, as they have more surface contact with the machine kernel than proper virtualization, mixing VMs for security with containers for efficiency could be a good compromise.
1. Create a linux image (you can use Clonezilla, g4u or Ghost) that requires labusers authenticate to either LDAP, AD or something so you have their actual user details for logging and auditing. Alternatively you could boot it from the network or from CD. Another alternative is to use deep freeze.
2. Ensure that the system is checked for integrity on startup and the latest image is downloaded and applied if it doesn't match the correct version. cron a reboot that forces this if you're worried about users doing stuff and not rebooting.
3. Ensure that logs are written to a syslog log server or that you get the authlogs somewhere (who logged in where, on what ip address and when etc...).
4. Give users as much access as you need to (yes, even root). If they do anything wrong you have audit logs and because of the imaging unwanted software and programs will be removed.
You can use cgroups to limit cpu/mem/etc . This is a relatively new feature to Linux.
I think FreeIPA can address most of your needs and if you are already running Fedora then adding it to your network should be fairly trivial. FreeIPA is kind-of like an Active Directory type dealie (and it can synchronise against AD) that offers a lot of integration and control.
I did my undergrad degree on a lab not unlike this (actually Sun workstations using NIS/NFS to mount home directories - this was the 1990s). These machines were likely an 1-2 orders of magnitude less powerful than even your smallest desktop - desktops with 32MB of RAM and servers with 128-256MB. There was no resource management aside from disk quotas and the lab worked fine.
Depending on what you mean by high-usage I would have thought even modest desktop systems would be powerful enough for just about anything people get up to in a university lab (unless you mean Z800s with 192GB of RAM and somebody with an application for a machine that big). You could try goosing your smaller desktops by searching for 20-40GB SSDs to use as system disks (this should be plenty for the O/S, installed applications and swap) or upgrading the memory; SSDs like that go for peanuts on ebay.
I saw someone suggesting that the users should play nice. That'd be great... and maybe they did, 30 years ago. (We'll ignore the late 80's early 90's stealing of someone else in the lab's xterm....)
I had a user last year - an intern - like everyone, NFS-mounted home directory. It was, of course, shared with a good number of other users. He ran a job that dumped a logfile in his home directory. MANY gigs of logfile, enough to blow out the filesystem. Users were not amused. *I* was NOT AMUSED, as my home directory was on this system, and my login was screwed up, as well as my firefox bookmarks.....
My question is what order of magnitude number of users - tens? hundreds? more? If Sometimes, human to human works.
ulimit might help, too. So might putting the abusers' home directories on the same filesystem, and let them duke it out....
mark