Slashdot Mirror
New Year's Resolutions For *nix SysAdmins (cyberciti.biz)
An anonymous reader writes: A new year, with old systems. It is time to break bad old habits and develop good new ones. This list talks about new years resolutions for Linux and Unix sysadmins. List includes turning on 2FA on all services, making peace with systemd, installing free SSL/TLS certificates, avoiding laptops with horrible screens or wireless whitelist in BIOS, building Linux gaming rig and more. What resolutions are on your list regarding sysadmin or IT work in 2016?
This is a problem with "old vs new".
sys V init is old. So are the old, genuine unix wizards.
SystemD is new. So is Pottering and Pals.
The divide comes from "old culture" vs "new culture." The old unix culture adores simplicity, sparseness, and adaptability. The new culture adores easiness, one-stop shopping, and cohesive wholes.
This argument will never really die. The old culture will point out the endless littany of security problems with software like systemd, simply because of the complexity and scope-creep of the project. (this increases its attack surface, and makes it attractive to malware authors and hackers.) The new culture will point out the endless littany of hair pulling and hours spent dealing with obtuse scripts and strange scripting behaviors for things they feel should be solvable with a mouseclick.
These two will never live under the same roof. Both are right, and both are wrong. There are systems and applications where sysVinit makes sense, and is desirable. There are systems and applications where systemd makes sense, and is desirable.
The real sin here is not heresey by either group.
The real sin is the decision of the foss community to pick a side, and in so doing, remove that choice from other people, by choosing to make systemd a hard requirement, solely for their own convenience.
systemd was thrust upon everyone and you want us to accept it just because? how about this: document the code, document the interfaces, solidify the ABI, make the code portable, do a security audit instead of trying to force me to use systemd!
my resolution is to uproot systemd's tendrils from an otherwise decent operating system.
Anons need not reply. Questions end with a question mark.
I don't buy that init scripts are simple or even particularly adaptable. I've hand crafted a few in my time and they are always a pain in the arse to manage.
If a daemon needs a complex init script to manage it, then systemd won't be able to manage it at all, because it can't do everything a script can do. If systemd can manage a daemon, then it didn't need a complex init script, and you did it wrong.
When you get to the point of having multiple scripts bring called which then call sub scripts and all of them somewhat unique to that machine... It's time to look for something better.
If you need that much complexity to run a daemon, then either your daemon sucks or you do. Unless, of course, you're just talking about simple things like script libraries. If those confuse you, perhaps Unix is not for you. Shell scripting is a core Unix feature.
If you could explain cogently what you meant, perhaps you would get different responses. But so far, you're talking bananas. Systemd's unit files are merely a handful of name-value pairs, they don't even need sections because all of the names are unique (even between sections) but they put sections into them anyway because systemd is all about obfuscation, and needless effort. Systemd's "additional functionality" (unit files, cgroups support) could be replaced by a relatively small shell script, based on existing init scripts. Indeed, any distribution which uses script libraries in its init scripts could have done this cheaply. For example, cgroups are a kernel feature manipulated via short, common commands, like creating directories.
I used to have a highly customised Firefox install. Eventually Firefox sucked enough to drive me to Chrome.
I used to use Chrome for Google services. Eventually Google fucked Chrome hard enough to break Youtube ad-blockers. Then I realized that people who use Chrome are Part Of The Problem.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
The "old culture" knows that a server is just part of a bigger process, and reliability and maintainability are more important than simplicity,
That is why Systemd is fail. Shell scripts have reliability and maintainability. Systemd complicates troubleshooting of the boot process, and requires that we trust code written by a known lame. Remember pulseaudio? I sure do. I spent hours fighting with it.
Without something like systemd, Linux cannot be enterprise ready.
Why?
"Rolling your own" scripts for failover and redundancy is the worst idea when more than one admin has to diagnose problems at 2:00 AM.
Only if your goal is to hire sysadmins too stupid to comprehend shell scripts. The problem is that people that dumb are also too stupid to troubleshoot problems. Try hiring some of the many out-of-work qualified sysadmins who actually know Unix, instead of an H1-B or some kid just out of school.
You want something supported by the vender, with standard configuration options, that can be easily understood by everyone on the team.
And that is how vendors have been using Unix since forever. With shell scripts, supported by the vendor, with standard control flow, that can be easily understood by anyone intelligent enough to be a systems administrator to begin with. Your complaint is that the point and click Microsoft-or-Apple-fanboy-wannabe-sysadmin cannot point and grunt his way through poorly implementing Unix configuration. No one who knows what they are doing will have sympathy for that idea, because there are so very many qualified professionals going unemployed right now. It's not hard to get people who know what they are doing, if you try to hire them. You don't want to do that. You want to hire idiots and you want magically good results. But someone incapable of systems administration without systemd will simply fall flat on their face when a problem crops up, and because they're using systemd it will be harder to troubleshoot the problem.
TL;DR: A monkey who can't understand a shell script can't be a real sysadmin anyway. All they can be is a button puncher. They should work in a NOC, and you should hire someone who knows Unix to administer Unix.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Except of course, that the old strategy doesn't work without constantly having sysadmins grooming and spoiling their pets (servers), something they seldom do other than as an afterthought after a long and hard failure. Heck, at some companies I've been to, nobody can reboot any servers without all the admins being alert and starting services manually.
If we are to compete with cloud solutions, we need to treat servers and services like cattle. Slaughter the ones that have problems, diagnose later and bring up new ones to replace bad parts of the whole system. This requires something like systemd on the infrastructure/platform side, and then it makes sense to have similar strategies on each server also - to support hot-swappable parts and a rapidly changing platform. To do this manually with scripts is really not feasible, as it doesn't work in practice and you're often dealing with race conditions that always can bring new surprises. The next OS update might change everything again as well, and does not support your assumptions, prayers and hopes.
What we need to support is private cloud solutions, or we will be out-competed by more efficient public cloud solutions. For this to happen, some of the grumpy old grey beards need to retire. You have planned for that, right? No, I think most of the old-timers have planned for nothing but their own benefits, and deserve absolutely no sympathy in this regard. Fuck you, for making the workplace a miserable place to be at!
Captcha: nestling
If a daemon needs a complex init script to manage it,
Define complex. 99% of init scripts are handling of the state of the daemon, PID files, configuration etc. Or to put it another way: Every daemon on my system can be started with a single line in the console, why do I need a script to manage it at all?
You are right. Init scripts don't need to be complex. Yet on every system they are still longer than 1 line, actually they are typically longer than 100 lines. I've written systemd scripts (or whatever they are called). I've written upstart ones. I've copied and pasted and modified a sysv init script and would have probably cut myself if I were forced into a position to write one from scratch.
Eventually Google fucked Chrome hard enough to break Youtube ad-blockers.
That is news to me. I haven't seen a youtube ad on my computer in many years.
This has been this way for a long time. Redhat maintainers actually wrote a long post about "choice" a decade ago. Their efforts for maintaining a distribution go up exponentially far more than simply maintaining 2 packages with 2 startup scripts if they give the user choice. Their summary was you have choice as far as what distribution you use. Don't like what is being provided, then fork or move to another distribution.
Quite frankly given the poor state of many distributions with regards to unresolved bugs and very slow moving adoptions of features I fully support them not wasting time on trying to please everyone.
They're not trying to force exclusive ipv6 on you, they're trying to make you go dual stack which you absolutely should.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
What we need to support is private cloud solutions,
No, junior. That's called a cluster. Cloud computing is when you spin up instances as you need them, and get billed appropriately. We don't need systemd for clusters. They predate it dramatically. And we're already using virtualization, which also doesn't require systemd. If there's anything else old that's new again that you would like explained for you using only short words, let me know. I'm here for you.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
As an example for the problems with troubleshooting, I've recently installed a few Ubuntu 15.10 systems (bare metal and in VMware VMs). In both scenarios, NFS filesystems fail to be mounted at boot. Why? I have no bloody clue. There's nothing in the logs. If I umount them (they are showing as mounted but give immediate I/O error on use) and remount by hand it all works just fine. No idea how to troubleshoot it. With sysvinit I could boot to an interactive shell in the late initramfs and step through every script by hand, and pinpoint exactly where something was failing. With systemd, even with correctly configured systems, I still experience occasional unrecoverable hangs at boot as it screws up its nondeterministic boot ordering and waits forever on something or other (who knows what it might be). systemd has absolutely not improved boot reliability. If I boot a FreeBSD system with exactly the same NFS configuration, it mounts everything perfectly at boot every damned time.
Another thing, is the broken compatibility with what came before. Example: I edit /etc/nfsidmap.conf to configure NFSv4 id mapping. Previously I'd reload/restart the nfs-common service with 'service' or 'invoke-rc.d'. Job done. What's the systemd equivalent? I don't know, and I can't see any obvious candidate. Since both these commands now forward to systemd, a diligent engineer would have made the old service names forward to the new target(s) to retain compatibility, maybe with a helpful message indicating what the new names were. There's nothing. I don't see any obvious nfsidmap or generic rpc or nfs units. Yes, it's partly me lacking familiarity with the new way of doing things; but the lack of care in having a smooth transition for admins makes for a terrible time, and the mess of units makes the whole thing difficult to comprehend as a whole and baroquely overcomplicated.
The most vocal opponents of systemd (whom you mislabel as "systemd trolls") tend to be professional system administrators and professional software developers with decades of experience.
Many of them have been responsible for hundreds of thousands of production servers and desktops. They've used more operating systems than you could probably count. They've dealt with nearly as many different init systems.
These decades of experience has taught them a thing or two. They've come to learn what works, and what doesn't.
Well, it turns out that systemd embodies just about everything that's known not to work well! It's rife with architectural problems, for example. Binary logging is a big no-no. Trying to do absolutely everything, and doing it all very poorly, is a big no-no. Not following the proven UNIX philosophy is a big no-no. These aren't just minor bugs that can be fixed with some code changes. These are deep flaws that require systemd to be thrown away.
It's no wonder we see Linux distribution mailing lists and bug trackers filled with so many reports of problems caused by systemd. Systemd has thrown out decades of accumulated knowledge and experience, and replaced it with approaches that are known to not work.
Systemd has caused a lot of inexcusable problems for a lot of its victims. We're talking about things as serious as Linux installations that don't boot properly. That's one of the worst things that can happen to a Linux installation; it renders it unusable!
Instead of labeling these professionals as "systemd trolls", you should be thankful that they've done the right thing and started moving away from Linux. Many of them have, or are in the process of, converting the systems they're responsible for from Linux to FreeBSD or OpenBSD. They just can't let their critical systems fall victim to systemd and the problems it has been shown to cause.
NFS filesystems fail to be mounted at boot. Why? I have no bloody clue. There's nothing in the logs.
Unlikely. If there was an error running the mount command, it was definitely recorded in the log. Did you use journalctl -u nfs or journalctl -b?
With sysvinit I could boot to an interactive shell in the late initramfs and step through every script by hand, and pinpoint exactly where something was failing
You can get a debug shell in systemd. The process is just a little bit different,
http://freedesktop.org/wiki/So...
With systemd, even with correctly configured systems, I still experience occasional unrecoverable hangs at boot as it screws up its nondeterministic boot ordering and waits forever on something or other (who knows what it might be).
I'm sorry, but that's bulls**t. If your system is randomly hanging, then it's not configured correctly. While the boot sequence of systemd can be characterized as non-deterministic, it is not random. If services are hanging because dependencies have not been met, then you need to specify your dependencies properly.
Another thing, is the broken compatibility with what came before. Example: I edit /etc/nfsidmap.conf to configure NFSv4 id mapping. Previously I'd reload/restart the nfs-common service with 'service' or 'invoke-rc.d'. Job done. What's the systemd equivalent?
Since you should be using a >3.5 kernel, nothing. The rpc idmapper has been replaced with the nfsidmap system call in the kernel. So there is no need to start/restart an idmapper service.
Yes, it's partly me lacking familiarity with the new way of doing things;
Understatement. Have you read any of the systemd docs or transition/setup guides available? There are a ton. Just do it.