Slashdot Mirror
New Year's Resolutions For *nix SysAdmins (cyberciti.biz)
An anonymous reader writes: A new year, with old systems. It is time to break bad old habits and develop good new ones. This list talks about new years resolutions for Linux and Unix sysadmins. List includes turning on 2FA on all services, making peace with systemd, installing free SSL/TLS certificates, avoiding laptops with horrible screens or wireless whitelist in BIOS, building Linux gaming rig and more. What resolutions are on your list regarding sysadmin or IT work in 2016?
is 4k baby!
maybe i should finally do dnssec. i've been planning to do it for about 5 years.
"making peace with systemd" Might as well make peace with terrorists.
This is a problem with "old vs new".
sys V init is old. So are the old, genuine unix wizards.
SystemD is new. So is Pottering and Pals.
The divide comes from "old culture" vs "new culture." The old unix culture adores simplicity, sparseness, and adaptability. The new culture adores easiness, one-stop shopping, and cohesive wholes.
This argument will never really die. The old culture will point out the endless littany of security problems with software like systemd, simply because of the complexity and scope-creep of the project. (this increases its attack surface, and makes it attractive to malware authors and hackers.) The new culture will point out the endless littany of hair pulling and hours spent dealing with obtuse scripts and strange scripting behaviors for things they feel should be solvable with a mouseclick.
These two will never live under the same roof. Both are right, and both are wrong. There are systems and applications where sysVinit makes sense, and is desirable. There are systems and applications where systemd makes sense, and is desirable.
The real sin here is not heresey by either group.
The real sin is the decision of the foss community to pick a side, and in so doing, remove that choice from other people, by choosing to make systemd a hard requirement, solely for their own convenience.
systemd was thrust upon everyone and you want us to accept it just because? how about this: document the code, document the interfaces, solidify the ABI, make the code portable, do a security audit instead of trying to force me to use systemd!
my resolution is to uproot systemd's tendrils from an otherwise decent operating system.
Anons need not reply. Questions end with a question mark.
I'm sort of right behind you. I can understand SystemD for creating the tools to make a good desktop. However, tools that make a good desktop do not necessarily make a good server. That's where I see the problem.
I run Linux Mint on my desktop and as long as it works, I don't really care what's under the hood. However, I also maintain servers where I care very much what happens under the hood. For example, I care about being able to read plain text logs and I don't understand what possible reason there could be for storing logs in a binary format. I also don't see the need to have a super complex system configuring my network interface simply because in a typical server environment your IP address doesn't change often, if at all.
I'm definitely looking at FreeBSD again (it's been 15 years since I touched any BSD outside of monowall/pfsense) and am using it for a new project at work that's currently in Alpha testing phase. I'm doing it because it's more in line with my KISS preferences but I will admit that I miss iproute2.
some karma... and kinda lukewarm about it.
This is a problem with "old vs new".
sys V init is old. So are the old, genuine unix wizards.
SystemD is new. So is Pottering and Pals.
The divide comes from "old culture" vs "new culture." The old unix culture adores simplicity, sparseness, and adaptability.
The "old culture" knows that a server is just part of a bigger process, and reliability and maintainability are more important than simplicity, sparseness (whatever that means in this context), and adaptability. Without something like systemd, Linux cannot be enterprise ready. "Rolling your own" scripts for failover and redundancy is the worst idea when more than one admin has to diagnose problems at 2:00 AM. You want something supported by the vender, with standard configuration options, that can be easily understood by everyone on the team. Sometimes the "most elegant" solution is not the best for the business.
The only thing worse than a Democrat is a Republican.
I don't buy that init scripts are simple or even particularly adaptable. I've hand crafted a few in my time and they are always a pain in the arse to manage. When you get to the point of having multiple scripts bring called which then call sub scripts and all of them somewhat unique to that machine... It's time to look for something better.
I have not played with systemd extensively, but it seems to have the right idea. While in theory scripts give you more control, in practice they just make it harder to admin the system and make you waste time hacking them. Having it all tied together and controllable from one place is much better.
I used to have a highly customised Firefox install. Eventually Firefox sucked enough to drive me to Chrome. At that point I realised it was better to just adapt my workflow to something better some times, instead of proclaiming it sucks because it doesn't do things the way I did a decade ago.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
I don't buy that init scripts are simple or even particularly adaptable. I've hand crafted a few in my time and they are always a pain in the arse to manage.
If a daemon needs a complex init script to manage it, then systemd won't be able to manage it at all, because it can't do everything a script can do. If systemd can manage a daemon, then it didn't need a complex init script, and you did it wrong.
When you get to the point of having multiple scripts bring called which then call sub scripts and all of them somewhat unique to that machine... It's time to look for something better.
If you need that much complexity to run a daemon, then either your daemon sucks or you do. Unless, of course, you're just talking about simple things like script libraries. If those confuse you, perhaps Unix is not for you. Shell scripting is a core Unix feature.
If you could explain cogently what you meant, perhaps you would get different responses. But so far, you're talking bananas. Systemd's unit files are merely a handful of name-value pairs, they don't even need sections because all of the names are unique (even between sections) but they put sections into them anyway because systemd is all about obfuscation, and needless effort. Systemd's "additional functionality" (unit files, cgroups support) could be replaced by a relatively small shell script, based on existing init scripts. Indeed, any distribution which uses script libraries in its init scripts could have done this cheaply. For example, cgroups are a kernel feature manipulated via short, common commands, like creating directories.
I used to have a highly customised Firefox install. Eventually Firefox sucked enough to drive me to Chrome.
I used to use Chrome for Google services. Eventually Google fucked Chrome hard enough to break Youtube ad-blockers. Then I realized that people who use Chrome are Part Of The Problem.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
The "old culture" knows that a server is just part of a bigger process, and reliability and maintainability are more important than simplicity,
That is why Systemd is fail. Shell scripts have reliability and maintainability. Systemd complicates troubleshooting of the boot process, and requires that we trust code written by a known lame. Remember pulseaudio? I sure do. I spent hours fighting with it.
Without something like systemd, Linux cannot be enterprise ready.
Why?
"Rolling your own" scripts for failover and redundancy is the worst idea when more than one admin has to diagnose problems at 2:00 AM.
Only if your goal is to hire sysadmins too stupid to comprehend shell scripts. The problem is that people that dumb are also too stupid to troubleshoot problems. Try hiring some of the many out-of-work qualified sysadmins who actually know Unix, instead of an H1-B or some kid just out of school.
You want something supported by the vender, with standard configuration options, that can be easily understood by everyone on the team.
And that is how vendors have been using Unix since forever. With shell scripts, supported by the vendor, with standard control flow, that can be easily understood by anyone intelligent enough to be a systems administrator to begin with. Your complaint is that the point and click Microsoft-or-Apple-fanboy-wannabe-sysadmin cannot point and grunt his way through poorly implementing Unix configuration. No one who knows what they are doing will have sympathy for that idea, because there are so very many qualified professionals going unemployed right now. It's not hard to get people who know what they are doing, if you try to hire them. You don't want to do that. You want to hire idiots and you want magically good results. But someone incapable of systems administration without systemd will simply fall flat on their face when a problem crops up, and because they're using systemd it will be harder to troubleshoot the problem.
TL;DR: A monkey who can't understand a shell script can't be a real sysadmin anyway. All they can be is a button puncher. They should work in a NOC, and you should hire someone who knows Unix to administer Unix.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Except of course, that the old strategy doesn't work without constantly having sysadmins grooming and spoiling their pets (servers), something they seldom do other than as an afterthought after a long and hard failure. Heck, at some companies I've been to, nobody can reboot any servers without all the admins being alert and starting services manually.
If we are to compete with cloud solutions, we need to treat servers and services like cattle. Slaughter the ones that have problems, diagnose later and bring up new ones to replace bad parts of the whole system. This requires something like systemd on the infrastructure/platform side, and then it makes sense to have similar strategies on each server also - to support hot-swappable parts and a rapidly changing platform. To do this manually with scripts is really not feasible, as it doesn't work in practice and you're often dealing with race conditions that always can bring new surprises. The next OS update might change everything again as well, and does not support your assumptions, prayers and hopes.
What we need to support is private cloud solutions, or we will be out-competed by more efficient public cloud solutions. For this to happen, some of the grumpy old grey beards need to retire. You have planned for that, right? No, I think most of the old-timers have planned for nothing but their own benefits, and deserve absolutely no sympathy in this regard. Fuck you, for making the workplace a miserable place to be at!
Captcha: nestling
If a daemon needs a complex init script to manage it,
Define complex. 99% of init scripts are handling of the state of the daemon, PID files, configuration etc. Or to put it another way: Every daemon on my system can be started with a single line in the console, why do I need a script to manage it at all?
You are right. Init scripts don't need to be complex. Yet on every system they are still longer than 1 line, actually they are typically longer than 100 lines. I've written systemd scripts (or whatever they are called). I've written upstart ones. I've copied and pasted and modified a sysv init script and would have probably cut myself if I were forced into a position to write one from scratch.
Eventually Google fucked Chrome hard enough to break Youtube ad-blockers.
That is news to me. I haven't seen a youtube ad on my computer in many years.
This has been this way for a long time. Redhat maintainers actually wrote a long post about "choice" a decade ago. Their efforts for maintaining a distribution go up exponentially far more than simply maintaining 2 packages with 2 startup scripts if they give the user choice. Their summary was you have choice as far as what distribution you use. Don't like what is being provided, then fork or move to another distribution.
Quite frankly given the poor state of many distributions with regards to unresolved bugs and very slow moving adoptions of features I fully support them not wasting time on trying to please everyone.
For example, I care about being able to read plain text logs and I don't understand what possible reason there could be for storing logs in a binary format.
ForwardToSyslog=yes in journald.conf
Everytime I see the complain about journalling in systemd I cry a little. The only thing journald does is allow logging earlier in the boot process before a syslog daemon has even started and give you options. One of those options include outputting everything back to your syslog daemon of choice if you're so deadset against having easy to browse journal entries which provide far more detail than the traditional syslog.
I also don't see the need to have a super complex system configuring my network interface simply because in a typical server environment your IP address doesn't change often,
You mean in *your* typical server environment. Systemd provides a lot of functionality that can expressly help in server environments such as actually intelligently knowing the state of the daemon not just relying on some PID file that may or may not accurately represent the state of the daemon, the ability to restart the sudden death of a daemon (configurable of course so you don't end up with a restart loop), and the above mentioned logging options which make it so much easier to track a misbehaving part of the system by cutting through the noise of logs.
What we need to support is private cloud solutions,
No, junior. That's called a cluster. Cloud computing is when you spin up instances as you need them, and get billed appropriately. We don't need systemd for clusters. They predate it dramatically. And we're already using virtualization, which also doesn't require systemd. If there's anything else old that's new again that you would like explained for you using only short words, let me know. I'm here for you.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
When I read that, I hear the voice from the "mongo db is webscale" cartoons.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
I've been using unix (ie. only Linux but we'll pretend that counts) for over 15 years now. Not quite the "old" you may think of but old enough.
I gave systemd a try. It actively fought me and I cannot accept that. It has too much of a "my way or the highway" mentality that you just can't fix without major C hacking and recompiling. If you don't like its way of doing things then too bad.
sysvinit scripts may be slower to boot and have fewer automatic/behinds-the-scenes features you want, but I can make any arbitrary change to them with minimal effort. I can run them with line-by-line tracing using "set -x" and find out exactly why it's hanging. I can rescue it with *any* install media even if it doesn't have systemd and '/etc/init.d/servicename start' will actually work.
systemd is fine for desktops run by people who think Firefox is the only app they really need to Surf The Net. sysvinit is designed for people who want control of their systems and want to be able to inspect what it's doing. And I'm sorry, I NEED the latter to do my job properly.
* Use tools such as Lets Encrypt on all websites (good idea in theory, in practice not feasible due to 3 month certs)
Take it as a good incentive to learn automation.
As an example for the problems with troubleshooting, I've recently installed a few Ubuntu 15.10 systems (bare metal and in VMware VMs). In both scenarios, NFS filesystems fail to be mounted at boot. Why? I have no bloody clue. There's nothing in the logs. If I umount them (they are showing as mounted but give immediate I/O error on use) and remount by hand it all works just fine. No idea how to troubleshoot it. With sysvinit I could boot to an interactive shell in the late initramfs and step through every script by hand, and pinpoint exactly where something was failing. With systemd, even with correctly configured systems, I still experience occasional unrecoverable hangs at boot as it screws up its nondeterministic boot ordering and waits forever on something or other (who knows what it might be). systemd has absolutely not improved boot reliability. If I boot a FreeBSD system with exactly the same NFS configuration, it mounts everything perfectly at boot every damned time.
Another thing, is the broken compatibility with what came before. Example: I edit /etc/nfsidmap.conf to configure NFSv4 id mapping. Previously I'd reload/restart the nfs-common service with 'service' or 'invoke-rc.d'. Job done. What's the systemd equivalent? I don't know, and I can't see any obvious candidate. Since both these commands now forward to systemd, a diligent engineer would have made the old service names forward to the new target(s) to retain compatibility, maybe with a helpful message indicating what the new names were. There's nothing. I don't see any obvious nfsidmap or generic rpc or nfs units. Yes, it's partly me lacking familiarity with the new way of doing things; but the lack of care in having a smooth transition for admins makes for a terrible time, and the mess of units makes the whole thing difficult to comprehend as a whole and baroquely overcomplicated.
The real sin is the decision of the foss community to pick a side, and in so doing, remove that choice from other people, by choosing to make systemd a hard requirement, solely for their own convenience.
Did you just try to make writing software to scratch your own itch sound like a bad thing? If Poettering wants to write systemd-only software, nobody's going to stop him. Nobody should be able to stop him. Nobody's going to force him to create or maintain SysV init scripts either. And if other projects find that depending on systemd is convenient, so can they. Open source is not a democracy. Developers do what developers want, regardless of what their users think and by users I mean everybody from other projects to distro packagers to system administrators to end users. The conflict was lost when systemd was voted in as the default, trying to bend the Debian system to force developers to preserve the old ways was a fool's errand. If it had passed, all that would have happened is that Debian would have lost them. Nobody can make that kind of committee decision stick.
I am a little suprised that someone with your UUID would have such a philosophy about the Debian project, but fair enough. I'd like to refer you to the Debian Manifesto, a document that was written when the Debian project was first founded. In particular, I'd like to refer you to section A.3:
thus, a distribution is created based on the needs and wants of the users rather than the needs and wants of the constructor.
, with "constructor" referring to the creator.
To be very clear, I am not advocating that Poettering shouldn't be programming what he pleases, nor that all work should be productive. However, I am saying that at least for the Debian project, the focus (used to be) on the users, and making a solid operating system useful to everyone. There's been a large resurgence in the "developers first" attitude the last couple years, and while it lends itself well to hobbyists, these kinds of people should not be working on a large and collaborative project with the goal of being useful to everyone - because, as you've made very clear, the center of that philosophy is all about your wants and needs, something which is directly contrary to the main goal of the Debian Project (and really, any large open source project whose goal is to be useful to more than the developers).
"Set a man a fire, he'll be warm for the rest of the night. Set a man afire, he'll be warm for the rest of his life."
Without something like systemd, Linux cannot be enterprise ready.
Why?
"Rolling your own" scripts for failover and redundancy is the worst idea when more than one admin has to diagnose problems at 2:00 AM.
Only if your goal is to hire sysadmins too stupid to comprehend shell scripts.
And there you've just answered your own question as to "why?" Not that their goal is to hire stupid sysadmins, but their goal is, to some extent, to avoid relying on people being particularly competent.
I see this as part of the disconnect between techies and business/manager types, and you're only illustrating it. For those who are pretty good at the tech stuff, they think, "That's ok, I'll just write a custom script, and if something breaks I'll figure it out." Meanwhile, the business guy has been burned by that too many times. For the handful of competent techs he's hired, he's also hired a couple incompetent guys, as well as some who were fairly competent but not as smart as they thought they were. He's seen a tech write that custom script, he's seen it work well enough for a while, and then seen it turn into a disaster as soon as someone else had to touch it.
It's hard to run a business relying on everyone to be a bunch of super-hero super-geniuses. Businesses want a solution that's drop-dead simple. They want things to be used in the way they're supposed to, and to be fully supported. They don't like customization unless it's something that was designed to be customizable, and supported in that customization. They want it all to be documented and clear, and they don't like relying on people to be good at their jobs, because no matter how hard you try to hire good people and how much you're willing to pay them, most people turn out not to be particularly good at their jobs.
I'm not saying these are unbreakable rules, but there is a sort of tendency. The point is, larger and established businesses don't really want a clever hack that saves them a few thousands of dollars at the risk of a failure that would cost millions. Whether systemd is relevant to that issue-- I don't want to get into that discussion. But you're asking why you need something "the point and click Microsoft-or-Apple-fanboy-wannabe-sysadmin" can figure out in order to be considered by some people to be "Enterprise ready", then that's why.
The most vocal opponents of systemd (whom you mislabel as "systemd trolls") tend to be professional system administrators and professional software developers with decades of experience.
Many of them have been responsible for hundreds of thousands of production servers and desktops. They've used more operating systems than you could probably count. They've dealt with nearly as many different init systems.
These decades of experience has taught them a thing or two. They've come to learn what works, and what doesn't.
Well, it turns out that systemd embodies just about everything that's known not to work well! It's rife with architectural problems, for example. Binary logging is a big no-no. Trying to do absolutely everything, and doing it all very poorly, is a big no-no. Not following the proven UNIX philosophy is a big no-no. These aren't just minor bugs that can be fixed with some code changes. These are deep flaws that require systemd to be thrown away.
It's no wonder we see Linux distribution mailing lists and bug trackers filled with so many reports of problems caused by systemd. Systemd has thrown out decades of accumulated knowledge and experience, and replaced it with approaches that are known to not work.
Systemd has caused a lot of inexcusable problems for a lot of its victims. We're talking about things as serious as Linux installations that don't boot properly. That's one of the worst things that can happen to a Linux installation; it renders it unusable!
Instead of labeling these professionals as "systemd trolls", you should be thankful that they've done the right thing and started moving away from Linux. Many of them have, or are in the process of, converting the systems they're responsible for from Linux to FreeBSD or OpenBSD. They just can't let their critical systems fall victim to systemd and the problems it has been shown to cause.
New Year's Resolutions For *nix SysAdmins
After 30 years as an admin and systems programmer, finally find out what that damn asterisk stands for.
It must have been something you assimilated. . . .
No competent administrator would run something as arcane, unreliable, and fragile as systemd on a server, given any sort of choice.
Goals for 2016?: remove those last few linux boxen and migrate the services to *BSD (Open is my choice, but it does have some lag on drivers; have to brush up on writing those, I guess).
NFS filesystems fail to be mounted at boot. Why? I have no bloody clue. There's nothing in the logs.
Unlikely. If there was an error running the mount command, it was definitely recorded in the log. Did you use journalctl -u nfs or journalctl -b?
With sysvinit I could boot to an interactive shell in the late initramfs and step through every script by hand, and pinpoint exactly where something was failing
You can get a debug shell in systemd. The process is just a little bit different,
http://freedesktop.org/wiki/So...
With systemd, even with correctly configured systems, I still experience occasional unrecoverable hangs at boot as it screws up its nondeterministic boot ordering and waits forever on something or other (who knows what it might be).
I'm sorry, but that's bulls**t. If your system is randomly hanging, then it's not configured correctly. While the boot sequence of systemd can be characterized as non-deterministic, it is not random. If services are hanging because dependencies have not been met, then you need to specify your dependencies properly.
Another thing, is the broken compatibility with what came before. Example: I edit /etc/nfsidmap.conf to configure NFSv4 id mapping. Previously I'd reload/restart the nfs-common service with 'service' or 'invoke-rc.d'. Job done. What's the systemd equivalent?
Since you should be using a >3.5 kernel, nothing. The rpc idmapper has been replaced with the nfsidmap system call in the kernel. So there is no need to start/restart an idmapper service.
Yes, it's partly me lacking familiarity with the new way of doing things;
Understatement. Have you read any of the systemd docs or transition/setup guides available? There are a ton. Just do it.
systemd notices that the log is corrupt, and... deletes it
That's not what happens.
See: the long and angry (and unfixed) bugzilla tickets.
Which one? The one that says journald will rotate the log on detection of corruption, thereby preserving it in an un tampered state while still allowing the uncorrupted portion to be read?
Basically, you need to be able to read the logs with the most minimal of tools, because you are going to be diagnosing it in a downed state most likely-- You cannot bank on having a full suite of binary manipulation tools on hand. You will be lucky if you have more than vi.
This is the biggest load of canned bullsh**t argument. Where do you guys come up with this crap? We don't work on 1970s mainframes anymore. Why would vi be the only thing available? Why would an emergency shell not have the journalctl utility? Why would there not be serial logging capability? Why would you not be able to boot with a rescue disk (in case all other things fail)? The answer is, there is no reason. It is a made up scenario that doesn't exist.
Also, text based logs compress REAALLY well for long term storage for audit purposes! Binary logs? Probably not so much.
More bulls**t. There is no reason to think the journal won't compress well. Being binary, or not, has little if anything to do with compressibility. Remember that ASCII is itself a binary format! Also, auditing is one of the primary strengths of journald. It is designed specifically with auditing in mind.
if the binary logs are in some stupid "easily damaged" format, then having a process suddenly die horribly from abnormal termination will result in corrupt logs
That makes no sense at all. A process dying does not result in a corrupt log. There would be an error message, or at the very least a sigterm message, printed in the log just as it would be on the console if it had been run from an interactive terminal.
Hell, the file chain can be damaged from FS corruption, and parts of the log will still be readable.
The journal is designed to preserve as much of the log as possible when corruption happens. Anything written before or after the corruption event will be accessible by journalctl.
Lift the hood on systemd sometime.
I'm not advocating for systemd. Just pointing out why "enterprise ready" to some extent can mean, "doesn't require any special genius to operate".
Translation, the business guy has hired cheap monkeys who turned out not to be able to handle any difficulty.
That's a really dumb translation, and shows you missed the whole point. You might get how the tech works, but you have no idea how to run things if you think the solution is to hire more expensive people. Sometimes you hire very expensive monkeys on the idea that they're brilliant and "worth it", and then they still make a mess of things. If you've actually had to do hiring, you probably know that it's very difficult to find good people, and "expensive" doesn't always mean "good".
And really, it doesn't make sense to always hire very experienced people. To keep things running well, you need a mix. You get some senior people to do the difficult stuff, and then you get some junior people to handle the stuff that no senior person is going to want to waste their time managing. And then you train the junior people as you go, which means at some point, you're going to have to hand off some of the senior work to the junior people, even if they're not 100% "ready", because sometimes that's how you learn.
It may be true that anyone can captain a ship in calm seas, but part of the goal of a business is to engineer the seas to be calm. A lot of the "old salts" get off on being the guy who heroically and astoundingly manages to patch up the Titanic before it goes down, but the business-oriented guy just wants to steer around the icebergs.