A Look at Facebook's Use of Systemd (phoronix.com)

At an event this month (you can find the video of it here), Davide Cavalca, a production engineer at Facebook, spoke about the growing adoption of systemd at the data centers of the company. From a report: Facebook continues making use of systemd's many features inside their data centers. Some of their highlights for systemd use in 2018 includes: Facebook's servers have been relying on systemd for about the past two years. Facebook is using CentOS 7 everywhere from hosts to containers. While relying on CentOS 7, Facebook backports a lot of packages including new systemd releases, Meson, other dependencies, and of course new Linux kernel releases. Facebook is working on "pystemd" as a Python (Cython) wrapper on top of SD-BUS.

Comments

Ugh

[Miser]

Let Facebook HAVE systemd. They can keep it.

Re:Ugh

[Anonymous Coward]

They just had a major security leak of 50 million users? Competent you say?

Re:Ugh

[iggymanz]

pfft, they make $86-93K, they are not highly skilled admins

Re:Ugh

[Joe_Dragon]

that is crap in the Bay area.

Re:Ugh

[Tailhook]

Amazon Linux — the image used on most AWS EC2 Linux instances — has also changed over to systemd with Amazon Linux 2, released last Jun 26. Amazon Linux was an important sysvinit holdout; if your users expected to use your work on EC2 with Amazon's distro you needed to care about sysvinit. Now that Amazon Linux has moved on there are no important distributions based on sysvinit remaining and it may be safely ignored.

Re:Ugh

[Anonymous Coward]

Yeah, all the highly skilled sysadmins updated their startup scripts years ago to start under systemd. It's only the lazy weak sysadmins who have spent the last four of five years whining that systemd once broke a script somewhere.

The highly skilled admins forked Debian, to create Devuan, a Debian free of systemd.

Re:Ugh

[gweihir]

Ignoring good technology in favor of hip but bad technology is not a sign of competence.

Re: Ugh

[quadcricket]

Devuan really is a thing of beauty.

Systemd

[93 Escort Wagon]

#systemctl enable usertracking.service
#systemctl start usertracking.service

Re:systemd

[Anonymous Coward]

SYSTEMD!

Slowly I turned...step by step...inch by inch...

Then I unzip, strip, touch, finger, grep, mount, fsck, more, yes, fsck, fsck, fsck, umount, sleep.

systemd

[Anonymous Coward]

triggered

It is a floor wax it is a dessert topping

[bobstreo]

Name three things that deserve each-other:

Systemd, Bookface, Python,

Re:It is a floor wax it is a dessert topping

[gweihir]

I disagree. Python is actually pretty nice, but it requires somebody with real skill to be used competently.

MOM! DAD! DON'T TOUCH THAT!

[Thud457]

a real "Nexus of Evil" to coin a phrase...

Re:MOM! DAD! DON'T TOUCH THAT!

[bobstreo]

a real "Nexus of Evil" to coin a phrase...

I am pretty sure Alphabet/Google has a trademark on Nexus (of Evil)

Wait

[ArchieBunker]

A Python wrapper for the systemd API? So now they're going through Python to go through systemd to make some calls...?

Re:Wait

[Anonymous Coward]

Combining Python and systemd is like mixing vomit with diarrhea.

Re:Wait

[Anonymous Coward]

It's end-to-end integration!

They’re using CentOS huh?

[93 Escort Wagon]

I can understand... these startups can’t afford the license fees Red Hat charges.

systemd has logged your complaint

[jfdavis668]

If anyone can ever figure out how to read the logs, someone may respond.

Re:systemd has logged your complaint

[Anonymous Coward]

LOL. It is a pain to remember things like:

systemctl restart openvpn@dev.service

The command isn't named systemd, there's an @ sign in the name, and the dev invokes magic to find /etc/openvpn/dev.ovpn.

But the logging issue is a bigger problem. I miss SysV scripts where even if something didn't get logged, you could at least see it on the console. Having to resort to using strace just sucks.

Re:systemd has logged your complaint

[DCFusor]

Yeah, you're one of the ones who can't even, that's obvious. When this crap just gets added to your system, how are you even supposed to know there's such a thing as journalctl? Some of us, do, you know, real work on this stuff, and only do sysadmin if it's required to set something nifty up. Having to also fix things someone else broke is extra work that isn't needed. The pushback isn't for no reason - systemd really screwed up a lot of people's day. If it just worked...probably would have been another story, despite the terrible architecture.

Re:systemd has logged your complaint

[Xenolith0]

So you're mad that you might have to read a manual to a highly complex system? I'm impressed that you do "real work" on a system that you have no idea how it works!

If you don't want to bother reading manuals because you're too busy doing "real work". Why did you even bother upgrading to RHEL7? Stick with RHEL6, which is supported until 2024.

And yes, systemd is incredibly well documented. Just the man pages alone on the system:
$ man -k systemd | wc -l
147
But for online documentation: https://www.freedesktop.org/wiki/Software/systemd/

In almost every way systemd is better than sysv, upstart or other init systems, and it fixes many long standing issues SysV had.

Facebook's use?

[jeff4747]

I can't watch the video because work, but from the text this doesn't look much like Facebook is using systemd. Facebook is using an operating system that uses systemd. And may someday have a python api to do what everyone else does with bash.

Only one good use-case

[DCFusor]

for systemd - tons of identical instances, usually virtual so they can really be the same. The moment you customize with shares mounted on other machines, use wifi instead of wired ethernet, need to run custom daemons and so on - all of which worked fine here for years - under systemd, you find out that nope, shares didn't mount because they tried and failed before there was an IP address (now fixed, which fix broke the workaround you needed at first) - things like conky get started in an endless timeout loop till the system dies from no memory (maybe fixed, so hard to get right I gave up on conky) and on and on for a long list of stuff that worked before systmd, and LP says "just don't do that, EWONTFIX". The whole list is too long for the margin here.
.

And his clan call us haters. They didn't have to make a ton of interlinked custom real hardware on prem systems work - broken time and time again by upgrades to that nasty piece of work...the web is full of workarounds that used to be the only things that worked, that are now broken as the systemd coders finally listened and fixed them but in such a way as to break the workaround, again, and again, and again.
Shades of lost productivity, near windows-like. Which of course doesn't affect RedHat revenue as they support - tons of identical instances...llike farcebook. The simplest case, get it right once and it's right everywhere. No edge cases. Wow, I'm er, underwhelmed.
In other words, this far fan-dancier init system doesn't (or didn't) handle anything complex, just added complexity to only handle the simplest stuff correctly. And was forced on us long before ready.
It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount. Glad it wasn't far away and I could just do that.
Not a hater, exactly. Just gheez - it caused me a ton of unnecessary unpaid work and stress and was arrogant about how it was somehow my fault for using what had been the documented standard ways of doing things. Way to win friends and influence people.

Re:Only one good use-case

[DCFusor]

What order, I did an NFS mount in /etc/fstab, which is the documented way that worked for years, and not only did systemd fail to mount it, it failed to respect the bg (backgrounding try again) flag, and then refused to let the system shut down since it couldn't unmount what it had failed to mount - which I verified by mounting manally at the CLI - and then it would unmount it and shutdown normally.
//

There was no fine manual to read that I'm aware of when this happened. Just oddball notes all over if you googled about how to fix just this particular bug - which workarounds worked until the next update. You might not have been paying attention.
//

Got my system in such a state? What a jerk, you're proving my point. Systemd got it into that state, I did zero, nada, nothing but allow an update to happen.
//

I don't have an employment contract - I work for my own outfit, which is a physics lab. It needs a lot of fairly custom machines from little to huge and fast for data aq and control - which is needed as the fusion reactor I'm developing works well enough to make too many neutrons to be safe near....
//

If it wasn't a bug that affected a lot of people, why did they complain and get the snotty remark - and then list workarounds findable on google?
//

That lovely restart loop on slow starting things is well documented as well, and has a workaround involving setting a longer timeout, which is a nasty hack...
//

Pretty sure it's not ignorant to trust "security" updates. I wouldn't have known systemd existed, much less have had time to retrain on hard to find or nonexistent documents that were changing daily...I'm REAL sure that something they broke isn't my fault, and people who tell me otherwise might be the problem.