A Look at Facebook's Use of Systemd (phoronix.com)

At an event this month (you can find the video of it here), Davide Cavalca, a production engineer at Facebook, spoke about the growing adoption of systemd at the data centers of the company. From a report: Facebook continues making use of systemd's many features inside their data centers. Some of their highlights for systemd use in 2018 includes: Facebook's servers have been relying on systemd for about the past two years. Facebook is using CentOS 7 everywhere from hosts to containers. While relying on CentOS 7, Facebook backports a lot of packages including new systemd releases, Meson, other dependencies, and of course new Linux kernel releases. Facebook is working on "pystemd" as a Python (Cython) wrapper on top of SD-BUS.

Comments

Ugh

[Miser]

Let Facebook HAVE systemd. They can keep it.

Re:Ugh

[Anonymous Coward]

They just had a major security leak of 50 million users? Competent you say?

Re:Ugh

[iggymanz]

pfft, they make $86-93K, they are not highly skilled admins

Re:Ugh

[wed128]

pfft, they make $86-93K, they are not highly skilled admins

This is evidence that they're not highly *paid* admins, and only has a passing correlation to skill

Re:Ugh

[Joe_Dragon]

that is crap in the Bay area.

Re:Ugh

[Anonymous Coward]

Yeah, all the highly skilled sysadmins updated their startup scripts years ago to start under systemd. It's only the lazy weak sysadmins who have spent the last four of five years whining that systemd once broke a script somewhere.

Re:Ugh

[Tailhook]

Amazon Linux — the image used on most AWS EC2 Linux instances — has also changed over to systemd with Amazon Linux 2, released last Jun 26. Amazon Linux was an important sysvinit holdout; if your users expected to use your work on EC2 with Amazon's distro you needed to care about sysvinit. Now that Amazon Linux has moved on there are no important distributions based on sysvinit remaining and it may be safely ignored.

Re:Ugh

[Anonymous Coward]

Nobody with any stroke wants to be a slave for Big Tech (tm).

Re:Ugh

[nagora]

You appear to think that sysvinit is the alternative to systemd. How quaint.

Re:Ugh

[Anonymous Coward]

Yeah, all the highly skilled sysadmins updated their startup scripts years ago to start under systemd. It's only the lazy weak sysadmins who have spent the last four of five years whining that systemd once broke a script somewhere.

The highly skilled admins forked Debian, to create Devuan, a Debian free of systemd.

Re:Ugh

[emacs_abuser]

I actually watched the video.
Seems like Facebook is finding systemd useful.
They really present some extreme use cases.
At one point they call systemd "awesome" so no doubt they will "keep it".

Re:Ugh

[Anonymous Coward]

Forked in 2014, stable release by May 2017, and yet whiners are STILL whining about systemd. For Christ's sake, why???

Re:Ugh

[gweihir]

Ahahahahahahaa, you think getting hacked proves competence?

Re:Ugh

[gweihir]

Ignoring good technology in favor of hip but bad technology is not a sign of competence.

Re: Ugh

[quadcricket]

Devuan really is a thing of beauty.

Re:Ugh

[Barsteward]

ROFL... aahh bless... bring back pen and paper....

Re:Ugh

[iggymanz]

A competent admin can make 50% more than that top end *away* from the bay area.

Re: Ugh

[Anonymous Coward]

It never left.

Systemd

[93 Escort Wagon]

#systemctl enable usertracking.service
#systemctl start usertracking.service

Re:systemd

[Anonymous Coward]

SYSTEMD!

Slowly I turned...step by step...inch by inch...

Then I unzip, strip, touch, finger, grep, mount, fsck, more, yes, fsck, fsck, fsck, umount, sleep.

Re:Systemd

[jmccue]

#systemctl enable usertracking.service #systemctl start usertracking.service

Meant to moderate funny but mouse slipped and it hit troll. Undoing mod.

systemd

[Anonymous Coward]

triggered

It is a floor wax it is a dessert topping

[bobstreo]

Name three things that deserve each-other:

Systemd, Bookface, Python,

Re:It is a floor wax it is a dessert topping

[shaitand]

Systemd, Bookface, Python, Vegans

Re:It is a floor wax it is a dessert topping

[gweihir]

I disagree. Python is actually pretty nice, but it requires somebody with real skill to be used competently.

This isn't noteworthy

[Anonymous Coward]

And systemd can go fuck itself

Re:This isn't noteworthy

[shaitand]

agreed

MOM! DAD! DON'T TOUCH THAT!

[Thud457]

a real "Nexus of Evil" to coin a phrase...

Re:MOM! DAD! DON'T TOUCH THAT!

[bobstreo]

a real "Nexus of Evil" to coin a phrase...

I am pretty sure Alphabet/Google has a trademark on Nexus (of Evil)

Re:MOM! DAD! DON'T TOUCH THAT!

[raftpeople]

"But master, if you created everything, who created you?" BZZZZAAAAAPPPPP

Wait

[ArchieBunker]

A Python wrapper for the systemd API? So now they're going through Python to go through systemd to make some calls...?

Re:Wait

[Anonymous Coward]

Combining Python and systemd is like mixing vomit with diarrhea.

Re:Wait

[Anonymous Coward]

It's end-to-end integration!

Re:Wait

[Anonymous Coward]

So a virus begins growing, thanks to social engineering*, and the infestation spreads, with no help at all from Microsoft. Oh, unless you count their recent moves as a distraction -- or misdirection. Well played.

Or to be more specific, thanks to the efforts of a well-meaning engineer at a well-known social network.

Re:Wait

[Anonymous Coward]

Like the Human iCentipad.

Re:Wait

[Anonymous Coward]

Don't forget d-bus and its xml-wrapping of requests and the various libraries that do the encoding and the interfacing to the encoding and the talking to d-bus and so on. This of course is obviously why d-bus needs to be in the kernel. Because logic.

I'm a little surprised zuckbook uses python, though. They're a php shop. Doesn't php have systemd api wrapper interfaces?

Re:Wait

[mishehu]

This is Mr. Chunks' winning combination!

Re:Wait

[mvdwege]

kdbus hasn't been an issue for at least 3 years, everyone is focusing on improving the userspace implementations of the DBus protocol.

But do go on, keep demonstrating that the anti-systemd side is mostly composed of idiots. The more you do your best, the sooner we can write you lot off as irrelevant.

Phew

[Anonymous Coward]

That explains the continual smell of decaying horse shit that wafts out of Facebook HQ.

It is a floor wax it is a holy war.

[Ostracus]

Oh! Holy Wars. Maybe this one will be better than the Vi-Emacs one.

Re:It is a floor wax it is a holy war.

[93 Escort Wagon]

Oh! Holy Wars. Maybe this one will be better than the Vi-Emacs one.

Nah. Either way, all the combatants do is type vicious comments on their devices.

Re:It is a floor wax it is a holy war.

[gweihir]

Hahahaha, I use Joe! I am a heretic to _both_ camps!

They’re using CentOS huh?

[93 Escort Wagon]

I can understand... these startups can’t afford the license fees Red Hat charges.

Re:They’re using CentOS huh?

[Anonymous Coward]

I can get RHEL through my site license, but I choose anything else becuase RHEL is always 4+ years behind with software versions.
That's nice for stability, but my users always request that I compile newer versions of stuff, so I end up with a worthless mess of compiled dependency libraries unless I start with a distro that has newer stuff built in. CentOS is a nice middle ground for a company which wants the RHEL stability but with some newer stuff beyond what epel can provide.

Re:They’re using CentOS huh?

[thule]

How about CentOS/RedHat base.... but running containers. You get the best of both worlds.

Python... the prime choice for robust software.

[Anonymous Coward]

"Who cares if it stable or robust! Have you seen how fast you can throw shit together in it?"

systemd has logged your complaint

[jfdavis668]

If anyone can ever figure out how to read the logs, someone may respond.

Re:systemd has logged your complaint

[Anonymous Coward]

LOL. It is a pain to remember things like:

systemctl restart openvpn@dev.service

The command isn't named systemd, there's an @ sign in the name, and the dev invokes magic to find /etc/openvpn/dev.ovpn.

But the logging issue is a bigger problem. I miss SysV scripts where even if something didn't get logged, you could at least see it on the console. Having to resort to using strace just sucks.

Re:systemd has logged your complaint

[Xenolith0]

But the logging issue is a bigger problem. I miss SysV scripts where even if something didn't get logged, you could at least see it on the console.

Yeah, I know what you mean! Typing "journalctl -u sshd" is impossible to remember! Much easier to just jump on the systemd-hate bandwagon and use strace.

It's SO-FUCKING-ANNOYING that systemd logs all stdout and stderr to the journal! I mean ugh! having to type journalctl. I can't even.

Re:systemd has logged your complaint

[DCFusor]

Yeah, you're one of the ones who can't even, that's obvious. When this crap just gets added to your system, how are you even supposed to know there's such a thing as journalctl? Some of us, do, you know, real work on this stuff, and only do sysadmin if it's required to set something nifty up. Having to also fix things someone else broke is extra work that isn't needed. The pushback isn't for no reason - systemd really screwed up a lot of people's day. If it just worked...probably would have been another story, despite the terrible architecture.

Re:systemd has logged your complaint

[Anonymous Coward]

"journalctl"

Wow that was hard. Bu-bu-bu the format is binary!!!! Yeah, because it merely massively improves security, tamper proofing, checksumming, range searches, sophisticated filtering. If you can't cope with binary, configure the thing to dump out to a text log as it goes. Desperately hard stuff I know.

Re:systemd has logged your complaint

[Xenolith0]

So you're mad that you might have to read a manual to a highly complex system? I'm impressed that you do "real work" on a system that you have no idea how it works!

If you don't want to bother reading manuals because you're too busy doing "real work". Why did you even bother upgrading to RHEL7? Stick with RHEL6, which is supported until 2024.

And yes, systemd is incredibly well documented. Just the man pages alone on the system:
$ man -k systemd | wc -l
147
But for online documentation: https://www.freedesktop.org/wiki/Software/systemd/

In almost every way systemd is better than sysv, upstart or other init systems, and it fixes many long standing issues SysV had.

Re:systemd has logged your complaint

[ArchieBunker]

Why did you feel the need to rewrite all the traditional text processing tools over again? So now you have binary logs and these new tools that do exactly what the old ones did...? Did someone use AIX one day and say hey now there's an idea Linux could use!

Re:systemd has logged your complaint

[Xenolith0]

See this for a detailed answer: https://www.loggly.com/blog/why-journald/

You can pipe the output of journalctl to whatever you want. So I'm not sure I understand the actual complaint. But further, while journal is great for catching start-up messages from services, it is not something you would use in an enterprise for true logging. Which is why on RHEL systems, journald is configured to forward everything to rsyslog which works the same as it always has. So... Nothing is really different, and you can do things as you always have.

Re:systemd has logged your complaint

[Anonymous Coward]

I've heard horror stories from seasoned admins. Lost log messages, when the system crashed, the journal would get corrupted. This complaint actually made it to their bug tracker where the lead developer of systemd closed the bug as "working as intended", then went all Linus on the submitter for not understanding how a journal becoming useless when the system crashes is perfectly acceptable. Why would someone ever want to know why their system crashed? They should just make sure it doesn't crash in the first place.

Re:systemd has logged your complaint

[jrumney]

$ man -k systemd | wc -l
147

You're not really helping your point there. Something as complex as systemd needs far more than 3 or 4 pages for its manual.

Re:systemd has logged your complaint

[Barsteward]

"When this crap just gets added to your system, how are you even supposed to know there's such a thing as journalctl?" - how did you learn to use anything on a computer? clue - RTFM

Re:systemd has logged your complaint

[Barsteward]

thats just for specific part of it. Description "systemd, init - systemd system and service manager"
I got 178 lines in my "man systemd" attempt and read this at the end of it (plus there are more notes after these ones)
"SEE ALSO The systemd Homepage[10], systemd-system.conf(5), locale.conf(5), systemctl(1), journalctl(1), systemd-notify(1), daemon(7), sd-daemon(3), systemd.unit(5), systemd.special(5), pkg-config(1), kernel-command-line(7), bootup(7), systemd.directives(7) "

Re:systemd has logged your complaint

[Barsteward]

oh god, what a load of bollox. Do you think the likes of Facebook with their huge system would use if that was the case. Are you making up or are you just reading/repeating loads of crap posted from the anti-crowd?

Re:systemd has logged your complaint

[Xenolith0]

Not sure if you're joking... But man -k lists all man pages about systemd. There are 147 man documents on a RHEL7 system, each multiple pages long.

Re:systemd has logged your complaint

[Anonymous Coward]

You're not really helping your point there.

You are being unintentionally ironic.

Something as complex as systemd needs far more than 3 or 4 pages for its manual.

Correct.

It is apparent you either do not know what "man -k" does, or you just missed it and went on systemd-hate autopilot. It is a common thing. You are not alone.

In case you actually do not know what "man -k" does: It lists the man-files related to the string you specify as the argument. In this case, the output lists 147 man-files (not lines) related to systemd, each of which typically contains several pages of documentation.

It is quite thoroughly documented.

Now, please continue with your systemd-hate. I am sure there are plenty more things you dislike. Some of which may be real, but as we have seen here, some of which may not be.

Re:systemd has logged your complaint

[ArchieBunker]

I hadn't touched Linux in a while and decided to use a BeagleBone board for a project. Even setting the IP address was a chore. I guess ifconfig and route weren't trendy enough so they had to be replaced with ip. Then all I needed was to set some IO pins at startup. Hmm no rc.local anymore. Try writing a new service in systemd. Oh it didn't work and doesn't say why. Ok let's try enabling rc.local from systemd. Oh that didn't work either. There isn't even a systemd manual, just people telling you how to get something working that they likely discovered by trial and error.

systemd hacking

[Anonymous Coward]

So that's why it's so easy to hack facebook.

Facebook's use?

[jeff4747]

I can't watch the video because work, but from the text this doesn't look much like Facebook is using systemd. Facebook is using an operating system that uses systemd. And may someday have a python api to do what everyone else does with bash.

Re:Facebook's use?

[thegarbz]

And may someday have a python api to do what everyone else does with bash.

If you think people are using bash to work with the SD-BUS API you either don't know Bash, or don't know the API being talked about.

Wretched hive of scum and villainy!

[Anonymous Coward]

Two asstastic products, that collapse into a singularity of asstatsticness together!

Comment

[Anonymous Coward]

Maybe they can restrictively license it so it stays away from our FLOSS operating systems! Wishful thinking...

Re:Comment

[shaitand]

I love this plan. For the first time every open source proponents push a movement to close source something!

Python? Cython?

[Anonymous Coward]

*sigh-thon is better.

Only one good use-case

[DCFusor]

for systemd - tons of identical instances, usually virtual so they can really be the same. The moment you customize with shares mounted on other machines, use wifi instead of wired ethernet, need to run custom daemons and so on - all of which worked fine here for years - under systemd, you find out that nope, shares didn't mount because they tried and failed before there was an IP address (now fixed, which fix broke the workaround you needed at first) - things like conky get started in an endless timeout loop till the system dies from no memory (maybe fixed, so hard to get right I gave up on conky) and on and on for a long list of stuff that worked before systmd, and LP says "just don't do that, EWONTFIX". The whole list is too long for the margin here.
.

And his clan call us haters. They didn't have to make a ton of interlinked custom real hardware on prem systems work - broken time and time again by upgrades to that nasty piece of work...the web is full of workarounds that used to be the only things that worked, that are now broken as the systemd coders finally listened and fixed them but in such a way as to break the workaround, again, and again, and again.
Shades of lost productivity, near windows-like. Which of course doesn't affect RedHat revenue as they support - tons of identical instances...llike farcebook. The simplest case, get it right once and it's right everywhere. No edge cases. Wow, I'm er, underwhelmed.
In other words, this far fan-dancier init system doesn't (or didn't) handle anything complex, just added complexity to only handle the simplest stuff correctly. And was forced on us long before ready.
It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount. Glad it wasn't far away and I could just do that.
Not a hater, exactly. Just gheez - it caused me a ton of unnecessary unpaid work and stress and was arrogant about how it was somehow my fault for using what had been the documented standard ways of doing things. Way to win friends and influence people.

Re:Only one good use-case

[thegarbz]

all of which worked fine here for years

Seriously you realise that systemd was build precisely because the things you listed didn't work fine for years.

shares didn't mount because they tried and failed before there was an IP address

Fix your startup order in the unit file.

things like conky get started in an endless timeout loop

Fix the start conditions in the unit file.

and LP says "just don't do that, EWONTFIX".

He was being polite. He should just tell people to RTFM.

And his clan call us haters.

No. Ignorant maybe, but not haters.

Shades of lost productivity, near windows-like.

Yeah that's what happens when you adopt a system and spend more effort complaining than simply reading the manual.

It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount.

It never ceases to amaze me how people can get their system in that kind of a state.

it caused me a ton of unnecessary unpaid work

Systemd can't help you with a poorly worded employment contract.

Re:Only one good use-case

[Anonymous Coward]

It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount.

It never ceases to amaze me how people can get their system in that kind of a state.

While not a failed mount, I just ran into this shutdown issue yesterday.

Clean Debian 9 install into a newly created VirtualBox VM.

"shutdown -r now" hangs halfway through the process with systemd starting a 15 minute timeout counter because it can't unmount "/"

Seeing as the only changes I made after boot were to set the hostname and lookup the MAC address.
At that point I added a dhcp reservation with fixed IP (on another system of course) and then came back to reboot the VM and make sure it got the right IP.

It certainly amazed me the system came out of the box in this kind of state.

Any ideas or suggestions what happened? Should I not have edited etc hosts and hostname as has been standard for decades? Should I not have typed "ip link show" as has been standard since "ipconfig" was removed? Is "shutdown -r now" no longer the correct way to reboot?

You are certainly not the first person to claim I fucked something up, but I am really really hoping you will be the first to tell me exactly what above was the fuckup and hopefully what to do instead.

Re:Only one good use-case

[DCFusor]

What order, I did an NFS mount in /etc/fstab, which is the documented way that worked for years, and not only did systemd fail to mount it, it failed to respect the bg (backgrounding try again) flag, and then refused to let the system shut down since it couldn't unmount what it had failed to mount - which I verified by mounting manally at the CLI - and then it would unmount it and shutdown normally.
//

There was no fine manual to read that I'm aware of when this happened. Just oddball notes all over if you googled about how to fix just this particular bug - which workarounds worked until the next update. You might not have been paying attention.
//

Got my system in such a state? What a jerk, you're proving my point. Systemd got it into that state, I did zero, nada, nothing but allow an update to happen.
//

I don't have an employment contract - I work for my own outfit, which is a physics lab. It needs a lot of fairly custom machines from little to huge and fast for data aq and control - which is needed as the fusion reactor I'm developing works well enough to make too many neutrons to be safe near....
//

If it wasn't a bug that affected a lot of people, why did they complain and get the snotty remark - and then list workarounds findable on google?
//

That lovely restart loop on slow starting things is well documented as well, and has a workaround involving setting a longer timeout, which is a nasty hack...
//

Pretty sure it's not ignorant to trust "security" updates. I wouldn't have known systemd existed, much less have had time to retrain on hard to find or nonexistent documents that were changing daily...I'm REAL sure that something they broke isn't my fault, and people who tell me otherwise might be the problem.

Re:Only one good use-case

[Anonymous Coward]

While not a failed mount, I just ran into this shutdown issue yesterday.

Clean Debian 9 install into a newly created VirtualBox VM.

You are certainly not the first person to claim I fucked something up, but I am really really hoping you will be the first to tell me exactly what above was the fuckup and hopefully what to do instead. (Emphasis added)

Use Devuan instead of Debian.

Pissed'em D...

[Anonymous Coward]

Ha ha ha!

Re:They're using CentOS huh?

[Anonymous Coward]

So, you're saying that packages built for Centos won't work on the corresponding version of RHEL? That doesn't match my experience at all.

Facebook are freeloaders, not freeriders.

Go Fuck Yourself

[Anonymous Coward]

Fuck off you systemd dev, you. You can shill and troll all you want but, it still doesn't change the FACT that everyone hates that steaming piece of shit, systemd.

Re: Go Fuck Yourself

[Anonymous Coward]

Why are you still whining?

  Just use Devuan and spare us your verbal diarrhea.

Just use Devuan and spare us your verbal diarrhea.

Just use Devuan and spare us your verbal diarrhea.

Crap Facebook uses crap systemd? Fits....

[gweihir]

Did it have a part in them getting hacked? Would not surprise me. These people are incompetent.

Re:Crap Facebook uses crap systemd? Fits....

[Anonymous Coward]

Last I checked Facebook had multiple kernel contributions. For a non OS/Hardware company FB is responsible for 0.9% of all kernel contributions.

Also Web app development vrs low level OS tools are very different layers of the stack.

Using slashdot after a decade, its still filled with tools who can't tell the difference between ?php to #include....

*nix needs a Registry

[aberglas]

A centralized place to store all a sytem's configuration, with a security model and lots of magic redirection that is different from the file system. All wired together with GUIDs.

SystemD is weak. It is a Registry would make *nix truly great.

That explains the Facebook downtime!

[Anonymous Coward]

Ah! that explains the Facebook downtime! systemd is productive in creating more problems. systemd is supported by thousands of coolies working for corporations and governments.