📌 20 de Novembro, 2023

Systemd: Hidden Gems for a Better Linux

Informática · Linux

📌 20 de Novembro, 2023

Systemd: Hidden Gems for a Better Linux

Informática · Linux

Systemd is incredibly versatile and most people, including myself, are unaware of its full potential. Despite its usefulness, it is often overlooked due to controversy and the current state of things when it comes to software development. Begin today your journey thought Systemd’s capabilities and discover how to efficiently manage systems with fewer processes than traditionally required.

I still find it a solution desperately looking for a problem to solve. Yes, you can control loads of dependencies, but you could already do that with the init scripts.

TheInsane42 @ lemmy.world

Systemd does a lot more than that – it solves tons of painful issues and provides a cohesive ecosystem of tools to manage Linux systems. Here are a few examples:

Yes, you read it right, Systemd can be used to create full fledged containers very much like LXD/LXC!

Managing Your Network

I’m struggling with networkd hysteresis (…) toggling the interface down and then back up does not (…) clear up the configuration, and setting the interface up does not reconfigure the interface. I have to run reconfigure for that. (…) what would the equivalent of ifdown and ifup be? This kind of feature would be useful for me to test config changes, debug networking issues, disconnect part of the network while I’m making some changes, etc.

dr_robot @ lemmy.world

That’s a valid issue and while systemd-networkd was designed for persistent configurations there are a few options:

  • systemctl reload systemd-networkd and networkctl reload. If you change units on the filesystem a systemctl daemon-reload is required before the previous commands;
  • networkctl commands such as reloadreconfigureupdown and renew. Read more about them here;
  • Temporary / volatile runtime units: manually drop a config under /run/systemd/network/ it will apply until you reboot the machine;
  • Transient scope units: those are kind of supersede temporary units as they are managed through a D-Bus interface so 3rd party applications can manage systemd. They don’t seem to work right now for network, but this allows you to change unit options dynamically.

In most cases you can have multiple network setups in /etc/systemd/network but only bring them online when required.

Network Online?

I hate systemd with a passion, as the refuses to wait for networking when you haven some service specified to be started After networking.

TheInsane42 @ lemmy.world

If you apply what is written at “Cut the crap! How do I make sure that my service starts after the network is really online?” it will work. This will ensure that all configured network devices are up and have an IP address assigned before the service is started.

Then I don’t understand why maintainers all keep using network.target.

TheInsane42 @ lemmy.world

Most services are able to dynamically accommodate networking changes and act accordingly. It’s rare to have services where we really need addresses and a proper link on startup. For those special cases, as described, you need to enable systemd-networkd-wait-online.service and include After=network-online.target + Wants=network-online.target in your service.

Programs should be designed to detect and react to networking changes, both Apache and Nginx are good examples of software that does that. There some cases such as “stuff” that needs to bind to non-existent IPs at boot (before networking) that can be dealt with ipv4.ip_nonlocal_bind and net.ipv6.ip_nonlocal_bind as described here.

It is important to also understand what “network up” really means a few other details: https://systemd.io/NETWORK_ONLINE/

Network Time Made Easy

NTP is a must have and while most systems come with chrony systemd also has you covered for NTP:

apt purge chrony
systemctl enable systemd-timesyncd.service

Now edit /etc/systemd/timesyncd.conf and add your favorite time servers:

FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org

Now a few useful commands to work with NTP time:

timedatectl set-ntp true  # update the time
hwclock --show            # show the current time in your hardware clock / UEFI
hwclock --systohc         # update the hardware clock with the system time we got from NTP

It was easy wasn’t it?

Fast and Secure DNS Resolution

Traditionally people are used to a bunch of nameserver entries at /etc/resolv.conf and while it works it doesn’t provide any security (encrypted DNS, query validation etc) nor does it provide efficient caching. Systemd can do it better with systemd-resolved.

systemd-resolved is a systemd service that provides network name resolution to local applications via a D-Bus interface, the resolve NSS service, and a local DNS stub listener on In order to use it we can:

apt install systemd-resolved
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Add our resolvers to /etc/systemd/resolved.conf:


For a more secure configuration we can, for instance, use DNS over TLS:

DNS= 2606:4700:4700::1111#1dot1dot1dot1.cloudflare-dns.com 2606:4700:4700::1001#1dot1dot1dot1.cloudflare-dns.com

Start it with:

systemctl enable systemd-resolved.service
systemctl start systemd-resolved.service
resolvectl status

In some cases you might also need to edit your network configuration to make sure your local router isn’t pushing a DNS and NTP servers into your system. Edit /etc/systemd/network/10-eth0.network and add the following section:


One more thing… Bonus feature!

For those running containers, virtual machines or are simply using a Systemd system as a router / gateway for a local network we can also leverage it as a local DNS server. Simply append DNSStubListenerExtra= to the [Resolve] section and you’re set. More info here.

Restrict a Service to a Network Interface

I just learned that VPN killswitches are a thing. I would like to [restrict Transmission] in case I forget to launch my VPN client before opening Transmission.


In some cases it might be useful to restrict a daemon to a certain network interface (eg. Transmission) and for those you can override the demon’s unit (systemctl edit transmission-daemon.service) with the following:

RestrictNetworkInterfaces=wg0 # --> your VPN interface

Another option would be to restrict the daemon to only be able to use a specific IP address:

IPAddressAllow= # --> your VPN IP here

Save the file and run systemctl daemon-reload followed by systemctl restart transmission-daemon.service and it should be applied.

This technique is safer than just telling a program to bind to a specific IP address because even if the program has a vulnerability or a bug systemd will still restrict it.

Non-Blocking PHP Job Queue

A common theme in web applications is to get a command from a client, eg. API POST request and then execute a long and expensive task in the background when possible – essentially a job queue. By leveraging Systemd’s ability to listen for connections we can build a very simple yet reliable job queue for PHP applications. Start with a socket and a service unit:


Description=AppQueue Simple Socket

ListenStream =
Accept = yes

WantedBy = sockets.target


Description=AppQueue Simple Service


ExecStart=/usr/bin/php cli.php


Create a cli.php and implemente a few lines of code to read all the standard input (via file_get_contents("php://stdin") and run tasks accordingly.

Then, in your main application, let’s assume index.php, you can “dispatch a background job” by connecting to the socket and sending data:

$task = "SEND-NEWSLETTER:1011"
$socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
socket_connect($socket, "", 50987);
socket_write($socket, $task, strlen($task));

Systemd will listen for connections at and once data is received it will launch a new PHP process and pass the data to its standard input. This way your client won’t have to wait for SEND-NEWSLETTER to finish as it will be running on a different process. And the best part? You can use journalctl to audit your jobs. 🙂

Closing Remarks

Systemd was crafted with a novel approach – inspired by Apple’s launchd – that contradicts traditional Unix philosophy, yet it has emerged as an impressively designed and unified ecosystem of tools and solutions to tackle common tasks.

It is included in most Linux distributions and can perform various functions such as networking configuration, containerization, process management, boot the system, listen for events and sockets, you name it. Now that you know how powerful, remarkably stable and mature it is… just use it!

Once you get into systemd you’ll have a few moments of pure amazement with it. One day you’ll be creating containers to suddenly realize that systemctl and journalctl can be used to inspect and affect a container in the same way you’re used to in your host machine. Another equally interesting moment in your journey is when you’ve an ARM system with 256 MB of RAM and you figure out that it just saved you 50 MB of precious RAM for other things.

Perhaps the kernel as systemd-kerneld?