Systemd is incredibly versatile and most people, including myself, are unaware of its full potential. Despite its usefulness, it is often overlooked due to controversy and the current state of things when it comes to software development. Begin today your journey thought Systemd’s capabilities and discover how to efficiently manage systems with fewer processes than traditionally required.

I still find it a solution desperately looking for a problem to solve. Yes, you can control loads of dependencies, but you could already do that with the init scripts.

TheInsane42 @ lemmy.world

Systemd does a lot more than that – it solves tons of painful issues and provides a cohesive ecosystem of tools to manage Linux systems. Here are a few examples:

• Start services in parallel (while init was one after the other) for a faster boot;
• Soft reboots;
• Defer service startup until they are actually needed;
• Replace a bunch of poorly integrated tools such as dhcpcd, dhcpv6, chrony, NetworkManager, resolvconf, logrotate;
• Has facilities to deal with IPv6 advanced use cases like IPv6-PD – no more need to install four packages in order to have IPv6;
• Easy policy-based routing tied to your network configuration without scripts;
• Centralizes logging and provides journalctl as a unified interface so you can quickly browse them by specific, filters, timestamps and events.
• Replaces crontab with something auditable that actually makes sense and is easy to use. Can also be configured to run “x seconds after each boot”, to avoid duplicate processes and much more;
• Monitors the status of your services and restarts them if they crash according to your rules. Can also be used to limit execution time and trigger notifications when stuff fails;
• Monitor files and directories for changes;
• Run programs and scripts when your hardware changes;
• Mount partitions / filesystems;
• Allows you to isolate your processes with sandboxing (like jails) techniques by restricting what resources (CPU time, RAM, networking, filesystems) they can access in a single unit file;
• Setup basic firewalls for your services;
• Setup port forwards / proxies for local and remote sockets. Can also be used to “convert” between TCP and UNIX sockets.
• Listen for connections in sockets and launch a program to handle incoming data when needed. Abuse it with PHP-FPM;
• Wake your system from sleep;
• It can be your boot loader;
• Create full fledged containers both privileged and non-privileged (here and here) while providing tools such as machinectl to manage them.

Yes, you read it right, Systemd can be used to create full fledged containers very much like LXD/LXC!

Managing Your Network

I’m struggling with networkd hysteresis (…) toggling the interface down and then back up does not (…) clear up the configuration, and setting the interface up does not reconfigure the interface. I have to run reconfigure for that. (…) what would the equivalent of ifdown and ifup be? This kind of feature would be useful for me to test config changes, debug networking issues, disconnect part of the network while I’m making some changes, etc.

dr_robot @ lemmy.world

That’s a valid issue and while systemd-networkd was designed for persistent configurations there are a few options:

• systemctl reload systemd-networkd and networkctl reload. If you change units on the filesystem a systemctl daemon-reload is required before the previous commands;
• networkctl commands such as reload, reconfigure, up, down and renew. Read more about them here;
• Temporary / volatile runtime units: manually drop a config under /run/systemd/network/ it will apply until you reboot the machine;
• Transient scope units: those are kind of supersede temporary units as they are managed through a D-Bus interface so 3rd party applications can manage systemd. They don’t seem to work right now for network, but this allows you to change unit options dynamically.

In most cases you can have multiple network setups in /etc/systemd/network but only bring them online when required.

Network Online?

I hate systemd with a passion, as the refuses to wait for networking when you haven some service specified to be started After networking.

TheInsane42 @ lemmy.world

If you apply what is written at “Cut the crap! How do I make sure that my service starts after the network is really online?” it will work. This will ensure that all configured network devices are up and have an IP address assigned before the service is started.

Then I don’t understand why maintainers all keep using network.target.

TheInsane42 @ lemmy.world

Most services are able to dynamically accommodate networking changes and act accordingly. It’s rare to have services where we really need addresses and a proper link on startup. For those special cases, as described, you need to enable systemd-networkd-wait-online.service and include After=network-online.target + Wants=network-online.target in your service.

Programs should be designed to detect and react to networking changes, both Apache and Nginx are good examples of software that does that. There some cases such as “stuff” that needs to bind to non-existent IPs at boot (before networking) that can be dealt with ipv4.ip_nonlocal_bind and net.ipv6.ip_nonlocal_bind as described here.

It is important to also understand what “network up” really means a few other details: https://systemd.io/NETWORK_ONLINE/

Network Time Made Easy

NTP is a must have and while most systems come with chrony systemd also has you covered for NTP:

apt purge chrony
systemctl enable systemd-timesyncd.service

Now edit /etc/systemd/timesyncd.conf and add your favorite time servers:

[Time]
NTP=193.136.152.71 193.136.152.72 194.117.47.42 194.117.47.44 162.159.200.1 162.159.200.123
FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org

Now a few useful commands to work with NTP time:

timedatectl set-ntp true  # update the time
hwclock --show            # show the current time in your hardware clock / UEFI
hwclock --systohc         # update the hardware clock with the system time we got from NTP

It was easy wasn’t it?

Fast and Secure DNS Resolution

Traditionally people are used to a bunch of nameserver entries at /etc/resolv.conf and while it works it doesn’t provide any security (encrypted DNS, query validation etc) nor does it provide efficient caching. Systemd can do it better with systemd-resolved.

systemd-resolved is a systemd service that provides network name resolution to local applications via a D-Bus interface, the resolve NSS service, and a local DNS stub listener on 127.0.0.53. In order to use it we can:

apt install systemd-resolved
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Add our resolvers to /etc/systemd/resolved.conf:

[Resolve]
DNS=1.1.1.1 8.8.8.8
FallbackDNS=9.9.9.9
DNSSEC=allow-downgrade
ReadEtcHosts=yes
Cache=yes
DNSStubListener=yes

For a more secure configuration we can, for instance, use DNS over TLS:

[Resolve]
DNS=1.1.1.1#1dot1dot1dot1.cloudflare-dns.com 1.0.0.1#1dot1dot1dot1.cloudflare-dns.com 2606:4700:4700::1111#1dot1dot1dot1.cloudflare-dns.com 2606:4700:4700::1001#1dot1dot1dot1.cloudflare-dns.com
DNSSEC=yes
DNSOverTLS=yes
ReadEtcHosts=yes
Cache=yes
DNSStubListener=yes

Start it with:

systemctl enable systemd-resolved.service
systemctl start systemd-resolved.service
resolvectl status

In some cases you might also need to edit your network configuration to make sure your local router isn’t pushing a DNS and NTP servers into your system. Edit /etc/systemd/network/10-eth0.network and add the following section:

[DHCPv4]
UseNTP=no
UseDNS=no
UseHostname=no

One more thing… Bonus feature!

For those running containers, virtual machines or are simply using a Systemd system as a router / gateway for a local network we can also leverage it as a local DNS server. Simply append DNSStubListenerExtra=10.0.0.1 to the [Resolve] section and you’re set. More info here.

Restrict a Service to a Network Interface

I just learned that VPN killswitches are a thing. I would like to [restrict Transmission] in case I forget to launch my VPN client before opening Transmission.

CowsLookLikeMaps@sh.itjust.works

In some cases it might be useful to restrict a daemon to a certain network interface (eg. Transmission) and for those you can override the demon’s unit (systemctl edit transmission-daemon.service) with the following:

[Service]
RestrictNetworkInterfaces=wg0 # --> your VPN interface

Another option would be to restrict the daemon to only be able to use a specific IP address:

[Service]
IPAddressDeny=any
IPAddressAllow=10.0.0.1 # --> your VPN IP here

Save the file and run systemctl daemon-reload followed by systemctl restart transmission-daemon.service and it should be applied.

This technique is safer than just telling a program to bind to a specific IP address because even if the program has a vulnerability or a bug systemd will still restrict it.

Non-Blocking PHP Job Queue

A common theme in web applications is to get a command from a client, eg. API POST request and then execute a long and expensive task in the background when possible – essentially a job queue. By leveraging Systemd’s ability to listen for connections we can build a very simple yet reliable job queue for PHP applications. Start with a socket and a service unit:

appqueue.socket:

[Unit]
Description=AppQueue Simple Socket

[Socket]
ListenStream = 127.0.0.1:50987
Accept = yes

[Install]
WantedBy = sockets.target

appqueue@.service:

[Unit]
Description=AppQueue Simple Service

[Service]
Type=oneshot
User=root
Group=root
MemoryMax=50M
WorkingDirectory=/web/yourapp
RemainAfterExit=no
StandardInput=socket
StandardOutput=journal

ExecStart=/usr/bin/php cli.php
StandardInput=socket

[Install]
WantedBy=multi-user.target

Create a cli.php and implemente a few lines of code to read all the standard input (via file_get_contents("php://stdin") and run tasks accordingly.

Then, in your main application, let’s assume index.php, you can “dispatch a background job” by connecting to the socket and sending data:

$task = "SEND-NEWSLETTER:1011"
$socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
socket_connect($socket, "127.0.0.1", 50987);
socket_write($socket, $task, strlen($task));
socket_close($socket);

Systemd will listen for connections at 127.0.0.1:50987 and once data is received it will launch a new PHP process and pass the data to its standard input. This way your client won’t have to wait for SEND-NEWSLETTER to finish as it will be running on a different process. And the best part? You can use journalctl to audit your jobs. 🙂

Closing Remarks

Systemd was crafted with a novel approach – inspired by Apple’s launchd – that contradicts traditional Unix philosophy, yet it has emerged as an impressively designed and unified ecosystem of tools and solutions to tackle common tasks.

It is included in most Linux distributions and can perform various functions such as networking configuration, containerization, process management, boot the system, listen for events and sockets, you name it. Now that you know how powerful, remarkably stable and mature it is… just use it!

Once you get into systemd you’ll have a few moments of pure amazement with it. One day you’ll be creating containers to suddenly realize that systemctl and journalctl can be used to inspect and affect a container in the same way you’re used to in your host machine. Another equally interesting moment in your journey is when you’ve an ARM system with 256 MB of RAM and you figure out that it just saved you 50 MB of precious RAM for other things.

Perhaps the kernel as systemd-kerneld?