Why systemd is a problem for embedded Linux

The Internet is full of rants about systemd, and I don’t want this
post to be another one. Many of the complaints people make about it
don’t stand up to much scrutiny, even the technical ones; and many
complaints are not even technical. My particular interest in Linux is
primarily for embedded applications; and there, I suggest, systemd is
creating a potential (technical) problem. In this article I will try to
articulate what the problem is; but I have no solution to offer.

Recapping the last ten years

systemd is a set of integrated applications concerned with system
management. It replaces not only the traditional init
process that brings up long-lived processes, but also much of the other
system infrastructure: user session management, device management,
logging, timing, and an increasing number of other functions.

The majority of Linux users are uninterested in the pros and cons of
systemd. A small number are violently opposed to it, and a small number
are violently opposed to those who are opposed to it. Nevertheless, most
mainstream Linux distributions have adopted it after a shorter (Fedora)
or longer (Debian) period of argument.

I think there’s little argument that the main target for systemd is a
general-purpose computer, with a modern, integrated graphical desktop
(Gnome, KDE). systemd does well in systems like this because it can
handle process initialization on demand and in parallel. This
potentially makes boot times faster, and keeps resource usage down,
because it isn’t necessary to start a lot of services that are used only
occasionally. I don’t think these advantages are the main reasons for
systemd’s widespread adoption (see below), but they’re certainly
important.

Embedded Linux: the problem

Unfortunately, what makes systemd good for general-purpose desktop
applications potentially makes it unsatisfactory for embedded Linux
systems. As an illustration, I’ll show some memory figures from the
Raspberry Pi 3B that’s currently on my workbench. The board is running
the DietPi Linux distribution – probably the best fully-maintained Pi
distribution, if you want a minimal system. Although DietPi uses systemd
(it has little alternative, as I’ll explain later) it doesn’t
necessarily use the full set of components. In fact, a minimal
installation of DietPi, for console operation, installs only the systemd
service manager (the ‘init’ process), the log daemon, and the
udev daemon.

This is the resource usage, as reported by top, for the
systemd init process (alone) on the Raspberry Pi 3B.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM
      1 root      20   0  168144  11404   8592 S   0.0   0.3

systemd init is not a particularly greedy user of memory by
contemporary standards – its virtual address space is 168Mb, but only
~8Mb is currently mapped to RAM. That’s about 0.3% of the Pi 3’s 4Gb of
RAM. But here’s the same figures for SystemV init, on
exactly the same hardware:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM
    1 root      20   0    2744   1532   1436 S   0.0   0.2

It’s much smaller. Just to be clear – I got the systemd
implementation and the SystemV init implementation from the same Debian
ARM binary repository. I haven’t used fiddly compiler optimizations or
anything like that, to bias the resource usage figures.

Now let’s look at the systemd logging daemon,
systemd-journald.

  PID USER     PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    147 root   20   0   41508   7744   6652 S   0.0   0.2   0:12.05 systemd-jour+

Again, it’s a small, but noticeable, user of the 4Gb RAM. And, for
comparison, these are the figures from my own syslogd-lite,
which I wrote specifically for embedded applications.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  117 root     20   0    1976     72      0 S   0.0   0.0   0:00.00 syslogd

Note that the memory and CPU usage of this logging daemon are both
essentially zero. This is an unfair comparison, in a way, because I
wrote syslogd-lite specifically to minimize resource usage, and it has
few features. But it shows that it’s plausible to write utilities
that
target embedded systems specifically, and sometimes it’s necessary.
systemd does not do this, and was never intended to. Running a
general-purpose software set like systemd on minimal, embedded hardware
can’t be expected to be effective.

With care, though, a minimal installation of systemd does
run on a low-resource ARM board like the Pi 3. In fact, it will run on a
board with 1Gb RAM, perhaps even lower. But, as the RAM decreases, the
proportion of it occupied by systemd increases.

What’s less obvious is the effect on boot time of the use of systemd
init. Proponents of systemd argue – correctly, I think – that it
decreases boot time in the kinds of system for which it was designed.
But on my Pi 3 it increases boot time quite noticeably. That is, the
systemd ‘init’ process takes an extra half-second or so to start,
compared to SystemV init. Half a second isn’t much, but in an embedded
application I actually care about boot time. I couldn’t care less on my
desktop computers, because they spend most of their lives suspended. I
rarely actually reboot them.

The extra start-up time of the init process is the result, I guess,
of the additional size and complexity of the systemd executable. It’s
about 200kB in size itself, and dynamically links 26 other libraries.
SystemV init, on the same hardware, is 40kB, and links only the standard
C library. The additional complexity of systemd is not wasted: it’s
needed for the additional functionality that systemd offers, in its
intended environment. But this functionality is mostly not needed in an
embedded application, so the additional complexity of systemd is a cost
without a benefit.

I’ve found that most of the services that systemd replaces have an
alternative that is smaller, and faster to start, in an embedded
environment. Unfortunately, some of these services don’t have
an alternative any more.

So what?

I’m not obliged to run systemd on my Raspberry Pi systems and, in
fact, usually I do not. I build my own Linux installation, using
binaries that I cherry-pick from the Debian repositories, and code I
write myself. Most of the binaries work without systemd. Some complain
about not finding systemd, but work anyway. Some things don’t work
without systemd: the Gnome display manager, for example, as it is built
for the standard Raspberry Pi, won’t work. It can be made to work, but
you have to build it from source. How long it will continue to work,
even if built from source, is open to question. But I’m not going to be
running Gnome on an embedded Linux board, so I don’t see this as a
problem for my applications.

The more fundamental problem is that the people who most like
systemd are distribution managers. Sure, there are administrators
who like it, and defend it vigorously; but most end users and
administrators don’t really care. But for the maintainers of mainstream
Linux distributions, systemd is like Christmas. systemd works reasonably
well for a whole range of usage scenarios and, best of all, it all comes
in one bundle. So a distribution manager doesn’t have to maintain,
integrate, and support a whole bunch of system utilities from different
sources – systemd provides everything in one huge build.

There are just a few Linux distributions that don’t use systemd, and
they are not widely used. Maintaining them is difficult, with just a
handful of volunteers. All the large, commercial Linux providers have
moved to systemd.

Only Gentoo and its derivatives (so far as I know) make systemd
optional, with fully-supported alternatives. And even Gentoo can’t be
made 100% free of systemd – not in practice.

Is it even
practical to avoid systemd any more?

Take, for example, the udev daemon. This service
monitors the kernel, and makes configuration changes when devices are
added and removed. It’s not an essential part of Linux, but it removes
the need for a whole heap of manual configuration.

At the time of writing there is no fully-featured
udev implementation outside of systemd. The original
udev code was absorbed into the systemd project about ten
years ago. The Gentoo alternative eudev is no longer fully
maintained. At present, if your Linux distribution requires
udev, you’re forced to use the version from systemd. This
version can be used without the rest of systemd, but it nevertheless
does systemd operations. In particular, it tries to communicate with
systemd over DBus. This communication (so far) fails gracefully if
systemd is not present, but it’s not clear how long this will continue
to be the case.

systemd is a tightly-coupled set of services. I don’t think a design
goal of the systemd maintainers is to make its components modular.
udev doesn’t need to talk to systemd but, clearly, there’s
some benefit to its doing so in a systemd installation. I understand
that there was a kind of ‘gentlemen’s agreement’ between the systemd
maintainers and the Gentoo maintainers, to keep udev
independent of the rest of systemd. I think we can see that this
agreement has broken down a little already; I suspect it will break down
more, if the systemd folks think that tighter integration will make
systemd work better.

Many parts of systemd continue – for now – to have non-systemd
alternatives. For example, systemd has a clock synchronizer
systemd_systemtimed. The systemd maintainers are perfectly
honest that this software lacks features that exist in alternatives like
Chrony and OpenNTPD, and it’s less accurate. systemd_timed
is included because it’s fast to start, and satisfactory for
applications where exact time synchronization is not critical.

At present, Chrony remains widely used, even in some distributions
(like Fedora) that use systemd. But with systemd becoming ubiquitous,
what motivation will there be to maintain non-systemd alternatives?
These alternatives could fall into disrepair, even though some are
superior to the systemd utilities – and even the systemd maintainers
admit this.

Similarly, systemd has a DHCP client. It isn’t the only DHCP client
that is available, but my concern is that one day it might be. In my
tests, I’ve found that the systemd components are larger, and slower to
start, than the traditional alternatives (where they still exist).
Again, this isn’t a criticism of systemd itself – I’m testing them in an
environment they were not designed for.

So where does that
leave embedded Linux?

I’ve found that many systemd components are less effective in an
embedded environment than the traditional alternatives. I’ve shown some
illustrative examples in this article, but I really don’t think there’s
much controversy here: this simply isn’t the environment that systemd
was designed for. But it’s getting increasingly difficult to find a
mainstream Linux distribution that doesn’t use systemd – even Raspberry
Pi distributions use it. As systemd absorbs more functionality into
itself, there’s going to be little motivation to maintain alternatives.
After all, if everybody uses systemd, what motivation is there to
support anything else? My concern is that we’re moving towards a future
where Linux is inconceivable without systemd. That will be a problem for
those environments where systemd really doesn’t shine.

I wish I knew the solution to this problem. There’s no point
complaining about systemd, because distribution maintainers have grown
to like it too much. And, in any event, well-reasoned, technical
concerns are drowned out by all the ranting and conspiracy theories. All
we can do – if we care – is to continue to use and support Linux
distributions that don’t insist on systemd, and stand ready to develop
or maintain alternatives to those bits of Linux that it absorbs.