Blog: Expanded Universe

How to make a systemd service

On a modern Linux computer, systemd lets you run a program and automatically restart it when the computer reboots and/or the program crashes. (Among many, many other things.)

tl;dr

Save a file like the following to /etc/systemd/system/myserver.service:

[Unit]
Description=My Python server

[Service]
User=brian
WorkingDirectory=/home/brian/myserver
ExecStart=/home/brian/myserver/.venv/bin/python /home/brian/myserver/myserver.py
Restart=always

[Install]
WantedBy=multi-user.target

This is a unit file. It's best to specify the command executable as an absolute path, as I've done here.

Then (with sudo as appropriate):

Don't like or subscribe

verbosity=2

To specify what a systemd service runs, you write down these things in the unit file:

If you normally run setup steps before starting a program, you'll need to incorporate the changes they make into the unit file. You can check environment variable changes with something like:

diff <(env | sort) <(source .venv/bin/activate; env | sort)

For the case I'm most familiar with, activating a Python virtualenv, that basically just prepends .venv/bin to PATH. But (as explained in the footnote) adding Environment=PATH=/path/to/.venv/bin in the unit file won't make bare python do the right thing. Instead, it's necessary to spell out the full path to .venv/bin/python . Per the Python docs, this is also sufficient.

In the service options, you can also specify the restart behavior. Restart=always is the most blunt-force setting, always restarting the program when it ends for any (non-systemd-mediated) reason. You can make it restart only when it crashes by replacing it with this line:

Restart=on-failure

You can make it wait, say, 5 seconds before restarting, by adding this line after. By default, systemd waits 100 ms.

RestartSec=5s

(systemd.time explains what units are supported, including ms, s, and m. You can also just omit s here, because many time fields assume seconds if there's no unit.)

systemd will give up on restarting if the program restarts too many times quickly, by default more than 5 times in 10 seconds3; you can customize this by setting StartLimitIntervalSec and StartLimitBurst (note these are [Unit] section options). You can list these failed units with systemctl --failed, and reset the failed status on a unit by running systemctl reset-failed myserver (or without arguments to reset all failures).

By default, anything the program prints goes to the journal. A few more administration commands:

verbosity=3

A service is just one of 11 types of systemd unit; if you're wondering, the other types are socket, target, device, mount, automount, timer, swap, path, slice, and scope. When writing a unit name on the command line, .service is generally implied, but you can also write out the full name myserver.service. For other types of units you'd have to write the full name out, name.type.4 You also have to write out the full name in config files.

Units can be related to each other in several ways — the relations are called "dependencies", which isn't my favorite name, but it's what we have. The most important ones are Wants and Before/After. A slightly less common one is Requires. There are a lot more I won't go into. Briefly:

A unit's dependencies can be specified in the [Unit] section with lines like Wants=otherserver.service, or in the filesystem as symlinks, e.g. of /etc/systemd/system/myserver.service.wants/otherserver.service to the otherserver.service unit file. This is a kind of facile description; the symlink representation is there for systemctl subcommands to examine and modify. Specifically, a unit file's [Install] section can specify a few "backwards" dependencies like WantedBy and RequiredBy, and then running systemctl enable on a unit will add the symlinks representing those dependencies from those other units to it. Meanwhile, running systemctl is-enabled checks for whether any symlink from another unit to this unit exists, while running systemctl disable deletes all symlinks from other units (note that neither of these are restricted to the symlinks asked for by [Install]). There is no separate "is enabled" flag anywhere. You can manually add/delete symlinks to effectively enable/disable services or set up their dependencies in ways not specified by the unit files.

Anyway, multi-user.target is a special unit that represents "setting up a multi-user system", which roughly just means booting the system normally. That's why, to declare a conventional service that starts on boot, we use an [Install] section that says the service is enabled by adding a symlink that represents "multi-user.target Wants this service".

Another useful special unit is network-online.target, representing when the network is, well, online. If a service needs to access the Internet to run, you might give it Wants=network-online.target and After=network-online.target.

systemd can also run services under your own account, so you don't have to sudo. In this setup:

  1. The unit file goes in ~/.config/systemd/user/myserver.service.
  2. The User= defaults to you, and the WorkingDirectory= defaults to your home directory.
  3. The [Install] section should say WantedBy=default.target. (The name default.target is an alias in the system service manager, but a real unit in the user service manager.)
  4. You'll need to add --user to systemctl and journalctl invocations.

However, by default, user services stop when you log out and don't start until you log in. To run them at boot regardless, run sudo loginctl enable-linger brian once (loginctl).

There are a bunch more (sub)types of services, specified with Type= in the [Service] section, with different definitions of what it means to start a service and how one determines whether the service is running. An interesting one is Type=oneshot, where the service isn't anything long-running at all; each time you systemctl start the service, it runs the ExecStart lines and then stops. Uniquely, Type=oneshot services can have multiple ExecStart lines.

RemainAfterExit=yes is another flag. You can pair it with Type=oneshot to get a "service" where nothing is running in steady state, but one thing runs on start and another thing runs on stop, as in this example from the docs:

[Unit]
Description=Simple firewall

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/simple-firewall-start
ExecStop=/usr/local/sbin/simple-firewall-stop

[Install]
WantedBy=multi-user.target

You can use the above as a way to run something on boot and shutdown.

A whole lot of other system operations are special targets, such as reboot and shutdown. You can attach commands to these operations with services that are WantedBy them, but there are some gotchas involving ordering and DefaultDependencies. The oneshot service with an ExecStop seems easier to get right.

systemd can place various limits on services and stop them if they exceed the limit (these count as failures, so if the service also has Restart=always or Restart=on-failure, it'll get restarted after):

There's also a long list of sandboxing settings: ProtectSystem=, NoNewPrivileges=, SystemCallFilter=, etc. This isn't the most robust sandbox on the planet, but it might catch some bugs. Look them up if you want to use them.

A "template" is a unit file whose name ends in @, like myserver@.service. It can be instantiated by trying to start a unit like myserver@foo.service. This replaces %i in the unit file with foo. Then you can start and independently manage multiple copies of nearly the same service.

if this were a recipe blog, this would be the first section, but i respect your time too much

I actually remember exactly where I first realized I could leave random projects running on my server in tmux. It was from an aside in Quick doesn't have to mean dirty, 2012, which explicitly suggested never doing that. Oops.

Still, the tmux solution is pretty convenient and approachable! It gives me a really tangible exhaustive list of my running projects (as distinct from all the normal system stuff running on the server), and lets me monitor each of them in an interactive interface with keybinds I already know. I was only motivated to turn them into systemd services because one of the random projects had a memory leak that caused one of the other random projects to crash (I think, this is a post hoc reconstruction), but I didn't notice for a few days, so I lost some data.

It's kind of a weird feeling writing programming documentation in 2026, isn't it?

demo server for messing around

To force us to think about the environment, we import and use humanize, a somewhat arbitrary choice of a non-default library.

import os
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import humanize

STARTED = time.time()

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/crash":
            self.log_message("crashing on purpose")
            os._exit(1)
        uptime = humanize.precisedelta(int(time.time() - STARTED))
        self.send_response(200)
        self.end_headers()
        self.wfile.write(f"Started {time.ctime(STARTED)}; up {uptime}\n".encode())

ThreadingHTTPServer(("127.0.0.1", 8000), Handler).serve_forever()
  1. The syntax is "inspired by shell syntax": it supports simple environment variable substitution, and you can write a simple file name instead of an absolute path, but systemd will only look for the executable in a few hardcoded directories. In particular, it does not consult your shell PATH, nor a PATH you declare as an environment variable. (Though, if the executable itself invokes other commands, those invocations might depend on PATH.) There's also some sigils you can prepend to the executable that are completely unrelated to shell syntax.↩

  2. You can write Environment= multiple times to specify multiple variables, or you can write multiple key-value pairs in one line separated by spaces. To specify a value that itself includes spaces, quote the entire key=value: Environment="KEY=value with spaces" KEY2=value2. Alternatively, you can specify an EnvironmentFile=/path/to/file pointing to a file of KEY=value lines.↩

  3. defaults from systemd-system.conf, DefaultStartLimitIntervalSec= DefaultStartLimitBurst=↩

  4. systemctl commands "Parameter Syntax"↩