Running a small Linux service with systemd

A long-running process needs more than a command that starts successfully in an SSH terminal. It needs an owner, a defined working environment, predictable restart behavior, useful logs, and a way to recover after maintenance. A small systemd unit can make those assumptions visible. The goal is to keep the service understandable enough that another administrator can inspect it without reconstructing an old shell session.

Identify the process and its requirements

Before writing a unit, find the executable's absolute path and determine whether it stays in the foreground. Record which directories it reads, which directories it writes, and whether it requires a privileged port. Create a dedicated service account when appropriate, but do not grant root merely to bypass a file-permission error.

Define the application's readiness test separately. A website might require a successful local HTTP request and a valid public TLS response. A worker might require a connection to its queue. Those checks describe the useful behavior; a process listing only establishes that something is running.

Choose a startup type deliberately

For a foreground application that does not implement systemd readiness notifications, Type=exec is a useful starting point. It lets the manager detect failure to execute the program, such as a missing binary or invalid service user. It does not guarantee that application initialization has completed. An application that supports readiness notifications can use Type=notify instead. The systemd service documentation explains these different startup contracts.

Here is an illustrative unit for an application already installed under /opt/lab-api:

[Unit]
Description=Example Lab API

[Service]
Type=exec
User=lab-api
Group=lab-api
WorkingDirectory=/opt/lab-api
ExecStart=/opt/lab-api/bin/lab-api --config /etc/lab-api/config.toml
Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/lab-api

[Install]
WantedBy=multi-user.target

The account, binary, configuration, and writable directory must exist before this example can run. Its paths are a design example, not an installer. Save a real unit with a name that clearly identifies the application.

Make permissions and isolation compatible

User= changes the process identity; it does not automatically repair ownership on existing data. Check the service account's access to every required path. A root-owned configuration file can remain readable by a narrowly scoped group while writable only by an administrator. Avoid placing credentials in the command line, where process inspection may expose them.

The filesystem and privilege controls above reduce the service's available surface, but each control needs compatibility testing. ProtectSystem=strict makes most of the filesystem read-only inside the service's mount namespace; ReadWritePaths= supplies exceptions without overriding normal file permissions. PrivateTmp= changes temporary-directory visibility. The systemd execution-environment documentation describes the exact scope and platform limitations of these settings.

Separate configuration loading from application changes

Validate the unit before making it persistent:

sudo systemd-analyze verify /etc/systemd/system/lab-api.service
sudo systemctl daemon-reload
sudo systemctl start lab-api.service
systemctl status lab-api.service --no-pager

Run the application's own configuration validator as well, if it has one. Unit validation cannot prove that a TOML file is valid or that the remote database password is correct. daemon-reload asks the manager to reread unit definitions; an application reload is a different operation and requires the application's support. Starting and enabling also have distinct purposes: enabling configures activation links, while starting activates the service now. See the systemctl reference.

Design failure behavior you can explain

An automatic restart is useful for an unexpected crash. It is less useful when a bad configuration creates a rapid loop. Choose a restart delay that allows investigation, and review the manager's start-rate limits rather than disabling them reflexively. Watch the restart count and the underlying exit reason.

Test a planned stop too. Confirm that the process exits, its listener disappears, and pending work is handled according to the application's design. Do not deliberately crash a production process just to prove the restart setting; use a separate test environment or a scheduled maintenance window for disruptive failure exercises.

Verify behavior and leave a recovery path

Read a bounded log interval with journalctl -u lab-api.service --since '10 minutes ago' --no-pager. Avoid sharing unreviewed logs: request URLs and application errors can contain credentials or personal data. Check the local listener and the real application response, then inspect the public endpoint from another machine when that endpoint is part of the service contract.

Keep the previous unit and application configuration together with a short change note. A rollback should identify the exact files to restore and the exact service to restart. After successful functional checks, enable the unit for boot if persistent operation is intended. This sequence makes a small service maintainable without turning a restart policy into a substitute for application health.