systemd is the init system and service manager that runs as PID 1 on virtually every major Linux distribution. Love it or hate it, understanding systemd is non-negotiable for anyone managing Linux servers. It handles service lifecycle, logging, scheduling, network configuration, device management, and much more.

This guide covers the core systemd components that every system administrator needs to master, from basic service management to advanced features like socket activation and resource control.

Unit Files: The Building Blocks

Everything in systemd revolves around unit files. A unit represents a resource that systemd manages. The most common unit types are:

Unit Type Suffix Purpose
Service .service Daemons and processes
Timer .timer Scheduled tasks (cron replacement)
Socket .socket IPC/network socket activation
Target .target Group of units (like runlevels)
Mount .mount Filesystem mount points
Path .path Filesystem path monitoring
Slice .slice Resource management groups (cgroups)

Unit files are stored in these locations, with later locations overriding earlier ones:

/usr/lib/systemd/system/   # Distribution-provided units (do not edit)
/etc/systemd/system/       # Administrator-created units (your customizations)
/run/systemd/system/       # Runtime units (transient, lost on reboot)

Service Management Essentials

The systemctl command is the primary interface for managing services:

# Start, stop, restart, reload a service
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx          # Reload config without restarting

# Enable/disable on boot
systemctl enable nginx          # Start at boot
systemctl disable nginx         # Do not start at boot
systemctl enable --now nginx    # Enable AND start immediately

# Check status
systemctl status nginx
systemctl is-active nginx       # Just "active" or "inactive"
systemctl is-enabled nginx      # Just "enabled" or "disabled"
systemctl is-failed nginx       # Just "failed" or not

# List all services
systemctl list-units --type=service
systemctl list-units --type=service --state=running
systemctl list-units --type=service --state=failed

# Show unit file contents
systemctl cat nginx.service

# Show all dependencies
systemctl list-dependencies nginx.service
systemctl list-dependencies --reverse nginx.service

Writing Custom Service Units

Here is a comprehensive service unit file with explanations for every section:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Server
Documentation=https://docs.example.com/myapp
After=network-online.target postgresql.service redis.service
Wants=network-online.target
Requires=postgresql.service
BindsTo=redis.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=notify
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=-/opt/myapp/.env
ExecStartPre=/opt/myapp/bin/check-config
ExecStart=/opt/myapp/bin/server --config /opt/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
TimeoutStartSec=30
TimeoutStopSec=30
Restart=on-failure
RestartSec=5
WatchdogSec=60

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
RestrictNamespaces=true
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
SystemCallFilter=@system-service

# Resource limits
LimitNOFILE=65535
LimitNPROC=4096
MemoryMax=2G
CPUQuota=200%
TasksMax=512

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

Service Types Explained

Type Behavior Use Case
simple Process started by ExecStart is the main process Most foreground daemons
exec Like simple, but considered started after exec() succeeds When startup confirmation matters
forking Process forks; parent exits; child is the daemon Traditional daemons (Apache, nginx)
notify Process sends sd_notify() when ready Modern daemons with readiness notification
oneshot Process runs once and exits Scripts, initialization tasks

Using Drop-in Overrides

Never edit distribution-provided unit files. Use drop-in directories instead:

# Create an override for docker.service
systemctl edit docker.service
# This creates /etc/systemd/system/docker.service.d/override.conf

# Or manually:
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/override.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --log-level=warn --storage-driver=overlay2
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
EOF

systemctl daemon-reload
systemctl restart docker
Tip: When overriding ExecStart, you must first clear it with an empty ExecStart= line before setting the new value. This is a common source of confusion.

Timers: The Modern Cron Replacement

systemd timers are more powerful than cron jobs. They support monotonic timers (relative to events), calendar expressions, randomized delays, and persistent scheduling. Every timer needs a matching service unit.

# /etc/systemd/system/backup.service
[Unit]
Description=Daily Backup Task

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
User=backup
Group=backup
StandardOutput=journal
StandardError=journal
# /etc/systemd/system/backup.timer
[Unit]
Description=Run Daily Backup at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=1800
Persistent=true
AccuracySec=1min

[Install]
WantedBy=timers.target
# Enable and start the timer
systemctl enable backup.timer
systemctl start backup.timer

# List all timers
systemctl list-timers --all

# Test when it will fire next
systemd-analyze calendar "*-*-* 02:00:00"

# Run the service manually (for testing)
systemctl start backup.service

Timer Calendar Expressions

# Common calendar expressions
OnCalendar=hourly                    # Every hour
OnCalendar=daily                     # Every day at midnight
OnCalendar=weekly                    # Every Monday at midnight
OnCalendar=monthly                   # First day of month
OnCalendar=*-*-* 04:00:00           # Every day at 4 AM
OnCalendar=Mon..Fri *-*-* 09:00:00  # Weekdays at 9 AM
OnCalendar=*-*-01 00:00:00          # First of every month
OnCalendar=*:0/15                    # Every 15 minutes

# Monotonic timers (relative to events)
OnBootSec=5min                       # 5 minutes after boot
OnUnitActiveSec=1h                   # 1 hour after last activation
OnStartupSec=30s                     # 30 seconds after systemd start

Journald: Centralized Logging

systemd's journal collects logs from all services, the kernel, and the early boot process into a single queryable database:

# View all logs
journalctl

# Follow logs in real time (like tail -f)
journalctl -f

# Logs for a specific service
journalctl -u nginx.service
journalctl -u nginx.service -f

# Logs since a time period
journalctl --since "2025-03-12 10:00:00"
journalctl --since "1 hour ago"
journalctl --since today

# Logs by priority
journalctl -p err                    # Errors and above
journalctl -p warning               # Warnings and above
journalctl -p 0..3                   # Emergency through Error

# Kernel logs only
journalctl -k

# Boot logs
journalctl -b                        # Current boot
journalctl -b -1                     # Previous boot
journalctl --list-boots              # List all boots

# Output formats
journalctl -u nginx -o json-pretty   # JSON format
journalctl -u nginx -o short-precise # Precise timestamps
journalctl -u nginx -o cat           # Message only, no metadata

# Disk usage
journalctl --disk-usage

# Clean up old logs
journalctl --vacuum-size=500M        # Keep only 500M
journalctl --vacuum-time=30d         # Keep only 30 days

Journald Configuration

Edit /etc/systemd/journald.conf:

[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=1G
SystemKeepFree=2G
SystemMaxFileSize=100M
MaxFileSec=1month
ForwardToSyslog=no
MaxRetentionSec=3month

Targets: Grouping Units

Targets are groups of units that represent system states (similar to traditional runlevels):

# Common targets
multi-user.target    # Full multi-user system (like runlevel 3)
graphical.target     # GUI desktop (like runlevel 5)
rescue.target        # Single-user mode (like runlevel 1)
emergency.target     # Minimal shell
reboot.target        # Reboot
poweroff.target      # Power off

# Check current target
systemctl get-default

# Set default target
systemctl set-default multi-user.target

# Switch target at runtime
systemctl isolate rescue.target

# Create a custom target
cat > /etc/systemd/system/myapp.target << 'EOF'
[Unit]
Description=My Application Stack
Requires=postgresql.service redis.service myapp.service
After=postgresql.service redis.service

[Install]
WantedBy=multi-user.target
EOF

Socket Activation

Socket activation allows systemd to listen on a port and start the associated service only when a connection arrives. This saves resources and enables zero-downtime restarts:

# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket

[Socket]
ListenStream=8080
Accept=no
ReusePort=true

[Install]
WantedBy=sockets.target
# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
Requires=myapp.socket

[Service]
Type=notify
ExecStart=/opt/myapp/bin/server
Sockets=myapp.socket

# The service receives file descriptors from the socket
# No need to bind to a port in the application code
# Enable the socket (not the service)
systemctl enable myapp.socket
systemctl start myapp.socket

# The service starts automatically on first connection
curl http://localhost:8080/

Resource Control with Cgroups

systemd uses cgroups to control and limit resources for each service:

# View resource usage for a service
systemd-cgtop

# Set resource limits in the service unit
[Service]
# Memory limits
MemoryMax=2G                  # Hard limit
MemoryHigh=1.5G               # Soft limit (triggers reclaim)
MemorySwapMax=0                # Disable swap for this service

# CPU limits
CPUQuota=200%                  # 2 CPU cores maximum
CPUWeight=100                  # Relative CPU weight (default: 100)

# I/O limits
IOWeight=100                   # Relative I/O weight
IOReadBandwidthMax=/dev/sda 50M
IOWriteBandwidthMax=/dev/sda 30M

# Task limits
TasksMax=512                   # Maximum number of tasks/threads

# Network (with systemd-networkd)
IPAddressDeny=any
IPAddressAllow=192.168.1.0/24
IPAddressAllow=127.0.0.0/8
# Check a service's resource limits
systemctl show myapp.service | grep -E "Memory|CPU|Tasks"

# Set limits at runtime (transient)
systemctl set-property myapp.service MemoryMax=1G CPUQuota=150%

Debugging systemd

# Analyze boot time
systemd-analyze
systemd-analyze blame              # Time per unit
systemd-analyze critical-chain     # Critical path
systemd-analyze plot > boot.svg    # Visual boot chart

# Check for broken units
systemctl --failed

# Verify a unit file for syntax errors
systemd-analyze verify /etc/systemd/system/myapp.service

# Security audit of a service
systemd-analyze security myapp.service

# Show properties of a unit
systemctl show myapp.service

# View dependency tree
systemctl list-dependencies myapp.service --all

# Debug a failing service
journalctl -u myapp.service -n 50 --no-pager
systemctl status myapp.service -l
Tip: The systemd-analyze security command scores your service from 0 (fully sandboxed) to 10 (completely exposed). Aim for a score below 5 for any internet-facing service. Each line item tells you what hardening directive to add.

Path Units: Filesystem Monitoring

Path units watch for filesystem changes and trigger services in response:

# /etc/systemd/system/config-watcher.path
[Unit]
Description=Watch for config changes

[Path]
PathModified=/etc/myapp/config.yaml
Unit=config-reload.service

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/config-reload.service
[Unit]
Description=Reload application on config change

[Service]
Type=oneshot
ExecStart=/bin/systemctl reload myapp.service

Temporary Files with systemd-tmpfiles

# /etc/tmpfiles.d/myapp.conf
# Type  Path                     Mode  User   Group  Age  Argument
d       /var/lib/myapp           0750  myapp  myapp  -    -
d       /var/log/myapp           0750  myapp  myapp  -    -
d       /run/myapp               0755  myapp  myapp  -    -
f       /var/log/myapp/app.log   0640  myapp  myapp  30d  -

# Apply the configuration
systemd-tmpfiles --create /etc/tmpfiles.d/myapp.conf

Understanding systemd deeply is essential whether you are managing bare-metal Linux servers or Docker hosts. Tools like usulnet run as systemd services themselves, and understanding unit files helps you configure startup behavior, resource limits, and logging for your container management platform.

Key takeaway: systemd is not just an init system. It is a system and service manager that handles services, logging, scheduling, device management, network configuration, and resource control. Learning it well eliminates the need for multiple separate tools and provides a consistent interface across all Linux distributions.