Linux errors and how to fix them

The file has no execute bit (-rw-r--r--). Without x, the kernel refuses to run it as a program, whoever owns it.

Check

ls -l deploy.sh   # no x in the mode (-rw-r--r--): this is your cause

Fix

chmod u+x deploy.sh (or 755 if others run it too).

The file system is mounted noexec (a hardened /tmp or /home often is). noexec forbids running any program from that mount, even one with the x bit.

Check

findmnt -no OPTIONS --target deploy.sh   # noexec in the list: the mount forbids it

Fix

Keep scripts where execution is allowed (/usr/local/bin, /opt). bash deploy.sh works, but it sidesteps a policy someone set on purpose.

A directory on the path is not searchable for you. Without x on a directory, nothing inside it can be reached, so the run fails the same way.

Check

namei -l "$PWD/deploy.sh"   # a directory without x for you: this is your cause

Fix

Add x for the group or others on that directory (chmod o+x DIR), never 777.

Windows line endings (CRLF). Each line ends in \r\n, so the shebang names /bin/bash\r, which does not exist.

Check

file deploy.sh   # with CRLF line terminators: this is your cause
head -1 deploy.sh | od -c | head -2   # \r before \n confirms it

Fix

sed -i 's/\r$//' deploy.sh, or dos2unix deploy.sh (apt or dnf install dos2unix).

The shebang names an interpreter that is not installed (python, an old path). The kernel reports the missing interpreter with the same words.

Check

head -1 deploy.sh   # the interpreter it asks for
command -v bash python3   # no output for that name: it is not installed

Fix

Install it, or point the shebang at one that exists, such as #!/usr/bin/env python3.

The file has the immutable attribute (chattr +i). An immutable file cannot be changed, renamed over or deleted, by anyone, until the flag is removed.

Check

lsattr /etc/nginx/conf.d/rate-limit.conf   # an i in the flags: this is your cause

Fix

sudo chattr -i FILE, edit it, then sudo chattr +i FILE again if the lock was intended.

The append-only attribute (chattr +a). Appends work, but rewriting, renaming or truncating fails with this error.

Check

lsattr /etc/nginx/conf.d/rate-limit.conf   # an a in the flags

Fix

sudo chattr -a FILE, make the change, and put the flag back if it was intended.

The directory is immutable. Then nothing in it can be created, renamed or removed, so the temporary file or the rename fails.

Check

lsattr -d /etc/nginx/conf.d   # an i on the directory itself

Fix

sudo chattr -i DIR, make the change, then restore the flag if it was intended.

root without the capability: inside a container, or chown on an NFS share exported with root_squash. That root is not the real root, so the kernel or the server says no.

Check

findmnt -no FSTYPE,OPTIONS -T /etc/nginx/conf.d   # nfs, proc or sysfs: the file system refuses
grep CapEff /proc/self/status   # fewer bits than the same line on the host: a restricted root

Fix

Make the change where root really has the right: on the host, or on the NFS server.

Files created by root with sudo (tar, cp, git pull) in the app's folders. They belong to root, so the service user cannot write them.

Check

namei -l /srv/shop/uploads   # root-owned parts on the path
sudo -u shop test -w /srv/shop/uploads && echo writable   # no output: the service user cannot write

Fix

sudo chown -R shop:shop /srv/shop/uploads.

The service runs as a different user than you think. The modes may be right for the user you expect, not for the one actually running.

Check

systemctl show -p User shop   # the user it runs as; empty means root
ps -o user= -p $(pgrep -o -f shop)   # the user of the running process

Fix

Give that user access, or set User= in the unit to the account the files belong to.

A secret tightened to root-only that the service reads itself. After chmod 600 the service user can no longer read it.

Check

ls -l FILE   # -rw------- root root: only root can read it

Fix

sudo chown root:shop FILE && sudo chmod 640 FILE.

On Rocky/RHEL, SELinux denies it. Modes and owners are right, but the file's label does not allow this service's domain.

Check

sudo ausearch -m avc -ts recent   # a denial naming the path: this is your cause

Fix

Relabel with semanage fcontext and restorecon (see SELinux denial). Never setenforce 0.

systemd sandboxing (ProtectHome=, ProtectSystem=) hides or locks the path for this service only. It works by hand but not in the service.

Check

systemctl show -p ProtectHome -p ProtectSystem -p ReadWritePaths shop   # yes or strict, and the path is not in ReadWritePaths

Fix

A drop-in with ReadWritePaths=/srv/shop/uploads, then daemon-reload and restart.

The file or a directory on its path is not readable by the web user (www-data on Ubuntu, nginx on Rocky). nginx gets "Permission denied" from the kernel and answers 403.

Check

sudo tail -n 5 /var/log/nginx/error.log   # (13: Permission denied): modes or SELinux
namei -l /var/www/html/report.html   # the first part without r (file) or x (directory) for the web user

Fix

chmod 644 FILE and 755 for folders.

A restrictive umask on the deploy user. New files are created 600 or 640, so only the newest pages fail.

Check

ls -l /var/www/html   # new files without r for others
umask   # as the deploy user: 0077 or 0027

Fix

Set umask 022 in the deploy user's profile or deploy tool, and fix the existing files.

A directory with no index file while autoindex is off. nginx refuses to list the folder and says 403.

Check

sudo tail -n 5 /var/log/nginx/error.log   # directory index of ... is forbidden: this is your cause

Fix

Add the index file, or fix the index directive.

An allow/deny rule in the config matches the client.

Check

sudo tail -n 5 /var/log/nginx/error.log   # access forbidden by rule: this is your cause

Fix

Correct the allow/deny lines, then sudo nginx -t && sudo systemctl reload nginx.

On Rocky/RHEL, an SELinux label. Modes are right, but nginx may only read files labelled for the web.

Check

sudo ausearch -m avc -ts recent   # a denial for nginx on that file

Fix

Relabel (see SELinux denial).

A cleanup job removed the release the link points at. The link survives its target and now points at nothing.

Check

readlink /srv/www/current   # the target
ls -l /srv/www/releases   # that release is not in the list: this is your cause

Fix

Point it at a release that exists: ln -sfn /srv/www/releases/NEW /srv/www/current, or atomically ln -s NEW current.tmp && mv -T current.tmp current.

The deploy wrote a relative target that resolves somewhere else. A relative target is read from the link's own folder, not from where the deploy ran.

Check

readlink /srv/www/current   # does not start with /: relative
find /srv/www -xtype l   # every link whose target is gone

Fix

Recreate the link with an absolute target.

A different PATH. cron uses /usr/bin:/bin; sudo uses secure_path, which on Rocky/RHEL leaves out /usr/local/bin. A tool in /usr/local/bin or your home is found by you, not by them.

Check

command -v backup-uploads.sh   # where your shell finds it
env -i PATH=/usr/bin:/bin /bin/sh -c 'backup-uploads.sh'   # not found here too: this is your cause
sudo grep secure_path /etc/sudoers   # the PATH sudo uses

Fix

Use absolute paths in the job, or set PATH= at the top of the crontab.

The script exists but its #! interpreter does not. dash reports that as "not found" too, which hides the real problem.

Check

head -1 "$(command -v backup-uploads.sh)"   # an interpreter that is not installed

Fix

Install the interpreter or correct the shebang.

The package is not installed on this machine.

Check

command -v backup-uploads.sh   # no output in your shell either

Fix

sudo apt install PACKAGE (Ubuntu suggests which); on Rocky, dnf provides '*/NAME' finds it.

The glob expands to more arguments than the kernel accepts in one command line. It happens only in folders with a huge number of files.

Check

ls /var/lib/php/sessions | wc -l   # hundreds of thousands: this is your cause
getconf ARG_MAX   # the limit in bytes

Fix

Let find do it without a glob: find /var/lib/php/sessions -type f -mmin +1440 -delete, or find DIR -type f -print0 | xargs -0 rm.

A runaway job or loop uses a whole CPU. One process near 100% in top is the sign.

Check

top   # press P: one process at the top near 100%
ps aux --sort=-%cpu | head

Fix

kill PID (SIGTERM) for a stray job, or limit it: sudo systemctl set-property NAME CPUQuota=50%.

Too many workers for the CPUs. Many processes share the CPUs, each modest, but together more than the box has.

Check

uptime && nproc   # load well above the CPU count
vmstat 1 5   # us+sy near 100 and a long r column

Fix

Lower the worker count to fit the CPUs, or add CPUs.

Processes waiting on disk, not CPU. Linux counts them in the load too.

Check

vmstat 1 5   # wa and b high, us low: see I/O wait

Fix

Treat it as disk (see High load, idle CPU).

On a cloud VM, CPU steal or used-up burst credits. The hypervisor gives you less CPU than the VM thinks it has.

Check

vmstat 1 5   # st high: the hypervisor takes the CPU

Fix

Move to a larger or non-burstable instance type.

SIGKILL cannot be caught, so no flush, no rename, no cleanup happened. The .part file and the lock are leftovers.

Check

ls -l /var/lib/batch   # .part and .lock files left behind
cat /var/lib/batch/batchd.lock && ps -p $(cat /var/lib/batch/batchd.lock)   # no process: the lock is stale

Fix

Remove the stale lock only after checking no process owns it, and rerun the job for a complete file. Next time: kill PID (TERM), wait, and use -9 only if it ignores TERM.

The parent never called wait() for its exited children. Until it does, each child stays as a zombie.

Check

ps -eo pid,ppid,stat,cmd | awk '$3 ~ /^Z/'   # the zombies and their parent (PPID)
ps -fp PPID   # the parent that does not reap

Fix

Restart or stop the parent: init then adopts and reaps the zombies. Fix the program to wait() (or ignore SIGCHLD).

In a container, PID 1 is your program. It does not reap orphaned children the way init does, so they pile up.

Check

ps -p 1 -o comm=   # your program, not an init

Fix

docker run --init, or tini as PID 1.

The PID file survived a crash or reboot, and its number now belongs to another process.

Check

cat /run/reportd.pid
ps -fp 1187   # another program: the file is stale

Fix

Remove the stale PID file, then start the service. Do not kill that PID: it is someone else's.

It really is running: a copy started by hand, outside systemd.

Check

pgrep -a reportd   # a real reportd: a second copy is running

Fix

Stop that copy, then start the service through systemctl.

A user or a service hit its process limit (ulimit -u, or TasksMax= for a service). Threads count too.

Check

ulimit -u   # the limit for this user
ps -eo user= | sort | uniq -c | sort -rn | head   # one user near it
systemctl show -p TasksMax -p TasksCurrent SERVICE   # current close to max

Fix

Stop the runaway: systemctl stop SERVICE, or pkill -u USER NAME. Raise the limit only once you know the load is real: TasksMax= in a drop-in, or nproc in /etc/security/limits.d/.

A fork loop, or a program that leaks threads. The count only climbs.

Check

ps -eLf | wc -l   # threads on the box; run it twice

Fix

Stop the program and fix the leak; a higher limit only delays it.

The whole box ran out of PIDs (kernel.pid_max) or of memory (fork: Cannot allocate memory).

Check

cat /proc/sys/kernel/pid_max /proc/loadavg   # 4th field of loadavg: running/total tasks near pid_max
free -h

Fix

Find and stop what holds the tasks or memory.

The machine ran out of memory and the kernel's OOM killer chose this process, usually the biggest.

Check

sudo journalctl -k | grep -i -e 'out of memory' -e oom-kill   # plain Out of memory: the whole box
free -h   # now, not at the time

Fix

Size the heap or workers to the machine (e.g. -Xmx1g), add memory, or move a workload away.

A cgroup limit was hit (MemoryMax=, or a container's memory limit). The box has memory free, but this group may not use more.

Check

sudo journalctl -k | grep -i 'memory cgroup out of memory'   # a line here: a limit
systemctl show reports -p MemoryMax -p MemoryCurrent

Fix

Raise MemoryMax= only if the service really needs more; otherwise find what grew.

No OOM line at all: something else sent SIGKILL, such as a supervisor or a timeout.

Check

sudo journalctl -k | grep -ci oom   # 0: not the OOM killer

Fix

See status=9/KILL.

ExecStart points at a path that does not exist (a typo, or the binary was not installed).

Check

journalctl -u orders-api -n 10   # No such file or directory: this is your cause
ls -l /usr/local/bin/orders-api

Fix

Fix ExecStart (absolute path), then sudo systemctl daemon-reload && sudo systemctl restart orders-api.

The file is not executable, or is a script with CRLF in its shebang.

Check

ls -l /usr/local/bin/orders-api   # no x
file /usr/local/bin/orders-api   # with CRLF line terminators

Fix

chmod 755 the file, or convert the line endings.

A bare command name that is not in systemd's fixed search path. systemd does not use your shell's PATH.

Check

systemctl cat orders-api | grep ExecStart   # a name without /

Fix

Write the absolute path in ExecStart.

On Rocky/RHEL, SELinux blocks a binary with the wrong label, usually one moved from a home folder.

Check

journalctl -u orders-api -n 10   # Permission denied
ls -Z /usr/local/bin/orders-api   # user_home_t: moved from a home folder

Fix

sudo restorecon -v /usr/local/bin/orders-api.

Someone ran systemctl mask, linking the unit to /dev/null.

Check

systemctl is-enabled nginx   # masked
ls -l /etc/systemd/system/nginx.service   # -> /dev/null

Fix

Ask why it was masked, then sudo systemctl unmask nginx && sudo systemctl enable --now nginx.

systemd works from the copy it loaded at the last daemon-reload, not from the file on disk.

Check

systemctl show api -p NeedDaemonReload   # yes: the files changed since systemd loaded them
systemctl cat api   # the files on disk

Fix

sudo systemctl daemon-reload && sudo systemctl restart api.

It was started but never enabled. On Rocky/RHEL packages install services disabled; Ubuntu usually enables them, so habits from one break on the other.

Check

systemctl is-enabled metrics-agent   # disabled: this is your cause

Fix

sudo systemctl enable --now metrics-agent.

The unit has no [Install] section, so enable has nothing to link.

Check

systemctl is-enabled metrics-agent   # static
systemctl show -p WantedBy metrics-agent   # empty: nothing starts it at boot

Fix

Add [Install] with WantedBy=multi-user.target, daemon-reload, enable.

It is enabled, but its dependency fails at boot, so it is skipped or fails too.

Check

journalctl -b -u metrics-agent   # it tried and failed, or a dependency failed

Fix

Fix the dependency the journal names.

Managing system services needs root, or a polkit rule that allows your user.

Check

id   # not root
sudo -l   # whether sudo allows it

Fix

sudo systemctl restart ssh (sshd on Rocky). For CI, a narrow sudoers rule for that exact command.

Over ssh or in a script there is no polkit agent to ask for a password, so it says Access denied instead of prompting.

Check

tty   # not a tty: nothing could prompt

Fix

Use sudo, or a NOPASSWD rule for that exact command.

The service crashed more than StartLimitBurst times within StartLimitIntervalSec.

Check

systemctl status billing   # Result: start-limit-hit
journalctl -u billing -n 50   # the first error, before the restarts, is the real one

Fix

Fix the cause, then sudo systemctl reset-failed billing && sudo systemctl start billing.

Another service owns the port, such as Apache or an old instance.

Check

sudo ss -tlnp 'sport = :80'   # users:(("apache2"...)) or httpd: this is your cause

Fix

sudo systemctl disable --now apache2 (httpd on Rocky), or move one service to another port.

A second copy of the same service is already running.

Check

sudo ss -tlnp 'sport = :80'   # nginx itself owns it
pgrep -a nginx   # more than one master

Fix

Stop the extra copy, then manage it only through systemctl.

A container publishes the port.

Check

sudo ss -tlnp 'sport = :80'   # docker-proxy

Fix

Change that container's -p mapping.

A systemd .socket unit holds it for socket activation.

Check

sudo ss -tlnp 'sport = :80'   # systemd
systemctl list-sockets | grep :80

Fix

sudo systemctl disable --now NAME.socket.

The config is invalid. nginx tests it before starting and exits with an error.

Check

sudo nginx -t   # an error with a file and line number: this is your cause

Fix

Fix the line it names, then start again.

Something the daemon needs fails: the port is in use, a file is missing, or a permission.

Check

journalctl -xeu nginx.service   # the first error line is the cause

Fix

Fix what the journal names (see Address already in use for ports).

On Rocky/RHEL, an SELinux denial stops it.

Check

sudo ausearch -m avc -ts recent   # a denial for nginx

Fix

Fix the label, port type or boolean (see the SELinux entries).

It starts before the service it needs: there is no After= ordering, so systemd starts both at once.

Check

systemctl cat inventory | grep -E 'After|Wants|Requires'   # no postgresql: this is your cause
journalctl -b -u inventory -u postgresql   # inventory failed before postgresql was up

Fix

A drop-in with [Unit] After=postgresql.service Wants=postgresql.service; daemon-reload; reboot to test.

The dependency has started but does not accept connections yet, or runs on another host: After= cannot help.

Check

journalctl -b -u inventory -u postgresql   # inventory failed after postgresql started

Fix

Make the app retry its first connection with backoff.

The unit has no [Install] section with WantedBy=.

Check

systemctl cat NAME   # no [Install] section

Fix

Add [Install] with WantedBy=multi-user.target, daemon-reload, enable.

It is meant to be started by something else, such as a .timer or a .socket.

Check

systemctl list-timers --all | grep NAME   # a timer runs it

Fix

Enable the timer instead: sudo systemctl enable --now NAME.timer.

The package is not installed, or the service has another name here (mysql on Ubuntu, mysqld or mariadb on Rocky; apache2 or httpd).

Check

systemctl list-unit-files | grep -i -e mysql -e maria   # the real name, or nothing

Fix

Use the real name from list-unit-files, or install the package that provides the service.

A unit file was added or renamed without systemctl daemon-reload.

Check

ls /etc/systemd/system /usr/lib/systemd/system | grep -i sql   # the file is there

Fix

sudo systemctl daemon-reload.

A typo, or a user service that needs systemctl --user.

Check

systemctl --user list-unit-files | grep -i NAME

Fix

Correct the name, or use systemctl --user.

A required line is missing, such as ExecStart= (the journal says: Service has no ExecStart=...).

Check

journalctl -b -u worker | tail   # Service has no ExecStart=

Fix

Add the line, then sudo systemctl daemon-reload && sudo systemctl start worker.

A typo in a key or a section name, or a value systemd cannot parse.

Check

systemd-analyze verify /etc/systemd/system/worker.service   # names the file and line
systemctl cat worker

Fix

Fix the line it names, daemon-reload, start.

The binary is built for another CPU: an arm64 build on x86_64, or the other way round.

Check

uname -m   # x86_64 or aarch64
file ./node_exporter   # ELF ... ARM aarch64 on an x86_64 box: this is your cause

Fix

Download the build for your architecture (amd64 for x86_64, arm64 for aarch64).

A script without a #! first line, started by systemd or exec. Without it the kernel does not know what runs it.

Check

head -1 SCRIPT   # must start with #!

Fix

Add a shebang such as #!/bin/bash as the first line.

The download is an archive or an HTML error page, not the program.

Check

file ./node_exporter   # gzip compressed data or HTML document

Fix

Unpack the archive, or download again from the right URL.

In a container, PID 1 is your program. Docker and most containers do not run systemd.

Check

ps -p 1 -o comm=   # bash, node or your app, not systemd

Fix

Run the program in the foreground as the container's command, not with systemctl.

WSL started without systemd.

Check

ps -p 1 -o comm=   # init, not systemd

Fix

Add [boot] and systemd=true to /etc/wsl.conf, run wsl --shutdown in Windows, then reopen it.

/etc/cron.d lines need a user field after the five time fields. Without it, cron takes the command's first word as the user name.

Check

cat /etc/cron.d/db-backup   # five fields then the command, no user: this is your cause
journalctl -u cron -n 20   # bad username (crond on Rocky, or /var/log/cron)

Fix

Add the user: */10 * * * * postgres pg_dump ....

Wrong owner or a writable mode. cron refuses files not owned by root or writable by others.

Check

ls -l /etc/cron.d   # owned by root, not group or world writable

Fix

sudo chown root:root FILE && sudo chmod 644 FILE.

On Ubuntu and Debian, a dot in the file name. cron skips the file without a word, so its jobs never run.

Check

ls /etc/cron.d   # db-backup.cron or similar

Fix

Rename it with letters, digits, - and _ only.

The last line has no newline (printf without \n is the usual source).

Check

tail -c1 file | od -c   # the last character is not \n

Fix

echo >> file, then crontab file.

No -U given: the client uses the Linux user name as the role (root here), and no such role exists.

Check

sudo -u postgres psql -c '\du'   # the roles that exist; root is not among them

Fix

Run the job as postgres: sudo -u postgres pg_dump ..., or user postgres in cron.d. Better: give the job its own role and pass -U.

PGUSER is set to a role that does not exist.

Check

printenv PGUSER PGHOST PGPORT   # what the job inherits

Fix

Unset or correct PGUSER.

It reached another cluster or port where that role was never created.

Check

pg_lsclusters   # Ubuntu: which cluster listens on which port

Fix

Pass the right -p (or -h) for the cluster. Do not create a superuser role named root to silence it.

Services do not read ~/.bashrc, ~/.profile, your exports or /etc/environment.

Check

sudo cat /proc/$(pgrep -o reports)/environ | tr '\0' '\n' | cut -d= -f1   # the names the running process has, without values

Fix

A drop-in with EnvironmentFile=/etc/app/app.env (mode 600), then daemon-reload and restart. Keep secrets out of Environment= in the unit: any user can read it with systemctl show.

The EnvironmentFile= path is wrong. With a leading - systemd skips a missing file silently.

Check

systemctl show reports-api -p EnvironmentFiles   # the files it loads; ignore_errors=yes means a missing one is skipped
ls -l /etc/app/app.env

Fix

Fix the path, daemon-reload and restart.

The variable is set for a different user than the service runs as.

Check

systemctl show -p User reports-api

Fix

Set it in the unit or its EnvironmentFile, not in a user's profile.

The file system is full.

Check

df -h /tmp   # Use% 100 on the file system that holds the path
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail   # the biggest folders

Fix

Delete or compress what you do not need; truncate a runaway log (truncate -s 0) rather than rm it.

Inodes are exhausted: space is left, but no new file can be created.

Check

df -i /tmp   # IUse% 100: inodes, not space

Fix

Remove the small files (see the inodes entry).

Deleted files are still held open. The space returns only when the process closes them.

Check

sudo lsof +L1   # (deleted) files still taking space

Fix

Restart the process that holds them.

Only root's reserve is left (5% on ext4 by default), so normal users fail first.

Check

df -h /   # Avail 0 while Use% shows about 95%

Fix

Free space as above. Lowering the reserve (tune2fs -m) buys minutes, not a fix.

A deleted file is still open by a process. It has no name, so du misses it, but its space stays used.

Check

sudo lsof +L1   # (deleted) files and the process holding them

Fix

sudo systemctl restart SERVICE, or sudo truncate -s 0 /proc/PID/fd/N.

Files are hidden under a mount point: they were written before the disk was mounted over that folder.

Check

sudo mkdir -p /mnt/rootfs && sudo mount -o bind,ro / /mnt/rootfs && sudo du -sh /mnt/rootfs/data   # space under the /data mount point; umount /mnt/rootfs after

Fix

Stop the writer, unmount, move the files, mount again.

du ran without sudo and skipped folders it could not read.

Check

sudo du -xsh /   # compare with df

Fix

Run du with sudo.

Millions of tiny files: sessions, cache, mail queues.

Check

df -i   # IUse% 100
sudo du --inodes -x / 2>/dev/null | sort -n | tail   # the folders with the most files

Fix

sudo find DIR -type f -mmin +1440 -delete for the folder that grew.

Mostly on ext4, which fixes the inode count when the file system is made. XFS (Rocky's default) allocates inodes as needed, so this is rare there.

Check

df -T /   # ext4

Fix

Clean up as above; for a volume that will always hold many small files, make it with more inodes (mkfs.ext4 -i).

ext4 found an error and remounted itself read-only (errors=remount-ro) to protect your data.

Check

sudo dmesg -T | grep -i -e ext4 -e 'read-only'   # Remounting filesystem read-only: this is your cause

Fix

Stop writers, snapshot the volume if you can, umount, run fsck on the device, mount again. Do not force rw on a damaged file system.

The disk went away or returned I/O errors, such as a detached cloud volume.

Check

sudo dmesg -T | grep -i -e 'I/O error' -e xfs   # I/O errors or an XFS shutdown

Fix

Fix the disk or attachment; for XFS run xfs_repair on the unmounted device.

The root file system in emergency mode, or mounted ro in fstab.

Check

findmnt -no OPTIONS -T /data   # ro here: the mount itself is read-only

Fix

In emergency mode: mount -o remount,rw /; otherwise fix the fstab options.

Read-only for one service only: ProtectSystem=strict or ReadOnlyPaths= in its unit.

Check

systemctl show SERVICE -p ProtectSystem -p ReadWritePaths   # strict, and the path is not listed

Fix

Allow the path with ReadWritePaths= in a drop-in.

A process has a file open or its working directory there (your own shell counts).

Check

sudo fuser -vm /data   # the processes using it
sudo lsof +D /data

Fix

Stop the service, cd out, then umount.

Something else uses it: a mount inside it, a swap file or a loop device.

Check

findmnt -R /data; swapon --show; losetup -a   # mounts, swap and loop devices on it

Fix

Unmount inner mounts or swapoff first. Avoid umount -l unless you know what holds it: the writes carry on unseen.

The file system is mounted.

Check

findmnt /dev/nvme1n1p1   # a TARGET: it is mounted

Fix

Unmount it first; for / use a rescue boot or a check at boot.

The partition was not grown; the file system already fills it.

Check

lsblk   # disk bigger than the partition: this is your cause

Fix

sudo growpart /dev/nvme0n1 1 && sudo resize2fs /dev/nvme0n1p1 (xfs_growfs / for XFS; growpart is in cloud-guest-utils on Ubuntu, cloud-utils-growpart on Rocky).

On LVM: the physical and logical volumes were not grown.

Check

sudo pvs; sudo lvs   # PFree or VFree unused

Fix

growpart, then sudo pvresize /dev/nvme0n1p3 && sudo lvextend -r -l +100%FREE VG/LV.

The kernel has not seen the bigger disk yet.

Check

lsblk   # the disk still at the old size

Fix

Rescan it (for a SCSI disk, echo 1 | sudo tee /sys/class/block/sdb/device/rescan) or reboot.

An fstab line for a disk that is missing or has a wrong UUID, without nofail.

Check

journalctl -xb | grep -i -e failed -e 'timed out'   # Timed out waiting for device: this is your cause
blkid   # compare the UUIDs with /etc/fstab
cat /etc/fstab

Fix

mount -o remount,rw /, fix fstab, systemctl daemon-reload, mount -a, then systemctl default.

A file system that fails its check.

Check

journalctl -xb | grep -i fsck   # errors from the check

Fix

Run fsck (ext4) or xfs_repair (XFS) on that unmounted device, then systemctl default.

A typo in the UUID, or the disk was replaced.

Check

sudo blkid   # the real UUIDs
lsblk -f

Fix

Copy the UUID from blkid into /etc/fstab, then sudo mount -a.

The disk was formatted again (mkfs gives it a new UUID) or is not attached.

Check

lsblk -f   # the disk is missing, or has another UUID

Fix

Attach it, or update fstab with the new UUID.

The device has no file system yet (a new volume).

Check

lsblk -f   # FSTYPE empty
sudo blkid /dev/sdc   # no output

Fix

Only for a new, empty volume: sudo mkfs.ext4 /dev/sdc (this erases it).

You mounted the whole disk (/dev/sdc) and the file system is on a partition (/dev/sdc1).

Check

lsblk -f   # the FSTYPE is on sdc1

Fix

Mount the partition and type that lsblk -f shows.

The helper is missing: nfs-common (nfs-utils on Rocky) for NFS, cifs-utils for SMB.

Check

ls /sbin/mount.nfs /sbin/mount.cifs   # missing

Fix

sudo apt install nfs-common or cifs-utils (dnf install nfs-utils on Rocky).

A misspelled mount option.

Check

sudo dmesg | tail   # unknown parameter or bad option

Fix

Fix the option.

The log folder is writable by a non-root group or by everyone.

Check

ls -ld /var/log/orders   # w for group or others
sudo logrotate -d /etc/logrotate.d/orders   # the same message in the dry run

Fix

Add su orders orders to the rule, or chmod 755 the folder; then sudo logrotate -f to test.

The rule renames the file (create) while the program keeps it open; compress then deletes it.

Check

sudo lsof +L1   # .log.1 (deleted) held by the program

Fix

Restart the program once to release the space.

No copytruncate and no postrotate reload, so the program is never told to reopen.

Check

grep -E 'copytruncate|postrotate' /etc/logrotate.d/*

Fix

Add postrotate with the program's reload command, or copytruncate (it can lose lines written during the copy).

SystemMaxUse= was raised somewhere.

Check

systemd-analyze cat-config systemd/journald.conf | grep -i -e maxuse -e keepfree   # the settings in effect, drop-ins included

Fix

sudo journalctl --vacuum-size=500M, and SystemMaxUse=500M in a drop-in under /etc/systemd/journald.conf.d/, then sudo systemctl restart systemd-journald.

Journal folders from another machine ID, such as an image cloned without a new /etc/machine-id, that journald does not clean.

Check

ls /var/log/journal; cat /etc/machine-id   # folders that are not this machine's ID

Fix

Remove those folders once you no longer need their logs.

The export was restored, moved or recreated; the client's handles point at nothing.

Check

findmnt -t nfs4
df -h 2>&1 | grep -i stale   # the stale mount

Fix

sudo umount -l /mnt/shared && sudo mount /mnt/shared.

The resolver answered that the name does not exist (NXDOMAIN): a typo, a missing search domain, or a private zone this resolver cannot see.

Check

dig api.payments.example   # status: NXDOMAIN: no such name there
dig @10.0.0.2 api.payments.example   # the server you expect answers it: the wrong resolver is in use

Fix

Fix the name or search domain, or use the resolver that serves that zone.

No answer at all: a wrong or unreachable nameserver, or systemd-resolved down.

Check

resolvectl status   # Ubuntu: the real servers behind 127.0.0.53
dig api.payments.example   # timed out: no server answers

Fix

Ubuntu: set nameservers in netplan, then sudo netplan apply (/etc/resolv.conf is generated and edits are lost). Rocky: sudo nmcli connection modify NAME ipv4.dns 10.0.0.2 && sudo nmcli connection up NAME.

No route or a firewall to the DNS server.

Check

dig @10.0.0.2 api.payments.example   # timed out, while other traffic works

Fix

Open UDP and TCP 53 to the resolver, or fix the route.

A stale line in /etc/hosts, which is read before DNS.

Check

getent hosts db.internal   # what programs get
dig +short db.internal   # what DNS says
grep db.internal /etc/hosts   # the old address: this is your cause

Fix

Remove the stale /etc/hosts line.

A cache: systemd-resolved or nscd.

Check

getent hosts db.internal   # old address, but not in /etc/hosts

Fix

sudo resolvectl flush-caches.

The app resolved the name once at start (a connection pool, or the JVM's DNS cache).

Check

getent hosts db.internal   # already right, yet the app still uses the old address

Fix

Restart the app, and set a DNS TTL it honours.

No default route.

Check

ip route   # no default line
ip route get 203.0.113.10   # Network is unreachable

Fix

sudo ip route add default via 10.0.1.1 dev eth0, and persist it in netplan (Ubuntu) or nmcli (Rocky).

The interface is down, or got no address from DHCP.

Check

ip -br a   # DOWN, or no IPv4 address

Fix

Bring it up and fix DHCP or the static address.

The name resolves to IPv6 and there is no IPv6 route.

Check

ip -6 route   # no default for IPv6

Fix

Use -4 for now, or add the IPv6 route.

The gateway is not on any directly connected subnet.

Check

ip -br a   # your address and prefix
ip route   # the connected subnets

Fix

Use a gateway inside your subnet (usually .1).

A firewall drops the packets: ufw or firewalld on the server, a cloud security group or network ACL on the way.

Check

sudo tcpdump -ni any tcp port 8080   # on the server while you retry: no SYN: dropped on the way; SYN but no reply: the host firewall
sudo ufw status verbose   # Ubuntu; sudo firewall-cmd --list-all on Rocky

Fix

Host firewall: sudo ufw allow from 10.0.1.0/24 to any port 8080 proto tcp. Security group: allow the port from the client's subnet only, not 0.0.0.0/0.

The address is wrong or the host is down.

Check

getent hosts web-01   # the address the client uses
curl -v localhost:8080   # on the server: works locally, so the service is up

Fix

Use the right address, or bring the host up.

Replies take another route back.

Check

sudo tcpdump -ni any tcp port 8080   # SYN-ACK leaves on another interface

Fix

Fix the return route on the server.

Nothing listens on that address and port: the service is down or on another port.

Check

sudo ss -tlnp 'sport = :8080'   # no line: nothing listens
systemctl status SERVICE

Fix

Start the service, or fix the port the client uses.

The service listens on 127.0.0.1 only, so it refuses connections from other hosts.

Check

sudo ss -tlnp 'sport = :8080'   # 127.0.0.1:8080 or [::1]:8080: local only

Fix

Bind it to the private IP (0.0.0.0 only if it must be reachable everywhere) and restart.

localhost resolved to ::1 while the service listens on 127.0.0.1 only.

Check

curl -v http://localhost:8080   # Trying [::1]:8080 then refused

Fix

Use 127.0.0.1 in the client, or make the service listen on both.

A firewall REJECT rule. It looks the same as a closed port.

Check

sudo ss -tlnp 'sport = :8080'   # it does listen
sudo nft list ruleset | grep -i reject

Fix

Allow the port in the firewall.

The new hostname is not in /etc/hosts.

Check

hostname
getent hosts $(hostname)   # no output: this is your cause

Fix

sudo sed -i 's/old-name/new-name/' /etc/hosts.

The system clock is behind.

Check

timedatectl   # System clock synchronized: no, and a wrong time
chronyc tracking   # Rocky (chrony): the offset

Fix

Ubuntu: sudo timedatectl set-ntp true (systemd-timesyncd). Rocky: sudo systemctl enable --now chronyd; sudo chronyc makestep for a large offset.

The certificate was issued minutes ago and this clock is slightly behind.

Check

echo | openssl s_client -connect HOST:443 -servername HOST 2>/dev/null | openssl x509 -noout -startdate   # compare with date -u

Fix

Sync the clock as above.

No rule names the user or one of their groups (sudo on Ubuntu, wheel on Rocky).

Check

groups alice   # no sudo or wheel
sudo -l -U alice   # as an admin: not allowed

Fix

sudo usermod -aG sudo alice (wheel on Rocky), or a narrow rule in /etc/sudoers.d via visudo.

Just added to the group, but the session still has the old groups.

Check

groups alice   # lists sudo
id   # in alice's session: sudo is missing

Fix

Log out and in again.

The rule names a different command line: another path (/bin or /usr/bin), unit name (orders or orders.service) or arguments.

Check

sudo -l -U deploy   # the exact commands allowed; compare letter by letter

Fix

Update the rule to the exact command, and check it with visudo -c.

A later rule for the same user overrides it: in sudoers the last match wins.

Check

sudo grep -rn deploy /etc/sudoers /etc/sudoers.d   # read the order

Fix

Remove or reorder the later rule.

sudo needs a password and there is no terminal to ask on.

Check

sudo -l -U deploy   # the rule has no NOPASSWD
sudo -n true && echo no password needed   # as deploy: fails

Fix

Allow the exact command without a password, in a drop-in edited with sudo visudo -f /etc/sudoers.d/deploy: deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart orders. For one interactive run: ssh -t HOST sudo COMMAND. Never pipe a password into sudo -S.

A syntax error in any sudoers file (missing colon after NOPASSWD, a typo).

Check

visudo -c   # as root, or pkexec visudo -c: names the file and line

Fix

Fix the file as root (console, rescue boot, or pkexec), then visudo -c.

A sudoers file with the wrong mode or owner.

Check

ls -l /etc/sudoers.d   # not 440 root root

Fix

chmod 440 and chown root:root the file, then visudo -c.

The private key is readable by group or others.

Check

ls -l ~/.ssh   # -rw-r--r-- on the key: this is your cause

Fix

chmod 600 ~/.ssh/id_ed25519, and consider the key exposed if others could log in to this machine.

Wrong user for the image: ubuntu on Ubuntu cloud images, rocky on Rocky.

Check

ssh -v deploy@db-01 2>&1 | grep 'Authenticating to'   # the user and host you really used

Fix

Log in as the image's user first.

The client did not offer the key (wrong file, too-open mode, another key in the agent).

Check

ssh -v deploy@db-01 2>&1 | grep -e Offering -e 'Authentications that can continue'   # no Offering line for your key: the client never sent it

Fix

Name the key: ssh -i ~/.ssh/id_ed25519, or IdentityFile in ~/.ssh/config; key mode 600.

The key is not in that user's authorized_keys, or StrictModes rejected the file.

Check

sudo journalctl -u ssh -n 20   # on the server (sshd on Rocky): bad ownership or modes

Fix

Add the public key; fix modes: ~/.ssh 700, authorized_keys 600.

On Rocky/RHEL, authorized_keys has the wrong SELinux label.

Check

ls -Z /home/deploy/.ssh/authorized_keys   # not ssh_home_t

Fix

sudo restorecon -Rv /home/deploy/.ssh.

The server refuses the key type: old ssh-rsa SHA-1 signatures are off in recent OpenSSH.

Check

sudo journalctl -u ssh -n 20   # a refused signature algorithm

Fix

Make an ed25519 key rather than re-enabling ssh-rsa on the server.

Home, ~/.ssh or authorized_keys is writable by group or others, or owned by someone else.

Check

ls -ld /home/deploy /home/deploy/.ssh   # w for group or others, or another owner
ls -l /home/deploy/.ssh

Fix

chmod 755 home (or 750), 700 ~/.ssh, 600 authorized_keys, all owned by the user.

The server was rebuilt or reinstalled, or its address now belongs to another machine.

Check

ssh-keygen -F db-01   # the key you saved
ssh-keyscan db-01 | ssh-keygen -lf -   # the key it shows now; compare with the server's console

Fix

Once you know the change is expected: ssh-keygen -R db-01 (and its IP address), then connect and accept the new key.

Someone is intercepting the connection: rare, but this is what the check is for.

Check

ssh-keyscan db-01 | ssh-keygen -lf -   # differs from the fingerprint on the server's console

Fix

Do not connect; investigate the network path.

Created with the default umask, or copied without setting the mode.

Check

ls -l /etc/payments/payments.env   # -rw-r--r--
systemctl show -p User payments   # who needs to read it

Fix

sudo chown root:payments FILE && sudo chmod 640 FILE, then restart the service to prove it still reads it.

It is loaded by systemd itself with EnvironmentFile=, so the service user does not need to read it at all.

Check

systemctl cat payments | grep -i environmentfile   # it is listed

Fix

systemd reads it as root, so root:root mode 600 is enough.

bind 0.0.0.0 in its config.

Check

sudo ss -tlnp   # 0.0.0.0 or [::] in Local Address: every interface

Fix

Bind to 127.0.0.1 (or the private interface) and restart.

protected-mode or auth turned off, so nothing stops a stranger once they reach it.

Check

grep -E '^(bind|protected-mode|requirepass)' /etc/redis/redis.conf   # values only you see; do not paste them

Fix

Require auth, keep protected-mode on, and keep a firewall as a second layer.

usermod -G without -a replaced every supplementary group.

Check

id alice   # only the groups you just named
grep alice /etc/group

Fix

Add back what was lost: sudo usermod -aG docker,sudo alice.

The account's shell is /usr/sbin/nologin (/sbin/nologin on Rocky).

Check

getent passwd www-data   # the last field is the shell

Fix

Run the command as that user instead: sudo -u www-data COMMAND, or for a shell once sudo -u www-data bash.

Password logins are enabled on an internet-facing sshd.

Check

sudo sshd -T | grep -i passwordauthentication   # yes: the value in effect
sudo grep 'Accepted password' /var/log/auth.log   # /var/log/secure on Rocky; any line: someone got in, treat it as an incident

Fix

PasswordAuthentication no, sudo sshd -t, sudo systemctl reload ssh (sshd on Rocky).

A drop-in in sshd_config.d (such as 50-cloud-init.conf) overrides the main file: the first value read wins.

Check

grep -ri passwordauthentication /etc/ssh/sshd_config.d/   # a yes that wins

Fix

Set it in that drop-in, or in one that sorts first. Add fail2ban: on Ubuntu its sshd jail is on after install; on Rocky it comes from EPEL and the jail must be enabled.

Renewal stopped: the timer is disabled or renewal fails.

Check

sudo certbot certificates   # expired on disk too
systemctl list-timers --all | grep -i certbot   # missing or never ran

Fix

sudo certbot renew && sudo systemctl reload nginx, then enable the timer your install has: certbot.timer (apt), certbot-renew.timer (EPEL on Rocky) or snap.certbot.renew.timer (snap).

Renewed, but the server was not reloaded, or it serves a copy of the old file.

Check

echo | openssl s_client -connect shop.example:443 -servername shop.example 2>/dev/null | openssl x509 -noout -enddate   # what clients get
sudo openssl x509 -enddate -noout -in /etc/letsencrypt/live/shop.example/fullchain.pem   # newer than the served date: this is your cause

Fix

Reload the server, and point it at /etc/letsencrypt/live/..., not a copy.

The server sends an expired intermediate certificate.

Check

echo | openssl s_client -connect shop.example:443 -servername shop.example -showcerts 2>/dev/null | grep -A1 ' s:'   # the chain it sends

Fix

Serve fullchain.pem, not an old chain file.

The client's clock is ahead.

Check

timedatectl   # on the client

Fix

Sync the client's clock.

Another apt or dpkg is running: often unattended-upgrades on a new server.

Check

ps -fp 2810   # the process holding the lock
systemctl status unattended-upgrades

Fix

Wait for it to finish, then retry. Never delete lock files: the lock goes when its process exits.

cloud-init is still installing packages on first boot.

Check

cloud-init status   # running: first-boot setup is not done

Fix

cloud-init status --wait, then retry.

An install was killed, or lock files were removed while it ran.

Check

sudo dpkg --audit   # the half-configured packages

Fix

sudo dpkg --configure -a, then sudo apt -f install.

The package lists are old or were never downloaded (a fresh image).

Check

apt-cache policy nodejs   # no candidate, or nothing at all

Fix

sudo apt update, then install again.

The name is wrong, or different on this release.

Check

apt-cache search nodejs   # the real names

Fix

Use the real name from apt-cache search.

It lives in a repository you have not added (universe, or the vendor's own).

Check

apt-cache policy   # universe or the vendor's repository missing

Fix

sudo add-apt-repository universe, or add the vendor's repository.

The repository's signing key is not installed, or the vendor rotated it.

Check

gpg --show-keys /etc/apt/keyrings/vendor.gpg   # no key ends with the NO_PUBKEY id

Fix

Download the vendor's current key, check its fingerprint against the one they publish, and sudo gpg --dearmor -o /etc/apt/keyrings/vendor.gpg.

The sources line points at a keyring (signed-by=) that is missing or holds the old key.

Check

grep -ri signed-by /etc/apt/sources.list /etc/apt/sources.list.d/   # Signed-By: in .sources files
ls -l /etc/apt/keyrings /usr/share/keyrings   # the file it names is there?

Fix

sudo install -d -m 0755 /etc/apt/keyrings, put the key there, and point the line at it: deb [signed-by=/etc/apt/keyrings/vendor.gpg] URL SUITE main, then sudo apt update.

The system clock is in the past (often a restored snapshot or a VM without NTP).

Check

timedatectl   # the wrong date, System clock synchronized: no

Fix

sudo timedatectl set-ntp true.

A kernel or libc update only applies after a reboot.

Check

uname -r   # the running kernel, older than the installed one
cat /run/reboot-required.pkgs   # Ubuntu: what asked for it
sudo dnf needs-restarting -r   # Rocky: whether a reboot is needed

Fix

Reboot in a window; check systemctl --failed afterwards.

The setting was changed with sysctl -w and never persisted.

Check

grep -r ip_forward /etc/sysctl.conf /etc/sysctl.d   # nothing: this is your cause

Fix

echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-forward.conf && sudo sysctl --system.

A file read later sets it back: sysctl.d files apply in name order, and the last value wins.

Check

systemd-analyze cat-config sysctl.d | grep -e '^# /' -e ip_forward   # every file in order: the last ip_forward wins

Fix

Remove the later line (99-sysctl.conf is /etc/sysctl.conf).

The package lives in EPEL, which is not enabled.

Check

dnf repolist   # no epel

Fix

sudo dnf install -y epel-release, then install again.

It needs CRB (PowerTools on Rocky 8), which is off by default.

Check

dnf repolist   # epel listed, crb not

Fix

sudo dnf config-manager --set-enabled crb (powertools on Rocky 8).

A typo, or a package name that differs from Debian's.

Check

dnf search htop
dnf provides '*/htop'   # the package that ships the file

Fix

Install the name dnf shows.

An exclude= line in dnf.conf or a repo file hides it.

Check

grep -r exclude /etc/dnf/dnf.conf /etc/yum.repos.d/

Fix

Change it only once you know why it was added.

Files carry the wrong SELinux label. mv keeps the label from where the file was, so files moved from a home folder stay user_home_t.

Check

ls -Z /srv/shop   # user_home_t, not httpd_sys_content_t
sudo ausearch -m avc -ts recent | audit2why   # says a label fixes it

Fix

sudo restorecon -Rv /srv/shop if the path has a default; otherwise see the next cause.

A path outside the policy's defaults (/srv, /data), so restorecon has no web label to apply.

Check

matchpathcon /srv/shop   # var_t or default_t

Fix

sudo semanage fcontext -a -t httpd_sys_content_t '/srv/shop(/.*)?' && sudo restorecon -Rv /srv/shop.

A boolean is off, such as httpd_can_network_connect for nginx proxying to a backend (a 502).

Check

sudo ausearch -m avc -ts recent | audit2why   # names the boolean (policycoreutils-python-utils)

Fix

sudo setsebool -P httpd_can_network_connect 1. Never setenforce 0, and do not feed denials to audit2allow blindly: it allows whatever was denied.

SELinux allows httpd_t to bind only http_port_t ports, and 8090 is not one.

Check

sudo ausearch -m avc -ts recent   # name_bind denied: SELinux
sudo semanage port -l | grep -w 8090   # not listed, or under another type

Fix

sudo semanage port -a -t http_port_t -p tcp 8090 (semanage is in policycoreutils-python-utils). If it says the port is already defined, use -m instead of -a.

A non-root process on a port below 1024 (any distribution).

Check

ps -o user= -p $(pgrep -o nginx)   # not root

Fix

Use a port above 1024 behind a proxy, or AmbientCapabilities=CAP_NET_BIND_SERVICE in the unit.

The port is not open in the zone the interface is in.

Check

sudo firewall-cmd --get-active-zones   # the zone the interface is in
sudo firewall-cmd --zone=public --list-all   # 8080/tcp not in ports or services

Fix

sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp && sudo firewall-cmd --reload. For a known service, --add-service=http reads better.

Opened at runtime only, lost after --reload or reboot.

Check

sudo firewall-cmd --permanent --list-ports   # missing here while --list-ports shows it

Fix

sudo firewall-cmd --runtime-to-permanent.

Open on the host, but a cloud security group drops it: that times out instead of No route to host.

Check

sudo tcpdump -ni any tcp port 8080   # no SYN arrives

Fix

Allow the port from the client's subnet in the security group. Do not stop firewalld to make it work.

The file system is XFS, Rocky's default, not ext4.

Check

df -hT /   # xfs

Fix

sudo xfs_growfs /. Next time use lvextend -r, which picks the right tool.

The volume group has less free space than requested.

Check

sudo vgs   # VFree
sudo pvs   # which disks have room

Fix

Ask for less (-l +100%FREE), or add a disk: pvcreate, vgextend, then lvextend -r.

A hard NFS mount whose server is down or unreachable.

Check

sudo dmesg | grep -i nfs   # not responding, still trying
ps -eo pid,stat,cmd | grep ' D'   # processes stuck on the mount
df -hl   # skips network mounts, so it does not hang

Fix

Fix the server or the network; to free the client, sudo umount -f -l /mnt/reports, then fix or remove its fstab line.

A leak in the program.

Check

ps -o pid,rss,etime,cmd -p PID; sleep 60; ps -o pid,rss,etime,cmd -p PID   # RSS up every time

Fix

Contain it: sudo systemctl set-property NAME MemoryMax=1G with Restart=always, and report the leak with the numbers.

Not a leak: a heap or cache that grows to its cap (a JVM up to -Xmx) and then stays flat.

Check

ps aux --sort=-rss | head   # watch for an hour: it levels off

Fix

Set the cap to fit the machine.

mkswap was not run on the file.

Check

file /swapfile   # data, not swap file

Fix

sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile, then a line in /etc/fstab.

The swap file is readable by others.

Check

ls -l /swapfile   # -rw-r--r--

Fix

sudo chmod 600 /swapfile.

A saturated disk: a backup, rsync or updatedb is reading or writing as fast as it can.

Check

iostat -x 1 5   # %util near 100 and a high await (sysstat)
sudo iotop -o   # which process

Fix

sudo ionice -c3 -p PID for the I/O hog, or reschedule it.

A cloud volume at its IOPS or throughput limit (burst credits used up).

Check

iostat -x 1 5   # flat r/s+w/s at the volume's limit with modest %util

Fix

Raise the volume's IOPS or throughput, or move the load.

A hung network mount.

Check

ps -eo pid,stat,cmd | grep ' D'   # stuck on an NFS path

Fix

See NFS server not responding.

The working set does not fit in RAM.

Check

vmstat 1 5   # si and so both high
free -h   # little available
top -o %MEM   # the biggest user

Fix

Cap the hog: sudo systemctl set-property NAME MemoryMax=1G, or stop it; add RAM if the load is real.

The kernel OOM killer or a MemoryMax= limit.

Check

systemctl show reports -p Result -p NRestarts   # Result=oom-kill
sudo journalctl -k | grep -i oom

Fix

Fit the service to memory (heap, workers), or give it more.

systemd-oomd under memory pressure, where it is enabled.

Check

journalctl -u systemd-oomd   # Killed ... due to memory pressure

Fix

Lower the memory use, or tune oomd for that slice.

A stop that ran past TimeoutStopSec=: systemd sends SIGKILL.

Check

systemctl show reports -p Result   # Result=timeout

Fix

Find why stopping hangs before raising TimeoutStopSec=.

Someone's kill -9.

Check

systemctl show reports -p Result   # Result=signal
journalctl -u reports | grep -i kill

Fix

Find who sent it.

The service's soft open-files limit (1024 by default) is too low for its load.

Check

cat /proc/$(pgrep -o gateway)/limits | grep 'open files'   # the limit the process really has
ls /proc/$(pgrep -o gateway)/fd | wc -l   # at the limit

Fix

Drop-in [Service] LimitNOFILE=65535, daemon-reload, restart.

A file descriptor leak.

Check

ls /proc/$(pgrep -o gateway)/fd | wc -l   # keeps climbing at low traffic: a leak

Fix

Fix the program; a bigger limit only delays it.

The program's own cap, such as nginx worker_rlimit_nofile.

Check

grep -r worker_rlimit_nofile /etc/nginx/

Fix

Raise that setting too.