Bash errors, explained

A typo in the command name. Bash runs exactly the word you typed, so ehco is looked up as a program and not found.

Check

type -a <name>   # prints nothing: no function, builtin or program by that name anywhere
command -v <correct-name>   # prints a path: the program exists and the name was wrong

Fix

Correct the name.

The program is installed outside $PATH (for example in /usr/local/bin when cron runs the script). The search only looks in the directories listed in $PATH, so a program elsewhere is invisible.

Check

echo "$PATH"   # the directories searched; compare with where the program lives
ls -l /usr/local/bin/<name> /opt/*/bin/<name> 2>/dev/null   # the program exists, just not in PATH

Fix

Call it by its full path, or set PATH at the top of the script: PATH=/usr/local/bin:/usr/bin:/bin. If it is not installed at all, install it.

Spaces around = in an assignment: NAME = value. With spaces, bash reads NAME as a command and = and value as its arguments.

Check

grep -nE '^[[:space:]]*[A-Za-z_][A-Za-z0-9_]* +=' <script>   # a line here: an assignment with spaces

Fix

Remove the spaces: NAME=value.

A script in the current directory run without ./. The current directory is not in $PATH, so deploy.sh is not found even when it is right there.

Check

ls -l ./<name>   # the file is here: run it as ./<name>

Fix

Run it as ./deploy.sh, or by its full path.

A function defined in your terminal, which child scripts never see. Functions live in the shell that defined them; a script starts a new bash without them.

Check

type <name>   # 'is a function' in your terminal; run the same in the script with bash -c 'type <name>'

Fix

Put the function in a file and source it from the script, or turn it into a script on $PATH.

The command runs under sudo, which replaces your PATH with its secure_path. The message then reads sudo: <name>: command not found.

Check

sudo printenv PATH   # the PATH sudo really uses; the program's directory is missing from it

Fix

Call the program by its full path under sudo, rather than widening secure_path.

The script has no execute bit (-rw-r--r--). The kernel only runs files marked executable for you.

Check

ls -l <script>   # no x in -rw-r--r--: this is your cause

Fix

chmod +x script.sh, and commit it with the bit: git update-index --chmod=+x script.sh.

The file system is mounted noexec, as /tmp is on many hardened systems. On such a mount the kernel refuses to run any file, whatever its mode.

Check

findmnt -no OPTIONS --target <script>   # noexec in the list: this is your cause

Fix

Run it through the interpreter, bash script.sh, or move the script to a normal directory. Do not remount /tmp without noexec to make it work.

A directory on the path cannot be entered. To reach a file you need x on every directory above it, so a locked parent gives the same error.

Check

namei -l <script>   # a directory without x for you on one of the lines: this is your cause

Fix

Ask the owner to grant access to that directory, or run the script from a place you can reach. Do not open it up with chmod 777.

A fixed file name in the shared /tmp that another user's job created first. Your script then tries to write or run a file it does not own.

Check

ls -l /tmp/<name>   # owned by another user: this is your cause

Fix

Create temporary files with mktemp instead of a fixed name.

Writing to a file the shell cannot open, such as sudo echo x > /etc/file. The redirection is done by your shell, as you, before sudo runs; the message then names the file, not the script.

Check

ls -l /etc/file   # owned by root, not writable by you: this is your cause

Fix

Let a root process open the file: echo x | sudo tee /etc/file > /dev/null.

A relative path, run from cron or another directory. ./lib/common.sh means "in the current directory", which is wherever the script was started from.

Check

pwd   # in the script: not the script's directory means this is your cause
ls -l -- "<path>"   # works from the script's directory, fails from here

Fix

Build paths from the script's directory: here=$(dirname "${BASH_SOURCE[0]}"); source "$here/lib/common.sh", or use absolute paths.

An unquoted path with spaces is split into several arguments. cat $file with My Report.txt asks for My and Report.txt, which do not exist.

Check

printf '[%s]\n' $file   # more than one line: the path was split

Fix

Quote it: cat -- "$file".

A carriage return at the end of the name, from a Windows file. The name really is report.txt\r, which is not the file on disk.

Check

printf '%q\n' "<path>"   # ends in $'\r' or shows other hidden characters: this is your cause

Fix

Strip \r from the data file: sed -i 's/\r$//' list.txt.

The parent directory of a redirection does not exist. > /var/log/app/run.log cannot create the file when /var/log/app is missing.

Check

ls -ld "$(dirname "<path>")"   # No such file or directory: create the directory

Fix

Create it first: mkdir -p /var/log/app.

A glob that matched nothing is passed on literally. With no .log files, *.log stays as the text *.log, which is not a file.

Check

echo *.log   # prints *.log itself: nothing matched

Fix

Use shopt -s nullglob before looping over globs, so an empty match gives no words.

The file was saved with CRLF line endings. The first line becomes #!/bin/bash\r, and there is no program called bash\r.

Check

head -1 <script> | cat -A   # ^M$ at the end: this is your cause

Fix

Convert the line endings: sed -i 's/\r$//' script.sh or dos2unix script.sh.

The #! line names an interpreter that is not installed, such as /bin/zsh, or has a typo in its path. The kernel looks for exactly that path.

Check

head -1 <script>   # the path after #!
ls -l <that path>   # No such file or directory: install it or fix the line

Fix

Install the interpreter, or point #! at one that exists: #!/usr/bin/env bash.

A program (not a script) whose dynamic loader is missing, such as a glibc build on Alpine (musl). The binary names its loader inside it, and that file does not exist here.

Check

file <program>   # 'interpreter /lib64/ld-linux-x86-64.so.2': check that this loader exists with ls -l

Fix

Use the build for this system's C library, such as a musl or static build on Alpine.

The file was edited on Windows or copied from a web page. Those save lines with CRLF endings, and bash only strips the LF.

Check

cat -A <script> | head   # ^M$ at the end of lines: this is your cause
file <script>   # 'with CRLF line terminators' says the same

Fix

sed -i 's/\r$//' script.sh, then run it again. Add *.sh text eol=lf to .gitattributes.

No #! line, so /bin/sh runs it. Without one, the program that started it decides which shell runs it, and that is usually sh.

Check

head -1 <script>   # no #! line: this is your cause
ls -l /bin/sh   # -> dash: sh is not bash here

Fix

Start the script with #!/usr/bin/env bash and make it executable.

Running it as sh script.sh. Naming the shell on the command line ignores the #! line. Docker's shell-form CMD and some cron lines do the same.

Check

grep -rn 'sh .*<script>' crontab Dockerfile 2>/dev/null   # sh <script> in how it is started

Fix

Run the file directly (./script.sh), or use bash script.sh. In a Dockerfile use the exec form: CMD ["./script.sh"].

#!/bin/sh on a script that uses bash features. The line asks for sh, and sh is not bash.

Check

shellcheck --shell=sh <script>   # lists the bash-only lines

Fix

Change the line to #!/usr/bin/env bash, or rewrite those lines in POSIX sh.

Tens of thousands of matching files. Every file name becomes one argument.

Check

ls | wc -l   # tens of thousands: this is your cause
getconf ARG_MAX   # the limit, in bytes

Fix

Let find do it: find . -name '*.log' -delete, or find . -name '*.log' -print0 | xargs -0 rm --.

A very large environment, which counts against the same limit. Even a short command fails when the environment is huge.

Check

env | wc -c   # size of the environment in bytes, without printing it; a large share of ARG_MAX is your cause

Fix

Find what fills it (often a variable that holds a whole file) and stop exporting it.

One argument longer than 128 KiB on Linux (MAX_ARG_STRLEN), such as a whole file passed with "$(cat file)". Each single argument has its own limit, separate from the total.

Check

wc -c < <file>   # over 131072: too big for one argument

Fix

Pass the file name or feed it on standard input instead: cmd < file.

A binary for another architecture: amd64 on an arm64 host, or the reverse, such as an image built on an Apple silicon Mac and run on an x86 server.

Check

file <program>   # ARM aarch64 or x86-64: compare with the next line
uname -m   # x86_64 or aarch64: a mismatch is your cause

Fix

Use the build for this machine's architecture, or build images for the target with docker buildx build --platform linux/amd64.

The download saved an HTML error page or an archive instead of the program.

Check

head -c 64 <program> | od -c | head -3   # <html or a gzip header instead of ELF: this is your cause

Fix

Download it again with curl -fL, and check it against its published checksum.

A container entrypoint script with no #! line, or a blank line or byte-order mark before it. Docker runs the entrypoint directly, with no shell to fall back on.

Check

head -c 64 <script> | od -c | head -3   # the first two bytes are not # !

Fix

Make #!/usr/bin/env bash the very first line.

#!/bin/bash runs the system bash 3.2 even when a newer bash is installed.

Check

head -1 <script>   # #!/bin/bash: the system 3.2
/bin/bash --version | head -1   # version 3.2: this is your cause

Fix

Install a current bash (brew install bash) and start the script with #!/usr/bin/env bash.

A newer bash from Homebrew is in $PATH, but the script is run as bash script.sh from a shell where it is not.

Check

type -a bash   # /bin/bash listed first in that shell

Fix

Put Homebrew's bin directory first in PATH, or run the script directly. If it must run on a stock Mac, keep to bash 3.2 features.

Logs or old backups filled the disk.

Check

df -h <path>   # Use% 100
du -xh --max-depth=1 <path> | sort -h | tail   # the directory that holds the space

Fix

Rotate or compress logs and remove old backups, then keep a fixed number (scripts/backup_rotate.sh -k).

A deleted file is still open in a running process, so its space is not freed. df counts it, du does not.

Check

lsof +L1   # deleted files still open, with their size and process

Fix

Restart the process that holds it (scripts/deleted_open_files.sh lists them), or truncate it through /proc/<pid>/fd/<n>.

Millions of small files used up the inodes while df -h still shows free space.

Check

df -i <path>   # IUse% 100: inodes, not bytes

Fix

Find the directory with the many small files (du --inodes -x <path> | sort -n | tail) and clean it up.

On ext4, the blocks reserved for root (5% by default): Avail is 0 for other users while Used is below Size.

Check

df -h <path>   # Avail 0 but Used below Size: the reserved blocks

Fix

Free space as above. Do not just lower the reservation; it is what lets root recover a full disk.

A large write to a small /tmp.

Check

df -h /tmp   # a small tmpfs

Fix

Write large temporary files elsewhere: TMPDIR=/var/tmp.

A missing then, do, ; or ;;. Without then, bash is still reading the condition when it reaches fi, which cannot appear there.

Check

bash -n <script>   # repeats the error without running anything
sed -n '<line-2>,<line>p' <script>   # look for the missing keyword just before the named line

Fix

Add the missing keyword or separator.

A missing space, as in if [$x = 1], or } without a ; before it. Bash needs spaces to see [ and ] as words, and a ; or newline before a closing }.

Check

shellcheck <script>   # names the missing space or separator on that line

Fix

Add the space or the ;: if [ "$x" = 1 ]; then, { cmd; }.

Windows line endings. The token is then shown as $'do\r' or similar: the \r makes do an unknown word.

Check

cat -A <script> | sed -n '1,12p'   # ^M$ at line ends: this is your cause

Fix

sed -i 's/\r$//' script.sh.

An unquoted ( or ) in an argument, such as echo (beta). Parentheses are shell syntax, so bash reads them as the start of a subshell in the wrong place.

Check

grep -n '[^$]([^)]*)' <script>   # an unquoted ( in a command's arguments

Fix

Quote it: echo "(beta)".

A missing fi, done, esac or }. Bash keeps reading for the closing word and finds the end of the file instead.

Check

bash -n <script>   # repeats the error without running anything
grep -nE '^\s*(if|fi|for|while|do|done|case|esac)\b' <script>   # count openers and closers; one more opener is your block

Fix

Close the block.

A here-document whose end marker is indented or has trailing spaces. Bash never sees the marker, so the rest of the file becomes the here-document, and the block around it is never closed.

Check

grep -n 'EOF' <script> | cat -A   # spaces before the marker or after it (before $): this is your cause

Fix

Put the end marker at the start of its line with nothing after it, or use <<- with tab indentation.

} on the same line as a command without ;. Then } is read as an argument, and the function is never closed.

Check

grep -nE '[^;{ ] *}$' <script>   # a } with no ; before it

Fix

Add the separator: f() { cmd; }.

An unbalanced " or '. Bash treats everything after it as one long string.

Check

bash -n <script>   # repeats the error; the line it names is where the quote opened

Fix

Close the quote.

An apostrophe inside single quotes: 'it's'. The apostrophe ends the string, and the final ' opens a new one that never closes.

Check

grep -n "'" <script>   # an odd number of ' on a line points to it

Fix

Write 'it'\''s', or use double quotes: "it's".

A $( without its ). Bash keeps reading for the end of the command substitution.

Check

grep -n '\$(' <script>   # check each one has its )

Fix

Add the closing ).

! inside double quotes in a typed command. Double quotes do not stop history expansion.

Check

set -o | grep histexpand   # histexpand on: this shell expands !

Fix

Use single quotes ('Pa55!word') or escape the ! with a backslash.

A password or message containing ! pasted into the terminal.

Check

set -o | grep histexpand   # on in your terminal

Fix

Turn history expansion off for the session with set +H, and keep passwords off the command line altogether.

An optional variable with no default. Without a default, reading it when nobody set it is an error under set -u.

Check

[[ -v NAME ]] && echo set || echo unset   # unset: this is your cause

Fix

Give it a default: ${DEBUG:-0}.

A missing argument: $1 when the script was run with none. Positional parameters are variables too.

Check

echo "$#"   # at the top of the script: 0 means no arguments were passed

Fix

Require it with a message: ${1:?usage: sync.sh <dir>}.

A typo in a variable name. The misspelled name was never set.

Check

grep -n 'NAME' <script>   # compare the spelling where it is set and where it is read

Fix

Fix the name.

A variable set in your login shell but not under cron or CI, or set but not exported. A child process only receives exported variables.

Check

printenv NAME > /dev/null && echo exported || echo 'not in the environment'   # not in the environment: the job never received it

Fix

Export it in the caller (export NAME=value), or set it in the job's own configuration.

An empty array expanded as "${arr[@]}" in bash older than 4.4 (macOS's 3.2). Those versions treat an empty array as unset.

Check

bash --version | head -1   # 3.x or 4.0 to 4.3: this can be your cause

Fix

Use ${arr[@]+"${arr[@]}"} on old bash, or run a current bash.

The variable was not passed in, or not exported by the parent script. The guard sees an unset variable.

Check

[[ -v NAME ]] && echo "set, ${#NAME} characters" || echo unset   # unset: it never arrived
printenv NAME > /dev/null && echo exported || echo 'not in the environment'   # in the caller: not exported

Fix

Pass the value: export NAME=value in the caller, or NAME=value ./script.sh.

A missing command-line argument, as in ${1:?usage}. The guard protects a positional parameter.

Check

echo "$#"   # 0: no arguments were passed

Fix

Run the script with the argument its usage message asks for.

It is set but empty. ${NAME:?} rejects empty values too; only ${NAME?}, without the colon, accepts an empty value.

Check

[[ -v NAME ]] && echo "set, ${#NAME} characters"   # set, 0 characters: empty, not missing

Fix

Find where it is set to an empty value (often a failed command substitution) and fix that.

Bash-only forms like ${var//a/b} or ${var^^} run by sh. That happens with no #! line, or when the script is started as sh script.sh.

Check

head -1 <script>   # no #! line, or #!/bin/sh: sh runs it
ls -l /bin/sh   # -> dash on Debian and Ubuntu: sh is not bash there

Fix

Start the script with #!/usr/bin/env bash and run it as ./script.sh, not sh script.sh.

A typo, such as ${x!} or a space inside the braces. Bash itself rejects it too.

Check

bash -n <script>   # bash reports the same line: the expansion is wrong, not the shell

Fix

Correct the expansion.

A library or env file marks settings readonly, with readonly or declare -r. Any later assignment to that name fails.

Check

declare -p <NAME>   # declare -r: it is read-only
grep -rn 'readonly <NAME>\|declare -r <NAME>' <script> <sourced files>   # where it was made read-only

Fix

Use another name, or set the value before the file that makes it read-only is sourced.

Assigning to one of bash's own read-only variables, such as UID or EUID.

Check

declare -p <NAME>   # declare -ir UID=...: a bash variable

Fix

Rename yours. Do not reuse names like UID, PWD or PATH for your own data.

A dash in a name, as in DB-HOST. A dash is not allowed in variable names.

Check

grep -n 'export\|read\|local' <script>   # a name with - or . in it

Fix

Use a valid name: DB_HOST.

A $ before the name, as in read $name. Bash expands it first, so read receives the value instead of the name.

Check

grep -nE '(read|export|unset) .*\$' <script>   # a $ where a name should be

Fix

Write the name without $: read -r name.

Spaces around = with export: export A = 1. Bash passes A, = and 1 as three names, and = is not valid.

Check

grep -nE 'export [A-Za-z_]+ +=' <script>   # spaces around =

Fix

Remove the spaces: export A=1.

IFS=, (or IFS=$'\n') set globally to parse one line. Every later unquoted expansion splits on commas only.

Check

printf '%q\n' "$IFS"   # not $' \t\n': it was changed
grep -n 'IFS=' <script>   # a line with only IFS=... on it

Fix

Set IFS only for the command that needs it: IFS=, read -r a b <<< "$line". Use arrays for lists.

IFS was changed in a function or sourced file and not restored.

Check

grep -rn 'IFS=' <script> <sourced files>   # where it was changed

Fix

Use local IFS inside functions, so the change ends with the function.

${arr[-1]} on an empty array, or ${arr[-n]} with n larger than the array.

Check

echo "${#arr[@]}"   # 0, or less than n: this is your cause
declare -p arr   # what the array really holds

Fix

Check the length before indexing: (( ${#arr[@]} )) && last=${arr[-1]}.

map[$key]=value with key empty or unset. The message then shows map[$key].

Check

printf '%q\n' "$key"   # '' (empty): this is your cause

Fix

Check that the key is not empty before using it: [[ -n $key ]] || continue.

Bash older than 4.2 (macOS's 3.2), which rejects negative indexes; assigning to one, such as arr[-1]=x, needs 4.3.

Check

bash --version | head -1   # 3.x or 4.0 to 4.1: this is your cause

Fix

Use ${arr[${#arr[@]}-1]} on old bash, or run a current bash.

An unquoted empty variable inside [ ]. Without quotes, an empty value turns into no word at all.

Check

bash -x <script>   # the trace shows [ != ok ]: the variable is gone

Fix

Quote the variable, [ "$x" != ok ], or use [[ $x != ok ]].

A command substitution that printed nothing. [ $(get_status) = up ] becomes [ = up ] when the command fails or prints nothing.

Check

bash -x <script>   # shows the empty substitution

Fix

Quote it and handle the failure: s=$(get_status) || exit 1; [ "$s" = up ].

A command inside [ ], as in if [ grep -q x file ]. [ is a command that evaluates tests; it does not run other commands.

Check

bash -x <script>   # shows [ grep -q x file ]

Fix

Run the command directly: if grep -q x file; then.

An unquoted variable with spaces in it. [ $a = b ] with a="x y" becomes [ x y = b ].

Check

bash -x <script>   # the trace shows the value split into several words

Fix

Quote variables: [ "$a" = "$b" ].

An unquoted variable with spaces, as in [ $name = x ]. Each word becomes a separate argument.

Check

bash -x <script>   # the trace shows the value split into several words

Fix

Quote: [ "$name" = x ].

A glob with more than one match, as in [ -f *.log ]. -f takes one file, and the glob gives several.

Check

ls *.log | wc -l   # more than 1: this is your cause

Fix

Test for any match with compgen -G '*.log' > /dev/null, or loop over the files.

cd $dir with spaces in the path. cd takes one directory.

Check

printf '[%s]\n' $dir   # more than one line: the path was split

Fix

Quote: cd -- "$dir" || exit 1.

A percentage or unit from df, free -h or du -h. 100% and 2.1G are text, not integers.

Check

printf '%q\n' "$value"   # a % or a unit letter: this is your cause

Fix

Strip the unit, v=${v%\%}, or ask the tool for raw numbers: df -P, df --output=pcent, du -b.

A decimal such as the load average 1.52. Bash arithmetic and [ only handle integers.

Check

printf '%q\n' "$value"   # a decimal point: this is your cause

Fix

Compare with awk: awk -v l="$load" 'BEGIN { exit !(l > 4) }'.

An empty value. [ "" -gt 90 ] has no integer on the left.

Check

printf '%q\n' "$value"   # '' (empty): the command that filled it printed nothing

Fix

Give it a default, ${v:-0}, and check the command that should have filled it.

A carriage return from a Windows file. 90\r is not an integer.

Check

printf '%q\n' "$value"   # ends in $'\r': this is your cause

Fix

Strip it: v=${v%$'\r'}. Do not switch to >, which compares text (100 sorts before 90).

No space before the closing bracket: [ "$a" = "$b"]. "$b"] is one word, so there is no lone ].

Check

sed -n '8p' <script> | cat -A   # a ] touching the word before it
shellcheck <script>   # SC1020: You need a space before the ]

Fix

Put spaces around every argument of [, including before ]: [ "$a" = "$b" ].

&& or || inside the brackets: [ -n "$a" && -n "$b" ]. The shell ends the [ command at &&, before the ].

Check

grep -nE '\[ [^]]*(&&|\|\|)' <script>   # && or || inside [ ]

Fix

Join tests outside the brackets, [ -n "$a" ] && [ -n "$b" ], or use [[ -n $a && -n $b ]] in bash.

A ] written as ), or left out after a line break.

Check

sed -n '8p' <script>   # no ] at the end of the test

Fix

Close the test with ].

A value with a unit, such as 91%. The % is read as the modulo operator with nothing after it.

Check

printf '%q\n' "$value"   # a % or unit: this is your cause

Fix

Clean the value first: v=${v%\%}.

An empty operand, as in $(( 3 + )). A variable that is empty leaves a gap in the expression.

Check

printf '%q\n' "$n"   # '' (empty): this is your cause

Fix

Give empty values a default: $(( ${n:-0} + 1 )).

A variable holding text. Text that is not a number or a variable name cannot be used in arithmetic.

Check

printf '%q\n' "$value"   # words or symbols: this is your cause

Fix

Validate numbers before using them: [[ $n =~ ^[0-9]+$ ]] || exit 2.

$(date +%m), %d or %H used in arithmetic. Those print zero-padded numbers like 08.

Check

echo $(( 07 )) $(( 010 ))   # prints 7 8: a leading zero means octal
grep -n 'date +%[mdHM]' <script>   # where padded values come from

Fix

Force base 10, $(( 10#$month + 1 )), or ask GNU date for no padding: date +%-m.

No matching lines, so the count is 0. The input was empty or the filter matched nothing.

Check

echo "count=$count"   # count=0: the input had nothing to count

Fix

Check the divisor first: (( count > 0 )) && echo $(( total / count )).

An unset or misspelled variable, which arithmetic treats as 0.

Check

[[ -v count ]] && echo set || echo unset   # unset: the name is wrong or it was never filled

Fix

Fix the name, or set it before dividing.

The variable is empty or unset. With no word after >>, there is no file to open.

Check

[[ -v LOG ]] && echo "set, ${#LOG} characters" || echo unset   # unset or 0 characters: this is your cause

Fix

Make sure it is set, and quote it: >> "${LOG:?LOG is not set}".

An unquoted path with spaces. It expands to several words, and a redirection takes one.

Check

printf '[%s]\n' $LOG   # more than one line: the path was split

Fix

Quote it: >> "$LOG".

The marker is indented. Only <<- allows leading tabs, and never spaces.

Check

grep -n 'EOF' <script> | cat -A   # spaces or ^I before the marker: this is your cause

Fix

Put the marker at the start of its line, or use <<- with tab indentation.

Trailing spaces or a \r after the marker. The line must be exactly the marker.

Check

grep -n 'EOF' <script> | cat -A   # a space or ^M before $: this is your cause

Fix

Remove them; strip \r with sed -i 's/\r$//' script.sh.

A different spelling: EOF versus END. The end marker must match the one after << exactly.

Check

grep -nE '<<-?.?[A-Z]+' <script>   # the marker it wants

Fix

Use the same marker in both places.

return "$value" used to hand back data. Exit statuses are numbers only.

Check

grep -n 'return' <script>   # a return followed by a word or a variable holding text

Fix

Print the result and capture it: get() { echo "$value"; } and v=$(get).

return used outside a function or a sourced file. It is refused there with its own message (can only return from a function or sourced script).

Check

grep -n 'return' <script>   # a return at the top level

Fix

Use exit at the top level of a script.

Code moved out of a function. The local keywords came along.

Check

sed -n '1,10p' <script>   # the local line is not inside a name() { ... } block

Fix

Drop local at the top level, or wrap the code in a function and call it.

A line copied from a function and pasted into the terminal. The terminal is not inside a function either.

Check

type <function>   # check you meant to run the function, not its body

Fix

Run the function, or drop local when trying lines by hand.

A letter missing from the option string. getopts only accepts the letters listed.

Check

grep -n getopts <script>   # the letter is not in the string

Fix

Add the letter, and a case for it.

A missing : after an option that takes a value ("e" instead of "e:"). Without it, getopts does not read a value, and the value is left as a stray argument.

Check

grep -n getopts <script>   # the letter has no : after it

Fix

Add : after it, and shift $((OPTIND - 1)) after the loop.

The script was simply called with a wrong option or without a value.

Check

./<script> -h   # compare with its usage

Fix

Handle ? and : in the case statement with a usage message and exit status 2.

A wrapper with the same name as the command it wraps: ls() { ls --color "$@"; }. The ls inside calls the function again, forever.

Check

type <name>   # 'is a function': the name is shadowed

Fix

Call the real command with command ls inside the wrapper, or rename the function.

Recursion without a stop condition.

Check

bash -x <script> 2>&1 | head -40   # the same call repeating

Fix

Add the condition that ends the recursion, or use a loop.

The replacement comes from a variable that contains slashes, as in sed "s/X/$PATH_VALUE/". The first / in the value ends the replacement, and the rest is read as flags.

Check

echo "s/X/$VALUE/"   # more than three / in the output: the value broke the command

Fix

Pick a delimiter that is not in the data: sed "s|X|$VALUE|".

A missing or extra delimiter typed by hand. Too few delimiters gives unterminated s command instead; extra text after the last one gives this message.

Check

echo '<the sed expression>'   # count the delimiters: there must be exactly three

Fix

Fix the expression.

find . -name *.log run where two or more .log files exist. The shell, not find, expands the unquoted pattern.

Check

echo find . -name *.log   # shows what find really received: several names after -name

Fix

Quote the pattern: find . -name '*.log'.

With exactly one match there is no error, which is the hidden form of the same bug. find quietly searches for that one name and misses every other file.

Check

echo find . -name *.log   # one file name after -name: the search is wrong even without an error

Fix

Quote the pattern: find . -name '*.log'.

The path is in an unquoted variable and has spaces, as in cp $SRC $DEST. /srv/reports/Q3 summary becomes two arguments.

Check

printf '[%s]\n' $SRC   # more than one line: the path was split

Fix

Quote: cp -r -- "$SRC" "$DEST/".

A file list built with $(ls) or $(find). The output is split on spaces and newlines, so names with spaces break apart.

Check

grep -nE '\$\((ls|find)' <script>   # a loop or command over $(ls) or $(find)

Fix

Loop over a glob (for f in /srv/reports/*; do) or use find ... -print0 | xargs -0.

The path really does not exist: a wrong name, or a mount that is missing.

Check

ls -ld -- "$SRC"   # No such file or directory even when quoted: the path is wrong

Fix

Fix the path, or check the mount before copying: mountpoint -q /srv || exit 1.

A label in front of the date (notAfter=). openssl and other tools print name=value.

Check

printf '%q\n' "$end"   # starts with a label: this is your cause

Fix

Strip labels first: ${end#notAfter=}.

A format GNU date does not read, or BSD date on macOS, which uses date -j -f instead of -d.

Check

date --version 2>/dev/null | head -1   # no output: BSD date
date -d "$end" +%s   # still invalid after cleaning: the format

Fix

Convert the format first, or on macOS use date -j -f '%b %d %T %Y %Z' "$end" +%s.

A carriage return at the end, from a Windows file or a web response.

Check

printf '%q\n' "$end"   # ends in $'\r': this is your cause

Fix

Strip it: end=${end%$'\r'}. Check the result: ts=$(date -d "$end" +%s) || exit 1.

curl without -f received an error page (401, 404, 502) and passed it on.

Check

curl -sS -o body.json -w '%{http_code} %{content_type}\n' <url>   # not 200, or text/html: this is your cause
head -c 200 body.json   # the start of the page instead of JSON

Fix

Use curl -fsS so HTTP errors stop the script before jq, and check the status code.

A proxy or login page answered instead of the API.

Check

curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' <url>   # a redirect to a login page

Fix

Call the API with its own credentials, past the proxy's login.

A script printed log lines to stdout ahead of the JSON.

Check

<command> | head -3   # log lines before the {

Fix

Send log messages to stderr, and check input with jq -e . >/dev/null before reading values from it.

The API returns a list ([{...}]) and the filter treats it as one object.

Check

jq 'type' <file>   # "array": use .[] or .[0] first

Fix

Match the filter to the data: .[].name or .[0].name.

Some records have no such key, so .items[] meets null.

Check

jq 'keys' <file>   # the key is missing from this record

Fix

Use .items[]? or (.items // [])[] where a key may be missing.

An error response with a different shape from the normal one.

Check

jq '.[0]' <file>   # or jq . <file>: an error object instead of data

Fix

Check the HTTP status first (curl -fsS), and use jq -e so a missing value fails the script instead of printing null.

cmd & ... wait with no PIDs. The bare wait returns 0 whatever the jobs did.

Check

grep -nE '^\s*wait\s*$' <script>   # a bare wait: this is your cause

Fix

Save each PID (pids+=($!)) and wait for each one: for p in "${pids[@]}"; do wait "$p" || failed=1; done, then exit non-zero if any failed.

$! not saved after each job. Without the PID there is nothing to wait for by name.

Check

cmd & p=$!; wait "$p"; echo $?   # the status that was being lost

Fix

Save $! right after each &.

Expecting set -e to cover jobs. It only sees commands in the foreground.

Check

jobs -l   # jobs still running when the script reports success

Fix

Wait for each job and check its status, or use xargs -P, whose exit status (123) reports a failure.

The job was started inside ( ), a pipeline or $( ). A subshell started it and owns it.

Check

jobs -l   # the PID is not in this shell's list

Fix

Start the job in the same shell that waits for it, and keep $! right after starting it.

The PID came from a file or from pgrep, or the job was started by another script, even one this script called.

Check

ps -o ppid= -p <pid>   # its parent is not this shell's PID ($$)

Fix

To wait for an unrelated process, poll: while kill -0 "$pid" 2>/dev/null; do sleep 1; done.

systemd stopping or restarting the service, kill, or a deploy. They send TERM.

Check

journalctl -u <service> | tail   # Stopping or Restarting at that time: this is your cause

Fix

Clean up in trap '...' EXIT (bash runs it on TERM, HUP and INT too), and make reruns safe.

The SSH session ending for a job started with &. The terminal going away sends HUP.

Check

echo $?   # 129: hangup

Fix

Run long work with nohup, setsid, tmux or a systemd unit.

A timeout limit. timeout sends TERM when the time is up and returns 124.

Check

grep -n 'timeout' <script>   # the limit that applies

Fix

Raise the limit to what the work needs, or make the work faster.

The out-of-memory killer. It sends KILL, which cannot be trapped; the shell prints Killed and the status is 137.

Check

dmesg | grep -i 'killed process'   # a line naming the process: memory

Fix

Give it the memory it needs, or reduce its memory use.

Running nohup cmd without redirecting its output. nohup only picks nohup.out when output would go to the terminal.

Check

tail nohup.out   # the command's output is here

Fix

Choose where output goes: nohup ./job.sh > job.log 2>&1 &.

A server that accepts connections and never answers. The connection opens and then nothing comes back.

Check

curl -v --connect-timeout 5 -m 10 -o /dev/null <url>   # 'Connected' then silence: the server is slow or stuck

Fix

Fix the server, and keep a limit on the call: curl -m 10.

A firewall or security group that drops the packets. The connection never opens, and curl waits until it times out.

Check

curl -v --connect-timeout 5 -m 10 -o /dev/null <url>   # stuck at 'Trying': nothing answers, usually a firewall

Fix

Allow the port from this host, or go through the right proxy.

A limit shorter than the work needs, or no limit at all, so the job hangs.

Check

time <command>   # how long it really takes
timeout 10 <command>; echo $?   # 124: it did not finish in 10 seconds

Fix

Give every network call a limit (timeout 10 cmd or curl -m 10) that fits the work, and treat 124 as a failure: || { echo 'timed out' >&2; exit 1; }.

A loop that starts & jobs without a limit, or a function that starts itself in the background.

Check

ps -L -u "$USER" --no-headers | wc -l   # threads, which is what the limit counts; near ulimit -u: the user limit
ulimit -u   # the limit

Fix

Stop the runaway jobs (pkill -u <user> -f <script>), then cap concurrency: xargs -P 8, or count jobs and wait -n before starting another.

A low per-user process limit on a shared host. It counts threads, so one Java or Node service can use most of it.

Check

ps -L -u "$USER" -o comm= | sort | uniq -c | sort -n | tail -3   # which programs hold the threads

Fix

Raise the limit for that user in /etc/security/limits.d/, or run the service as its own user.

A systemd service with a small TasksMax.

Check

systemctl show -p TasksMax,TasksCurrent <unit>   # TasksCurrent near TasksMax: the unit's limit

Fix

Raise TasksMax= in a drop-in for that unit.

A container's pids limit (--pids-limit, or the cgroup pids.max).

Check

cat /sys/fs/cgroup/pids.max /sys/fs/cgroup/pids.current   # inside a container (cgroup v2): current near max

Fix

Raise the container's pids limit, or reduce what runs in it.

126 or 127: the command could not be run at all. See the Permission denied and command not found entries.

Check

echo $?   # right after the command: 126 or 127

Fix

Fix the permission (126) or the name and PATH (127).

137: the process was killed with SIGKILL, in containers usually because it hit its memory limit.

Check

dmesg | grep -i 'killed process'   # a line naming it: the OOM killer
kubectl describe pod <pod> | grep -i oomkilled   # in Kubernetes: Reason OOMKilled

Fix

Give it the memory it needs, or find what makes it grow.

130 or 143: it was stopped on purpose, by Ctrl+C or by kill, systemctl stop or timeout.

Check

journalctl -u <service> | tail   # a stop or restart at that time

Fix

See the Terminated entry; add cleanup with a trap.

grep found nothing. grep returns 1 for no match, and set -e treats that as a failure.

Check

bash -x <script> 2>&1 | tail -5   # the last traced line is the command that ended it

Fix

Add || true where no match is fine, or branch on it: if grep -q x file; then.

(( count++ )) or let when the value was 0. An arithmetic command whose result is 0 returns status 1.

Check

bash -x <script> 2>&1 | tail -5   # (( count++ )) as the last line

Fix

Use count=$((count + 1)), which always succeeds.

[ cond ] && action as the last line of a function, when cond is false. The function then returns 1, and the caller stops.

Check

trap 'echo "failed at line $LINENO: $BASH_COMMAND" >&2' ERR   # add near the top: prints the command that failed

Fix

Write if cond; then action; fi instead of cond && action.

diff found differences, or cmp found the files differ. Both return 1 for "different", which is not an error.

Check

bash -x <script> 2>&1 | tail -5   # diff or cmp as the last line

Fix

Capture the status: rc=0; diff a b || rc=$?, and treat 2 or more as the real error.

cmd | gzip > file without pipefail. A pipeline's status is the last command's, so gzip's success hides cmd's failure.

Check

echo "${PIPESTATUS[@]}"   # right after the pipeline: a non-zero before the last number is your cause

Fix

Use set -o pipefail.

local x=$(cmd). The status checked is that of local, which succeeded, not that of cmd.

Check

grep -nE 'local [a-z_]+=\$\(' <script>   # each line here hides a status

Fix

Declare, then assign: local x; x=$(cmd).

The step is in a function called as if deploy; then or deploy || alert. bash turns set -e off for every line inside deploy while it runs in that position.

Check

bash -x <script>   # the trace continues past the failing line inside the function

Fix

Check the steps that matter explicitly inside the function (cmd || return 1), and do not rely on set -e there.

$? read after another command. $? holds the status of the most recent command only, so an echo or [ in between replaces it.

Check

grep -n '\$?' <script>   # a line before it that is not the command you mean

Fix

Capture it at once: cmd; rc=$?, or test the command directly with if.

A failed cd in a script without set -e, or in one of the places above. A plain failed cd does stop a set -e script.

Check

head -5 <script>   # no set -e: every unchecked command can fail and carry on

Fix

cd dir || exit 1, and start scripts with set -euo pipefail.

cmd | while read ...; do total=...; done. Each part of a pipeline runs in its own subshell.

Check

grep -n '| *while' <script>   # a line here that sets variables or calls exit is your cause

Fix

Redirect into the loop instead: while read -r line; do ...; done < file or done < <(cmd).

The file does not end with a newline. Files from some editors, exports or printf without a final \n do this.

Check

tail -c 1 <file> | od -c   # no \n shown: the last line is unterminated

Fix

while read -r line || [[ -n $line ]]; do.

head, grep -q or grep -m1 stopped reading early, as designed, and the pipeline runs under pipefail in strict mode.

Check

echo "${PIPESTATUS[@]}"   # right after it: 141 for the writer and 0 for head
bash -o pipefail -c 'yes | head -1 >/dev/null; echo $?'   # prints 141: the same effect

Fix

Where stopping early is the point, avoid the pipe (grep -m1 pattern file, awk 'NR == 1' file), or accept 141 for that one pipeline: first=$(cmd | head -1) || [ $? -eq 141 ].

The reading command crashed or failed.

Check

echo "${PIPESTATUS[@]}"   # a non-zero status for the last command too

Fix

Fix the reader; that is the real error.

A program outside /usr/bin and /bin. With cron's short PATH it is not found (look for command not found in cron's mail or log).

Check

env -i HOME=$HOME PATH=/usr/bin:/bin /bin/sh -c '/path/to/script.sh'   # fails the same way: the environment is the cause

Fix

Set PATH at the top of the script, or call programs by their full path.

Variables exported in .bashrc or your profile. Cron does not read them.

Check

grep -n 'export' ~/.bashrc ~/.profile   # anything the script relies on

Fix

Have the script load its own config by absolute path: . /etc/app/env.

Relative paths. Cron starts the job in your home directory, not the script's.

Check

grep -nE '(^| )\./' <script>   # paths starting with ./

Fix

Use absolute paths, or cd "$(dirname "${BASH_SOURCE[0]}")" || exit 1 first.

No #! line, so sh runs it, or % in the crontab line, which cron turns into a newline.

Check

head -1 <script>   # no #!: sh runs it
crontab -l | grep -n '%'   # an unescaped % cuts the command

Fix

Add #!/usr/bin/env bash, and escape % as \% in crontab.

The job runs as another user (root's crontab or /etc/cron.d), or never started. Another user has a different HOME, keys and permissions.

Check

journalctl -u cron -u crond --since today   # or grep CRON /var/log/syslog: did it start, and as which user
crontab -l; sudo ls /etc/cron.d   # where the job is really defined

Fix

Put the job in the crontab of the user it should run as.

The server is down or busy (503, 502, 429). It answered, but with an error.

Check

curl -sS -o /dev/null -w '%{http_code} %{remote_ip}\n' <url>   # 5xx or 429: the server side; note the address curl reached

Fix

Retry with a limit and backoff (scripts/retry.sh), and alert if it keeps failing.

A 401 or 403. The credentials are missing or wrong, or belong to another environment.

Check

curl -sS -o /dev/null -w '%{http_code}\n' <url>   # 401 or 403: authentication, not the network

Fix

Fix the credentials the job uses; do not retry.

A wrong host name, or no DNS from that machine (6), or the wrong address.

Check

getent hosts <host>   # nothing: DNS fails here; an unexpected IP: wrong record

Fix

Fix the name or the resolver on that host.

A firewall, or the service not listening (7).

Check

curl -v --connect-timeout 5 <url> 2>&1 | grep -E 'Trying|Connected|refused'   # refused: nothing listens; no Connected: blocked

Fix

Start the service, or open the port from this host.

A proxy setting (https_proxy, no_proxy) present in your shell but not in the job, or the reverse.

Check

[ -n "${https_proxy}${HTTPS_PROXY}" ] && echo 'a proxy is set'   # compare your shell with the job, without printing it

Fix

Set the same proxy variables in the job's environment.

(60) a missing CA bundle, or a proxy that re-signs TLS. curl cannot trust the server's certificate.

Check

curl -v <url> 2>&1 | grep -E 'issuer|CAfile'   # an issuer you do not expect: a re-signing proxy

Fix

Give curl the right CA bundle with --cacert. Do not use -k, which accepts any server.

The job runs without a terminal (cron, CI, ssh host cmd), and the sudoers rule for this user asks for a password.

Check

sudo -n true; echo $?   # 1 and 'a password is required': no passwordless rule

Fix

Run the job as a user that already has the rights it needs, or allow exactly that command without a password in a drop-in edited with visudo -f /etc/sudoers.d/deploy: deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart app.

The rule allows a different command or path than the one the script runs. sudo then falls back to asking for a password.

Check

sudo -n -l   # the commands allowed without a password; compare the exact path and arguments

Fix

Make the script run exactly the allowed command (full path), or correct the rule. Use sudo -n in scripts so they fail at once instead of waiting. Never feed a password to sudo -S from a script or a file.

Cron or another user runs the job with a different HOME, so ~/.aws is not the one you configured.

Check

aws sts get-caller-identity   # run as the job's user: the same error means it has no identity
id -un; echo "$HOME"   # in the job: another user or HOME

Fix

Give that user its own profile, or better a role (below).

AWS_PROFILE is set in your shell but not exported to the job.

Check

env | grep -E '^AWS_' | cut -d= -f1   # names only, never values; compare your shell with the job
aws configure list   # the Type column shows where each value came from (env, shared-credentials-file, iam-role); keys are masked

Fix

Export AWS_PROFILE in the job's environment, or pass --profile.

On EC2 the instance has no role, or the metadata service needs IMDSv2 and the hop limit is too low for a container.

Check

aws ec2 describe-instances --instance-ids <id> --query 'Reservations[].Instances[].[IamInstanceProfile.Arn,MetadataOptions.HttpPutResponseHopLimit]'   # None: no role; 1 with containers: raise to 2

Fix

Attach an instance role with only the permissions the job needs, and set the hop limit to 2 for containers.

SSO credentials expired and need aws sso login.

Check

aws configure list   # Type sso, and get-caller-identity fails

Fix

Run aws sso login for people; give unattended jobs a role instead.

The command runs under sudo, which uses root's HOME and drops your AWS_* variables.

Check

sudo aws configure list   # nothing found as root

Fix

Do not run the AWS CLI under sudo; run the job as the user that has the profile or role.

Credentials copied into environment variables from an earlier aws sts assume-role or aws sso login. Exported keys win over the profile, so the CLI never refreshes them.

Check

aws configure list   # Type 'env' for access_key: exported keys are being used, not the profile

Fix

Let the CLI refresh credentials itself: use a profile with role_arn and source_profile (or credential_process), or the instance role, and unset the exported keys.

A job that runs longer than the role's maximum session duration. Even a refreshable session can hit that limit when the keys were fixed at the start.

Check

aws iam get-role --role-name <role> --query Role.MaxSessionDuration   # in seconds; shorter than the job

Fix

Use a profile or role the CLI can renew, or split the job.

The host clock is wrong. The CLI does not refresh in time, and requests may fail with RequestExpired or Signature expired instead.

Check

date -u; timedatectl show -p NTPSynchronized   # minutes off, or NTPSynchronized=no: the clock

Fix

Fix the clock with NTP.

The backup volume was not mounted, so the folder existed but was empty.

Check

mountpoint <source>   # 'is not a mountpoint': the volume was missing
ls -A <source> | head   # empty: this is your cause

Fix

Check the mount before syncing (mountpoint -q) and refuse an empty source. scripts/s3_backup_sync.sh does both.

A variable for the source path was empty or wrong, so sync read the wrong directory.

Check

aws s3 sync <source> s3://<bucket>/<prefix> --delete --dryrun | grep -c delete:   # a large count before a real run: stop

Fix

Require the variable (${SRC:?}) and run --dryrun first.

The order of source and destination was swapped, so the bucket was made to match an old local copy.

Check

grep -n 's3 sync' <script>   # the first path is the source

Fix

Put the source first.

Recovery, for any of the above: on a versioned bucket a deleted object is only hidden behind a delete marker.

Check

aws s3api list-object-versions --bucket <bucket> --prefix <prefix> --max-items 5   # DeleteMarkers listed: the old versions can be restored

Fix

Restore the objects by deleting the delete markers (or copying the previous versions back), then fix the job.

The database (often a managed service such as RDS) was upgraded and the backup host was not.

Check

psql -Atc 'show server_version'   # the server's major version
pg_dump --version   # an older major version: this is your cause

Fix

Install the client tools of the server's major version (or newer) on the backup host.

Several PostgreSQL versions are installed and PATH picks the old pg_dump.

Check

type -a pg_dump   # the one the job runs comes first
ls /usr/lib/postgresql/   # Debian and Ubuntu; /usr/pgsql-<n>/bin on RHEL: the newer one is already there

Fix

Call the right one by its full path, for example /usr/lib/postgresql/16/bin/pg_dump on Debian.

~/.pgpass belongs to another user, or cron runs as a different user or HOME. libpq reads the file of the user running the job.

Check

id -un; echo "$HOME"; ls -l "${PGPASSFILE:-$HOME/.pgpass}"   # in the job: No such file, or another owner, is your cause

Fix

Create ~/.pgpass for the user that runs the job, with host:port:database:user:password.

Its permissions are wider than 0600, so libpq ignores it. It prints a warning saying so just before this error.

Check

stat -c '%a %U' "${PGPASSFILE:-$HOME/.pgpass}"   # anything but 600 and the job's user

Fix

chmod 600 ~/.pgpass.

The host, port, database or user in the file does not match the connection. The host is compared as text, so db.internal and its IP are different entries.

Check

cut -d: -f1-4 "${PGPASSFILE:-$HOME/.pgpass}"   # the host:port:db:user keys, without the passwords; compare with -h, -p, -d and -U

Fix

Add a line that matches exactly how the job connects, or use * for a field.

By hand you connect over the Unix socket, where pg_hba.conf allows peer; the job uses -h, which hits a host rule that needs a password.

Check

PGCONNECT_TIMEOUT=5 psql -w -h db.internal -U backup -d shop -c 'select 1'   # the same connection as the job, never prompting: this error here, but psql without -h works

Fix

Give the job a .pgpass entry for the TCP connection, and pass --no-password (or -w) so a missing password fails at once instead of waiting.

The backup user has SELECT, SHOW VIEW, TRIGGER, EVENT and LOCK TABLES, but not PROCESS. That is the usual least-privilege set.

Check

mysql --defaults-extra-file=/etc/mysql/backup.cnf -e 'SELECT CURRENT_USER(); SHOW GRANTS'   # with the job's own credentials: the account really used; no PROCESS in its grants

Fix

Add --no-tablespaces: most dumps do not need tablespace definitions. Grant PROCESS only if you do need them.

The job logs in as a different account from the one you granted. MySQL picks the account by user name and the host it sees you connect from.

Check

mysql --defaults-extra-file=/etc/mysql/backup.cnf -e 'SELECT USER(), CURRENT_USER()'   # CURRENT_USER() is not the account you granted: another one matched

Fix

Grant to the account in CURRENT_USER(), or fix the job's user and host.

The client was upgraded and the dump options were not.

Check

mysqldump --version   # 8.0.21 or later

Fix

Pin the dump options in one script (scripts/mysql_backup.sh) and test it before upgrading.

mysqldump -u backup -pS3cret ... in a script or crontab.

Check

grep -nE -- '-p[^ ]|--password=' <script> | cut -d: -f1   # line numbers only, not the password
crontab -l | grep -cE -- '-p[^ ]|--password='   # a count above 0: the crontab has one

Fix

Put the credentials in an option file ([client] with user= and password=), chmod 600 it, and pass --defaults-extra-file=/etc/mysql/backup.cnf as the first option. Then change the password, because it has been exposed.

--password=... added by a wrapper that builds the command line, such as a backup tool or a Makefile.

Check

ps -o args= -C mysqldump | grep -c -- '--password='   # while it runs: a count above 0, without printing it

Fix

Configure the wrapper to use an option file. Do not move the password to MYSQL_PWD: that hides the warning but is deprecated and no safer.

The new image crashes on start (CrashLoopBackOff) or fails its readiness probe. The pods exist but never count as ready.

Check

kubectl get pods -l <selector>   # CrashLoopBackOff, or Running with 0/1 READY: this is your cause
kubectl logs <new-pod> --previous   # why the last start crashed

Fix

Roll back with kubectl rollout undo deployment/<name> and confirm with rollout status. Then fix the image or the probe and release again.

The image cannot be pulled (ImagePullBackOff): a wrong tag, or no pull secret for a private registry.

Check

kubectl describe pod <new-pod> | tail -20   # Failed to pull image ... not found, or unauthorized

Fix

Fix the tag, or add the pull secret to the service account. Roll back first if users are affected.

Not enough CPU or memory to schedule the new pods. They stay Pending.

Check

kubectl describe pod <new-pod> | tail -20   # FailedScheduling: Insufficient cpu or memory

Fix

Lower the requests to what the app needs, or add capacity.

A ResourceQuota or LimitRange rejects the new pods, so the ReplicaSet never creates them.

Check

kubectl get events -n <ns> --sort-by=.lastTimestamp | tail   # 'exceeded quota' or FailedCreate: no pods were made at all

Fix

Raise the quota, or lower the requests, so the rollout's extra pods fit.

A PDB that can never allow a disruption: minAvailable equal to the replica count, or maxUnavailable: 0.

Check

kubectl get pdb -A   # ALLOWED DISRUPTIONS 0 with all replicas healthy: the budget itself is the cause

Fix

Scale the Deployment up first so the budget allows one eviction, then fix the budget (maxUnavailable: 1).

Other replicas are not ready, so the budget is already used up.

Check

kubectl get pods -n <ns> -l <selector> -o wide   # replicas not READY elsewhere

Fix

Fix or wait for the unready replicas; the drain then continues.

All replicas run on the node being drained, so the next one cannot leave until the first is ready elsewhere.

Check

kubectl get pods -n <ns> -l <selector> -o wide   # every replica on this node

Fix

Wait for the evicted replicas to become ready on other nodes, and spread replicas with topology spread constraints. Do not bypass the budget with --disable-eviction.

Caches, scratch space or buffers kept in emptyDir. That data is meant to be disposable.

Check

kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.volumes}'   # emptyDir used for cache or tmp

Fix

Drain with --delete-emptydir-data (-e in scripts/k8s_node_drain.sh).

Data that should be on a PersistentVolume is in emptyDir.

Check

kubectl exec -n <ns> <pod> -- du -sh <mount path>   # data you need to keep

Fix

Move it to a PersistentVolume first, then drain.

A debug pod started with kubectl run, left behind.

Check

kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.ownerReferences}'   # empty: no controller

Fix

Delete it yourself if it is a leftover, then drain.

A workload created from a manifest as a bare Pod.

Check

kubectl get pod <pod> -n <ns> -o yaml | head -20   # kind: Pod with no ownerReferences, and labels of a real app

Fix

Recreate it under a controller (a Deployment) first. --force deletes it for good.

A new host not yet in known_hosts.

Check

ssh-keygen -F <host>   # nothing: the host is unknown

Fix

Check the key's fingerprint against the host (console, cloud API or your inventory), then add it: ssh-keyscan <host> >> ~/.ssh/known_hosts.

A rebuilt host with a new key. ssh prints REMOTE HOST IDENTIFICATION HAS CHANGED.

Check

ssh-keyscan <host> 2>/dev/null | ssh-keygen -lf -   # the fingerprint the host offers now; compare with the console or inventory

Fix

If it matches the rebuilt host, remove the old line with ssh-keygen -R <host> and add the new key.

A different user (cron, CI) with its own empty known_hosts.

Check

sudo -u <job user> ssh-keygen -F <host>   # nothing for that user

Fix

Give that user the checked keys too, or a shared /etc/ssh/ssh_known_hosts.

An IP address or name now used by a different server (common with cloud IPs), or a real man-in-the-middle.

Check

ssh-keyscan <host> 2>/dev/null | ssh-keygen -lf -   # a fingerprint that matches no server you own: stop and investigate

Fix

Do not connect until you know which server it is. Never set StrictHostKeyChecking=no to make a loop pass: that accepts any server, including an attacker's.

The key is not in that host's authorized_keys for this user, or the user name is wrong for that host.

Check

ssh -v -o BatchMode=yes <user>@<host> true 2>&1 | grep -E 'Offering|Authentications'   # the key was offered and refused: not authorised for this user

Fix

Add the job's public key to the host for the right user.

Cron or CI runs without your ssh agent or key file, so it offers nothing.

Check

ssh-add -l   # in the job: 'Could not open a connection' or no identities
ssh -v -o BatchMode=yes <user>@<host> true 2>&1 | grep -c Offering   # 0: no key offered

Fix

Point ssh at the key with -i in the job.

authorized_keys, ~/.ssh or the home directory has permissions or an owner sshd rejects. sshd then ignores the file.

Check

sudo journalctl -u ssh -u sshd | tail   # on the server: 'bad ownership or modes' names the file

Fix

chmod go-w ~; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys, owned by that user.

An RSA key used with an old server or client that can only sign with ssh-rsa (SHA-1), which OpenSSH 8.8 and later refuse by default.

Check

ssh -v <user>@<host> true 2>&1 | grep -iE 'no mutual signature|ssh-rsa'   # a line here: the signature algorithm

Fix

Replace old RSA keys with ed25519, rather than re-enabling ssh-rsa.

sshd is stopped or listens on another port (refused).

Check

nc -vz -w 5 <host> 22   # refused: nothing listens on 22
ssh -v -o ConnectTimeout=5 <user>@<host> true   # 'Connection refused' after 'Connecting to'

Fix

Start sshd on the host (through the console), or connect to the right port with -p.

A firewall or security group drops port 22 from where you are (timed out), or a host firewall rejects it (refused or no route to host).

Check

nc -vz -w 5 <host> 22   # timed out: dropped; compare with a host that works from the same place

Fix

Allow port 22 from your address only, or go through the bastion.

The host is down, or its name points at an old address.

Check

getent hosts <host>   # the address ssh will use; compare with the inventory

Fix

Fix the DNS record or the inventory, or start the host.

Connecting to a private address from outside the network.

Check

getent hosts <host>   # 10.x, 172.16-31.x or 192.168.x from outside

Fix

Connect through the VPN or the bastion (ssh -J bastion <host>).

The key was copied or unpacked with default permissions (0644), or a CI job wrote it from a secret without setting its mode.

Check

stat -c '%a %U' <key>   # anything but 600 or 400, or another owner: this is your cause

Fix

chmod 600 <key> and chmod 700 ~/.ssh, owned by the user who runs ssh. In CI, write it with install -m 600.

The key is on a mounted drive that does not keep Unix permissions, such as a Windows drive under WSL. chmod has no effect there.

Check

chmod 600 <key>; stat -c '%a' <key>   # still 777 or 644: the drive cannot keep the mode

Fix

Copy the key into ~/.ssh on the Linux file system and use it from there.

A typo, a missing semicolon, or a directive from a module that is not loaded.

Check

sudo nginx -t   # names the file and line of the parse error

Fix

Fix that line, run nginx -t again, then reload (not restart) and check the site answers.

An include that points to a missing file or certificate.

Check

sudo nginx -t   # 'No such file' or 'cannot load certificate' with the path

Fix

Restore the file or fix the path.

The port is taken: bind() to 0.0.0.0:80 failed (98: Address already in use) at start, which nginx -t does not catch.

Check

sudo ss -ltnp 'sport = :80'   # another process holds the port

Fix

Stop the other process, or move one of them to another port.

You edited a file that is not loaded, or the service starts nginx with another -c file.

Check

sudo nginx -T | grep '^# configuration file'   # the files really loaded
systemctl cat nginx | grep ExecStart   # a -c option points elsewhere

Fix

Edit the file that is loaded, or include yours from it.

The status is not captured, so set -e (or the CI step) stops the script on 2.

Check

terraform plan -detailed-exitcode -input=false > /dev/null; echo $?   # 2: changes, not an error

Fix

Capture the status and branch on it: rc=0; terraform plan -detailed-exitcode ... || rc=$?; case $rc in 0) ...;; 2) ...;; *) ...;; esac.

if terraform plan ...; then sends drift to the error branch, because 2 is not 0.

Check

grep -n 'detailed-exitcode' <script>   # used inside if or with &&

Fix

Use the || rc=$? and case form. scripts/terraform_drift_check.sh reports 0, 1 drift and 2 error.

Someone is applying right now, or a drift check or CI plan holds the lock while it runs.

Check

terraform plan -lock-timeout=60s   # succeeds after a wait: it was a run in progress

Fix

Wait, or use -lock-timeout.

A run was killed and left a stale lock.

Check

aws dynamodb get-item --table-name <lock-table> --key '{"LockID":{"S":"<bucket>/<key>"}}'   # Info shows who holds it and since when (the <key>-md5 item is the state checksum, not the lock)
aws s3api head-object --bucket <bucket> --key <key>.tflock   # with use_lockfile = true (Terraform 1.10 and later)

Fix

Remove a lock with terraform force-unlock <ID> only after you know its run is dead.

The lock call itself failed: no permission on the lock table (or the S3 lock file), or a wrong table name or region. The detail then shows an AWS error such as AccessDeniedException instead of Lock Info.

Check

aws sts get-caller-identity   # the identity Terraform runs as
aws dynamodb describe-table --table-name <lock-table> --region <region>   # ResourceNotFoundException: wrong name or region

Fix

Give that identity dynamodb:GetItem, PutItem and DeleteItem on the lock table only, or fix the backend's table name and region.

Logging request bodies or query strings.

Check

./scripts/pii_scan.sh -c /var/log/shop   # counts per kind of data and file
grep -c 'email=' /var/log/shop/app.log   # a count above 0: query strings with personal data are logged

Fix

Log ids and events instead of values.

print(user) or a debug log level left on in production.

Check

grep -rn 'log.*level\|DEBUG' /etc/shop/   # debug level in the production config

Fix

Set the production log level to info or above.

Exceptions that include the input value, or access logs with personal data in URLs (/users/priya@example.com).

Check

./scripts/pii_scan.sh -c /var/log/nginx   # hits in access logs: personal data in URLs

Fix

Keep personal data out of URLs, and mask what you cannot change on the way out (app | scripts/pii_redact.sh).

The existing copies, for any of the above.

Check

./scripts/pii_scan.sh -c <log platform export or backup>   # the copies hold it too

Fix

Redact or expire the existing files and copies, and record what was exposed in case it counts as a breach.

A production dump was restored into staging to reproduce a bug.

Check

psql -h staging -Atc 'select count(*) from customers where email not like ''%@example.%'''   # above 0: real addresses in staging

Fix

Delete the copy, and reproduce bugs with synthetic data or a masked extract made inside production's controls.

A few real rows were exported as fixtures.

Check

./scripts/pii_scan.sh -c fixtures/ testdata/   # hits in fixtures

Fix

Replace them with synthetic data: generated names, @example.com addresses, numbers that fail the Aadhaar checksum.

A dump was shared with a vendor or a contractor.

Check

grep -rl 'pg_dump\|mysqldump' /var/log/<transfer logs>   # where dumps were sent

Fix

Ask for the copy to be deleted, and share only masked extracts under an agreement.

The default umask 022 creates files as 644.

Check

ls -l /var/backups/   # r for group or other: readable by others on the host

Fix

chmod 600 the files, and set umask 077 in the backup job.

Off-site copies protected only by S3's default SSE-S3 (on for every new object since January 2023), which anyone who can read the bucket can still read, or copied to storage with no encryption at all.

Check

aws s3api get-bucket-encryption --bucket <bucket>   # AES256 is SSE-S3; aws:kms with your own key limits who can decrypt

Fix

Encrypt off-site copies with SSE-KMS under a key only the backup and restore roles can use, or client-side.

A bucket policy or ACL that allows public or cross-account reads.

Check

aws s3api get-public-access-block --bucket <bucket>   # all four settings should be true

Fix

Block public access on the bucket and remove the grants.

Old backups never deleted, so years of data sit in one place.

Check

ls -lt /var/backups/ | tail   # the oldest files and their dates

Fix

Delete backups beyond the retention period, and rotate in the job (scripts/backup_rotate.sh).

Export jobs write files and no job removes them.

Check

./scripts/retention_cleanup.sh -d 365 -p '*.csv' /srv/exports   # dry run: lists what a 365-day limit would delete
du -sh /srv/exports   # how much is kept

Fix

Agree a retention period per data set, review the dry run, then schedule it with -y and a deletion log.

Log retention left at "forever" on the log platform.

Check

aws logs describe-log-groups --query 'logGroups[?!retentionInDays].logGroupName'   # on CloudWatch: groups that never expire

Fix

Set a retention period on every log group.

Nobody deletes for fear of deleting the wrong thing.

Check

./scripts/retention_cleanup.sh -d 365 -p '*.csv' /srv/exports   # the dry run shows exactly what would go

Fix

Make the dry run part of review, and keep the deletion log as the record.