Python errors and fixes for DevOps

A different interpreter runs the script than the one you installed into, for example the system python3 instead of the venv's. Each interpreter has its own site-packages, so the package is simply not on this one's path.

Check

python3 -c 'import sys; print(sys.executable)'   # run it the way the job runs: a path outside your venv is this cause
head -1 script.py   # which interpreter the shebang names

Fix

Run the script with the environment's own interpreter (/opt/app/.venv/bin/python script.py) in the crontab, unit file or CI step.

The package was never installed in this environment, so no interpreter here can import it.

Check

python3 -m pip show requests   # 'Package(s) not found': it is not installed for this interpreter

Fix

Install it into the interpreter that runs the script: python3 -m pip install requests (inside the venv), and add it to requirements.txt.

The import name differs from the package name, so installing the name you import installs nothing useful: import yaml comes from PyYAML.

Check

python3 -m pip show PyYAML   # installed under its distribution name?

Fix

Install the distribution that provides the module (pip install PyYAML for yaml, opencv-python for cv2).

The name moved or was removed in a new major version of the library, so the module exists but no longer has it.

Check

python3 -c 'import y; print(y.__file__, getattr(y, "__version__", "?"))'   # a newer version than your code was written for
pip index versions y

Fix

Import from the library's current location, or pin the version your code was written for until you update the code.

A local file shadows the library (a json.py or requests.py next to your script): Python imports your file, which does not have the name.

Check

ls *.py   # anything named like the library?
python3 -c 'import y; print(y.__file__)'   # your folder, not site-packages: this cause

Fix

Rename the local file and delete its __pycache__.

A circular import: two of your modules import each other, so one asks for a name before the other has finished defining it.

Check

Read the traceback: the same two modules of yours appear in the import chain

Fix

Move the shared code into a third module that both import.

A file of yours has the library's name (requests.py, json.py, email.py), so import requests imports your file, which imports itself.

Check

python3 -c 'import requests; print(requests.__file__)'   # your folder, not site-packages: this cause
ls *.py

Fix

Rename the file that shadows the library, and delete its __pycache__.

The module and a class in it share a name: after import datetime, now() is datetime.datetime.now().

Check

python3 -c 'import datetime; print(dir(datetime))'

Fix

Call the name where it lives (from datetime import datetime).

Two of your modules import each other, so one is used before it has finished loading.

Check

Read the traceback: the same two modules of yours appear twice

Fix

Move shared code into a third module to break the cycle.

You are running the system pip, so the install would go into the OS's Python.

Check

python3 -m pip --version   # a path under /usr/lib: the system Python
ls /usr/lib/python3*/EXTERNALLY-MANAGED   # the marker file pip obeys

Fix

Create a venv and use its pip and python: python3 -m venv .venv && .venv/bin/pip install -r requirements.txt. For command-line tools, pipx install <tool>. Use apt for OS-wide Python packages. Avoid --break-system-packages: a later apt upgrade can break the OS's tools or your packages.

The distribution has another name on PyPI, so the name you asked for does not exist: yaml is PyYAML, cv2 is opencv-python.

Check

pip index versions PyYAML   # versions listed under the real name: this cause

Fix

Use the PyPI distribution name in requirements.txt.

The pinned version does not exist, or has no wheel for this Python version or platform, so pip filters it out.

Check

pip index versions PyYAML   # is your pinned version in the list?
python3 -V; pip debug --verbose | head   # the Python version and platform tags pip accepts

Fix

Pin a version that exists for your Python, or use a Python version the package supports.

A private index or proxy is not configured, so pip searches an index that does not have the package.

Check

pip config list   # index-url and extra-index-url in effect

Fix

Configure the index in pip.conf or with PIP_INDEX_URL where the job runs.

cron or the CI step runs /bin/sh, and dash has no source.

Check

ls -l /bin/sh   # -> dash: this cause

Fix

Do not activate at all in automation (see the next cause); in sh scripts, . (dot) is the POSIX spelling of source.

Activation was never needed: it only puts the venv's bin first on PATH for the current shell. Calling the venv's interpreter directly does the same job.

Check

/opt/app/.venv/bin/python -c 'import sys; print(sys.prefix)'   # prints the venv: the interpreter alone uses it

Fix

Call /opt/app/.venv/bin/python script.py directly.

The effective execution policy is Restricted (or AllSigned), so Activate.ps1 is blocked before it starts.

Check

Get-ExecutionPolicy -List   # the first scope that is not Undefined wins

Fix

Allow local scripts for your user: Set-ExecutionPolicy -Scope CurrentUser RemoteSigned, or skip activation and run .venv\Scripts\python.exe script.py.

Only python3 is installed, so the name python does not exist.

Check

command -v python3 python   # only python3 printed: this cause

Fix

Call python3 (or the venv's bin/python). On Debian and Ubuntu, apt install python-is-python3 adds the python name if you need it.

PATH under cron or sudo is shorter than your shell's, so a Python in /usr/local/bin or your home is not found there.

Check

echo $PATH   # compare with the job's PATH (cron: /usr/bin:/bin)

Fix

Use absolute interpreter paths in automation, or set PATH where the job is defined.

An unpinned transitive dependency (urllib3, pulled in by requests) moved to a new major version that removed the argument.

Check

pip freeze | grep -i urllib3   # 2.x: this cause
pip show requests | grep Requires

Fix

Pin every package to the versions you tested (pip freeze), then update the code to the new API (allowed_methods) in its own change.

The body of the block is not indented further than its header.

Check

python3 -m py_compile script.py   # names the line

Fix

Indent the block by four spaces more than its header line.

An editor mixed in tabs, so a line that looks indented is not indented the way Python counts it.

Check

cat -A script.py | sed -n '9,14p'   # tabs show as ^I

Fix

Replace tabs with spaces and set the editor to four spaces.

A line is indented more than the statements around it.

Check

python3 -m py_compile script.py   # names the line

Fix

Align the line with the other statements of its block.

Code pasted from a chat or web page brought extra leading spaces or non-breaking spaces.

Check

cat -A script.py | sed -n '1,20p'   # unexpected characters before the code

Fix

Re-indent the pasted lines, or run a formatter.

Tabs and spaces are mixed in the same block.

Check

grep -nP '^\t' script.py   # every line that starts with a tab

Fix

Replace tabs with four spaces throughout (expand -t 4), or let your editor convert them.

A closing bracket or quote is missing in a long call or literal.

Check

python3 -m py_compile script.py   # names the opening bracket

Fix

Close the bracket Python points at; check the lines after it for the missing one.

A missing import.

Check

grep -n 'json' script.py   # used, but never imported?

Fix

Import the name at the top of the module.

A typo in the name, or a different case (Path and path).

Check

grep -n -i 'path' script.py

Fix

Use the name exactly as it is defined.

The name is set only inside an if branch, a loop that did not run, or another function.

Check

python3 -m py_compile script.py   # passes: a NameError only appears when the line runs

Fix

Set the variable before the branches that may skip it.

Assigning to a name anywhere in a function makes it local in the whole function, so total += 1 cannot read the module-level total.

Check

Find every assignment to the name inside the function: each one makes it local

Fix

Pass the value in and return the new one. Use global or nonlocal only when you really mean to share it.

The variable is assigned only in a branch or in a try block that failed, and read afterwards (often conn in a finally block).

Check

Read the branch or try block that sets it: can it be skipped?

Fix

Initialise it before the branch or the try (conn = None, then if conn is not None: conn.close() in finally).

A parameter was added, renamed or removed and not every caller was updated.

Check

python3 -c 'import inspect, alerts; print(inspect.signature(alerts.send_alert))'   # what it takes now

Fix

Call it with the parameters it has now; give a new parameter a default so old callers keep working.

A library changed its signature in a new version.

Check

pip show <library>   # which version is installed

Fix

Read the signature of the installed version, or pin the version your code was written for.

A method defined without self, or called on the class instead of an instance, so the arguments shift by one.

Check

Read the def line and the call

Fix

Add self, or call it on an instance.

A function returned None: it found nothing, or reached its end without return.

Check

print(repr(x)) just before the line
Read the function: every path should return something

Fix

Check for None where you call the function (if host is None:), report it, and continue or exit.

A dict .get() returned None for a missing key.

Check

print(repr(d.get(key)))   # None for some records

Fix

Give .get() a default, or handle the missing key explicitly.

The result of an in-place method was assigned: list.sort() and dict.update() change the object and return None.

Check

grep -n '= .*\.sort()' script.py

Fix

Call the method on its own line, or use sorted() which returns a new list.

A default of None that is never replaced with a real value.

Check

print(repr(x))   # None at that point

Fix

Create the value when it is None (if tags is None: tags = []).

re.match(), re.search() or .get() found nothing and returned None.

Check

print(repr(m))   # None for the input that failed

Fix

Check the match before using it (if m is None: ...).

An assignment from a method that returns None (items = items.append(x)).

Check

Read the line that sets the variable

Fix

Call the method without assigning its result.

An optional field is absent from some records, and the code indexes it with [].

Check

python3 -c "import json; print([u for u in json.load(open('users.json')) if 'email' not in u][:3])"   # records without the key

Fix

Use .get(key, default) for optional keys.

os.environ['X'] for a variable that is not set; cron and CI start with few variables.

Check

env | cut -d= -f1 | sort   # the variable names this process has, without their values

Fix

Set the variable where the job is defined; for required settings, fail early with a clear message.

sys.argv[1] when the script was run without an argument.

Check

print(sys.argv)

Fix

Use argparse, which reports a missing argument clearly.

line.split()[n] on an empty or truncated line.

Check

print(len(fields), repr(line))   # fewer fields than expected

Fix

Catch (ValueError, IndexError) per line and count the skipped ones.

A truncated or malformed line, so another field ends up where the number should be.

Check

grep -n -v -E '[0-9.]+$' access.log | head   # lines that do not end in a number

Fix

Validate each value where you read it; skip and count bad records instead of stopping.

A unit or a separator in the value ("80%", "1,024").

Check

print(repr(value))

Fix

Strip the unit or separator explicitly before converting.

The line is split in the wrong place, for example a comma inside a quoted CSV field.

Check

print(line.split(','))

Fix

Parse structured formats with their modules (csv, json).

os.environ values, input(), sys.argv and CSV fields are always strings.

Check

print(repr(limit), type(limit))   # '90' <class 'str'>

Fix

Convert once where the value is read: limit = int(os.environ.get("DISK_ALERT_PERCENT", "90")); for text, f"{count} hosts" instead of +.

An argparse option without type=int.

Check

grep -n 'add_argument' script.py

Fix

Add type=int to the option.

A field contains the separator: a comma inside a quoted CSV value, a colon in an IPv6 address.

Check

print(repr(line), line.split(','))

Fix

Use the csv module for CSV; split with a limit (line.rsplit(':', 1)).

A blank, short or comment line.

Check

grep -n -v -E '^[^,]+,[^,]+$' hosts.csv | head

Fix

Skip and count lines with the wrong number of fields.

A dict iterated without .items(), so each item is a key string.

Check

Read the for line

Fix

for k, v in d.items().

datetime.now() and utcnow() are naive, while ISO times with +00:00 from an API are aware.

Check

python3 -c "from datetime import datetime; print(datetime.now().tzinfo)"   # None: naive

Fix

Use aware times throughout: datetime.now(timezone.utc); attach a zone to naive inputs with .replace(tzinfo=...) only when you know it.

subprocess.run without text=True returns bytes.

Check

print(type(out), repr(out[:40]))   # <class 'bytes'>, b'...'

Fix

Ask for text: subprocess.run(..., capture_output=True, text=True).

A file or socket opened in binary mode ('rb', or a raw socket) is being mixed with str.

Check

print(f.mode)   # 'rb'

Fix

Open in text mode with an encoding, or decode once with .decode('utf-8').

A datetime, set, Decimal or Path is in the data being dumped.

Check

print({k: type(v).__name__ for k, v in report.items()})   # the type the message names

Fix

Convert first: .isoformat() for datetimes, list() for sets, str() for paths; or json.dumps(obj, default=str).

An unquoted value that contains : (a URL with a port is fine, note: run at: 02:00 is not).

Check

python3 -c "import sys, yaml; yaml.safe_load(open(sys.argv[1]))" deploy.yaml   # line and column

Fix

Quote the value (note: "run at: 02:00").

Indentation that does not line up, or a tab: found character '\t' that cannot start any token.

Check

sed -n '5,8p' deploy.yaml | cat -A   # tabs show as ^I

Fix

Indent with spaces only, aligned with the sibling keys.

A value that starts with a character YAML reserves (*, &, @, a backtick, %) and is not quoted.

Check

sed -n '7p' deploy.yaml

Fix

Quote the value.

No base case, or one that is never reached.

Check

Read the repeated frame in the traceback: "[Previous line repeated 996 more times]"

Fix

Fix the base case, or turn the recursion into a loop with an explicit stack.

Following a cycle: symlinks, parent links, a property that calls itself.

Check

find /path -type l   # symlinks that point back up

Fix

Track what you have visited; for directory trees use os.walk, which does not follow symlinks by default. Do not raise sys.setrecursionlimit to hide it: it moves the crash and can crash the interpreter.

The file is not UTF-8, often Windows-1252 or Latin-1 from another system.

Check

file -i customers_export.csv   # charset=iso-8859-1 or unknown-8bit
python3 -c "print(open('customers_export.csv','rb').read()[1820:1840])"   # the bytes around the position in the message

Fix

Open with the encoding the file was written in: open(path, encoding='cp1252').

open() without encoding= uses the machine's locale, so the same script reads differently on another machine.

Check

python3 -c 'import locale; print(locale.getpreferredencoding())'

Fix

Always pass encoding=. errors='ignore' only hides corrupted data.

A relative path resolves against the working directory, which differs under cron, systemd or CI.

Check

python3 -c 'import os; print(os.getcwd())'   # run the way the job runs: not the script's folder

Fix

Anchor files that ship with the script to it: Path(__file__).resolve().parent / 'config.json'.

A parent directory is missing, or the path is not the one you meant.

Check

ls -l "$(dirname script.py)"
print(repr(path))

Fix

Create the directory first (path.parent.mkdir(parents=True, exist_ok=True)) and build paths with pathlib.

The path is not what you meant (a missing separator puts the file in the parent directory), and that directory is not writable.

Check

print(repr(path)) before the write

Fix

Build paths with pathlib (Path('/srv/archive') / name).

The job runs as a different user than in your test, or the directory is owned by root.

Check

id; ls -ld /srv /srv/archive   # who you are, and who may write there

Fix

Write where the job's user owns the directory, or give that user its own directory. Not chmod 777.

SELinux or AppArmor denies it although the mode bits allow it (common for services on RHEL and Fedora).

Check

ls -Z /srv/archive; ausearch -m avc -ts recent   # as root: an AVC denial names the process and the path

Fix

Give the directory the right label (semanage fcontext and restorecon), never setenforce 0.

Files or responses are opened and never closed.

Check

ls /proc/<pid>/fd | wc -l   # rising over time: a leak

Fix

Open files and responses in with blocks.

A new connection per request without a session or pool.

Check

ls -l /proc/<pid>/fd | grep -c socket

Fix

Reuse a requests.Session.

The limit itself is low for this service: the soft limit is often 1024 (systemd: LimitNOFILE=).

Check

grep 'open files' /proc/<pid>/limits   # the running process's limit, not your shell's (ulimit -n)

Fix

Raise the limit (LimitNOFILE= in the unit) only for a service that really needs many files, after any leak is fixed.

The file system is full.

Check

df -h /backups   # 100% used
du -xh --max-depth=1 /backups | sort -h | tail   # what fills it

Fix

Free space or write elsewhere; write to a temporary name and rename only when the write succeeded, so no truncated file is left as if complete.

There is free space but no free inodes (millions of small files).

Check

df -i /backups   # IUse% 100%

Fix

Delete or archive the small files.

Deleted files are still held open by a running process, so their space is not freed.

Check

lsof +L1   # deleted files still open

Fix

Restart or signal the process that holds them (for a logger, reopen its log).

/tmp is a small tmpfs in memory.

Check

df -h /tmp   # tmpfs

Fix

Write large temporary files to a disk-backed directory (tempfile.mkdtemp(dir=...)).

The reader closed the pipe early (head, grep -q), and Python raises on the next write to stdout.

Check

python3 list_hosts.py | head -1; echo "${PIPESTATUS[0]}"   # non-zero with a traceback

Fix

Catch it at the top level: except BrokenPipeError: then os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno()) and sys.exit(1), as the Python documentation for the signal module shows.

For sockets: the other side closed the connection.

Check

Did the server restart or time out the connection at that moment?

Fix

Treat it as a closed connection and reconnect if that makes sense.

The kernel's out-of-memory killer: the machine ran out of memory and chose this process.

Check

dmesg -T | grep -i -E 'killed process|out of memory'   # names the process

Fix

Stream input line by line, process in batches, and bound caches.

A container or cgroup memory limit, lower than the host's free memory (Kubernetes reports OOMKilled).

Check

dmesg -T | grep -i 'memory cgroup out of memory'
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'   # OOMKilled

Fix

Reduce memory use as above, or raise the container's limit to what you measured.

Reading a whole large file, or an unbounded cache, makes memory grow with the input.

Check

/usr/bin/time -v python3 script.py   # Maximum resident set size

Fix

Iterate over the file instead of read(); give caches a size limit.

Nothing listens on that host and port: the service is down, or the port is wrong.

Check

python3 scripts/port_check.py host:8080   # refused from here
ss -ltnp | grep 8080   # on the server: no line means nothing listens

Fix

Start the service, or use the right host and port.

The service listens on 127.0.0.1 only, so connections from other hosts are refused.

Check

ss -ltnp | grep 8080   # on the server: 127.0.0.1:8080 means local only

Fix

Make it listen on the right interface, and allow only the hosts that need it in the firewall.

A firewall rejects the traffic with a reset rather than dropping it.

Check

sudo iptables -S | grep -i reject   # or nft list ruleset

Fix

Allow the traffic from the client's address.

A firewall, security group or network ACL silently drops the traffic.

Check

python3 scripts/port_check.py api.example.internal:443   # timed out, not refused
nc -vz -w 5 api.example.internal 443

Fix

Allow the traffic from the client's address in the security group or firewall.

The address is not reachable from here: a private IP from outside the VPC or VPN, no route, no NAT gateway.

Check

ip route get <address>   # which route, if any

Fix

Use an address that is reachable from where the script runs, or add the route or NAT gateway.

A proxy is required (HTTPS_PROXY) and not set, or is set and not reachable.

Check

env | grep -i -o '^[a-z_]*proxy'   # which proxy variables are set

Fix

Set the proxy where the job runs.

The host is down and nothing answers for it.

Check

Does it time out from other hosts too?

Fix

Bring the host back; keep a short connect timeout, separate from the read timeout: timeout=(5, 30).

The server uses a private CA that requests does not trust: requests uses the certifi bundle, not the system store.

Check

openssl s_client -connect host:443 -servername host -showcerts </dev/null   # the issuer at the top of the chain

Fix

Trust the right CA: verify='/etc/ssl/certs/corp-root-ca.pem' or REQUESTS_CA_BUNDLE. Never verify=False: it accepts any certificate, including an attacker's.

The server does not send its intermediate certificate, so clients that do not cache it cannot complete the chain.

Check

openssl s_client -connect host:443 -servername host -showcerts </dev/null | grep -c 'BEGIN CERTIFICATE'   # 1: only the leaf was sent

Fix

Fix the server's chain: configure the full chain file.

The certificate expired or does not name the host.

Check

python3 scripts/cert_checker.py host:443   # expiry and names

Fix

Renew the certificate, or connect with the name it was issued for.

A TLS-inspecting proxy replaces the certificate with one signed by its own CA.

Check

openssl s_client -connect host:443 -servername host </dev/null | grep issuer   # the proxy's CA

Fix

Trust the proxy's CA explicitly where the job runs.

The server accepted the connection but answers slowly or not at all.

Check

curl -m 10 -sv http://status.internal:8080/api/health   # connects, then waits
time python3 status_check.py

Fix

Keep the timeout, catch requests.exceptions.RequestException, report the service as down, and exit non-zero; retry with backoff if it makes sense.

A dropped connection with no reset (a NAT or firewall forgot it), so only a timeout notices.

Check

Does it fail after an idle period and work on retry?

Fix

Retry idempotent calls once with a new connection, and keep the timeout.

The body is an HTML error page (502, 503) or empty, because the call failed.

Check

curl -s -o /dev/null -w '%{http_code} %{content_type}\n' URL   # not 200, or text/html

Fix

Call resp.raise_for_status() (or check status_code) before resp.json().

A login redirect returned an HTML page instead of the API's answer.

Check

print(resp.status_code, resp.url, resp.headers.get('Content-Type'))   # a login URL

Fix

Send the credentials the API expects, and treat a redirect to a login page as an authentication failure.

A typo, or an internal name used outside the network or VPN where it exists.

Check

getent hosts db.internal   # no output: the name does not resolve here

Fix

Use a name that resolves where the script runs.

DNS is not configured in this container or CI runner.

Check

cat /etc/resolv.conf   # nameserver lines

Fix

Fix DNS for that environment (the container's network or the runner's resolver).

Retries in a tight loop multiply the request rate exactly when the API is struggling.

Check

Count requests per minute in the API's logs
resp.headers.get('Retry-After')

Fix

Limit attempts, back off exponentially with random jitter, and honour Retry-After.

Many hosts retry in step after an outage, all at the same moment.

Check

Do the 429s start right after a restart or outage, from many hosts?

Fix

Add random jitter to retries and schedules.

No pagination or caching: the script re-fetches everything each time.

Check

Count calls per run

Fix

Page through results once, and cache what does not change.

Another process, often an older copy of the same script, already listens on the port.

Check

ss -ltnp 'sport = :9100'   # the process that holds it
lsof -i :9100

Fix

Stop the old process or choose another port.

The previous run's socket is in TIME_WAIT and the server does not set SO_REUSEADDR.

Check

ss -tan 'sport = :9100'   # TIME-WAIT lines, no LISTEN

Fix

Set sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) before bind (socketserver: allow_reuse_address = True).

The password is wrong or was rotated, or the variable that holds it is empty or not set where the job runs (cron, CI, the container). The server finds the account, the password does not match, and it answers 1045.

Check

mysql -h db.example.internal -P 3306 -u app -p -e 'SELECT USER(), CURRENT_USER();'   # with the password from the secret store: works here, so the script gets another password
python3 -c "import os; print(repr(os.environ.get('DB_USER')), bool(os.environ.get('DB_PASSWORD')))"   # run it the way the job runs: shows whether the credentials reach the process, without printing the password; False: empty or not set

Fix

Give the process the current password from the environment or a secret store. In a URL, percent-encode an @, : or / in the password.

Another account matched first. MySQL sorts accounts by host, most specific first ('10.0.3.15', then '10.0.3.%', then '%'), and uses the first that matches, so 'app'@'%' with the password you know can lose to a more specific 'app' row with another password, or to an anonymous ''@'localhost' when you connect from the server itself.

Check

mysql -h db.example.internal -P 3306 -u app -p -e 'SELECT USER(), CURRENT_USER();'   # USER() is the identity the client supplied (user@client-host as the server sees it); CURRENT_USER() is the account actually used to authenticate. ''@'localhost' or another host than you expect: another account matched
SELECT user, host, plugin FROM mysql.user WHERE user IN ('app', '') ORDER BY host;   # as an administrator: every account that could match; user '' is an anonymous account

Fix

Remove the anonymous accounts (DROP USER ''@'localhost';, as mysql_secure_installation does) or the stale, more specific row, then check CURRENT_USER() again.

No account exists for this user from this host: 'app'@'localhost' does not let app in from 10.0.3.15. The client's address may also not be the one you expect (NAT, a proxy, a new subnet). With no row to match, the server answers 1045.

Check

SELECT user, host FROM mysql.user WHERE user = 'app';   # as an administrator: no host pattern covers the address in the message

Fix

As an administrator, create an account for the application's network, for example CREATE USER 'app'@'10.0.3.%' IDENTIFIED BY ... and GRANT SELECT, INSERT ON shop.* TO 'app'@'10.0.3.%'. The server still picks the account by its host-matching rules and the accounts that exist, so run the CURRENT_USER() check afterwards to see which one it chose.

Socket and TCP are different paths: the mysql client and mysqlclient reach localhost through the Unix socket and 127.0.0.1 through TCP, so the two can match different accounts.

Check

mysql -u app -p -e 'SELECT CURRENT_USER()'; mysql -h 127.0.0.1 -u app -p -e 'SELECT CURRENT_USER()'   # different accounts, or one refused

Fix

Connect the way the account was set up, or create the account for the host the server sees.

The account signs in another way. Root on Debian, Ubuntu and MariaDB uses the unix_socket or auth_socket plugin, so a password login as root is refused (1698). An Amazon RDS account with IAM authentication needs a fresh token (valid for 15 minutes) over TLS, not a password.

Check

SELECT user, host, plugin FROM mysql.user WHERE user IN ('app', 'root');   # unix_socket, auth_socket or AWSAuthenticationPlugin

Fix

For root, create an application user instead of changing root's plugin; with RDS IAM authentication, connect with a token from aws rds generate-db-auth-token over TLS.

The server is not running, or the host or port (3306) is wrong, so the host answers with a refusal.

Check

python3 scripts/port_check.py db.example.internal:3306   # from where the script runs: refused
systemctl status mysql   # on the server (or mysqld, mariadb): inactive

Fix

Start the server, or use the right host and port.

The server listens on 127.0.0.1 only (bind-address = 127.0.0.1, which the Debian and Ubuntu packages set) or has skip_networking on, so other hosts are refused.

Check

mysql -e "SHOW VARIABLES WHERE Variable_name IN ('bind_address', 'port', 'skip_networking', 'socket')"   # on the server: what it actually uses
ss -ltnp | grep 3306   # on the server: 127.0.0.1:3306 means local connections only

Fix

Set bind-address to the private interface, restart, and allow only the application's hosts in the firewall or security group.

A firewall or security group drops the traffic, so nothing answers and the client times out.

Check

python3 scripts/port_check.py db.example.internal:3306   # timed out, not refused

Fix

Allow port 3306 from the application's hosts only.

The host name does not resolve from where the script runs.

Check

getent hosts db.example.internal   # no output: DNS, not MySQL

Fix

Use a name that resolves where the script runs.

For localhost (2002), the mysql client and mysqlclient use the Unix socket, and its path differs from the server's or the server is down. PyMySQL uses TCP unless you pass unix_socket=.

Check

mysql -e 'SHOW VARIABLES LIKE "socket"'   # on the server: the path it uses

Fix

Connect to 127.0.0.1, or pass the server's socket path (unix_socket=).

A typo, the name for another environment (shop_prod and shop), or a space or newline left in the variable.

Check

python3 -c "import os; print(repr(os.environ.get('DB_NAME')))"   # the exact name the script gets: repr shows case, spaces and newlines

Fix

Use the right name for this environment.

The database was never created on this server: provisioning or the first migration has not run.

Check

SHOW DATABASES LIKE 'shop%';   # as an administrator: not there, or there under another name

Fix

Create it: CREATE DATABASE shop CHARACTER SET utf8mb4; and grant the application user access to it.

The host or port points at another server than you think, such as another environment's.

Check

mysql -h db.example.internal -u app -p -e 'SELECT @@hostname, @@port; SHOW DATABASES'   # which server you reached, and the databases this user may see

Fix

Fix the host and port in the configuration for this environment.

Database names are case-sensitive on Linux servers by default, so Shop is not shop.

Check

SHOW DATABASES;   # the exact case

Fix

Use the name exactly as it is listed.

Connections opened per request, thread or loop and never closed, so they pile up until the limit.

Check

SELECT user, SUBSTRING_INDEX(host, ':', 1) AS client, command, COUNT(*) FROM information_schema.processlist GROUP BY 1, 2, 3 ORDER BY 4 DESC;   # many Sleep from one client: a leak

Fix

Close every connection in a finally block (or contextlib.closing).

Pools in many processes and workers add up to more than max_connections (default 151).

Check

SHOW VARIABLES LIKE 'max_connections';
SHOW GLOBAL STATUS LIKE 'Threads_connected';   # at the limit, spread over many clients

Fix

Use one small pool per process and size it so processes × pool size stays well under max_connections.

Slow queries or a lock wait: every request holds its connection longer, so connections pile up although nothing leaks.

Check

SELECT id, user, time, state FROM information_schema.processlist WHERE command = 'Query' ORDER BY time DESC LIMIT 10;   # many long-running queries, or waiting for a lock

Fix

Fix the slow query or the lock that holds everyone up. Raising max_connections only buys time: each connection costs server memory.

Idle connections held for hours, kept until wait_timeout (8 hours by default).

Check

The processlist above: many Sleep with a large time

Fix

Return connections to the pool promptly, and recycle idle ones.

The connection sat idle longer than the server's wait_timeout (8 hours by default) or something on the path: a proxy, a load balancer or NAT (an AWS NAT gateway drops idle connections after 350 seconds). The server or the middlebox closed it, and the next query finds it gone.

Check

SELECT @@global.wait_timeout;   # what a script's connection gets; in the mysql shell, the session's wait_timeout comes from interactive_timeout instead
Compare the idle time before the failing query with these timeouts

Fix

Do not keep connections idle across long waits: connect per run, or check before use (in PyMySQL, conn.ping() and a new connection if it raises; its reconnect=True is deprecated in recent versions; in SQLAlchemy, pool_pre_ping=True and pool_recycle below the shortest idle timeout on the path).

A query or row larger than max_allowed_packet (64 MB by default in MySQL 8.0, 16 MB in MariaDB): the server drops the connection instead of reading it.

Check

SELECT @@global.max_allowed_packet;   # smaller than the statement or row you send

Fix

Send large data in batches, or raise max_allowed_packet on the server.

The server restarted, failed over, or was killed (the OOM killer), which closes every connection.

Check

SHOW GLOBAL STATUS LIKE 'Uptime';   # seconds since the server started: shorter than the job has run means it restarted
journalctl -u mysql --since '1 hour ago'   # on the server (or mysqld, mariadb): a restart, a crash, the OOM killer

Fix

Find why it restarted; retry work that is safe to repeat.

The client reads a large result more slowly than net_write_timeout (60 seconds) allows, so the server gives up on it.

Check

SELECT @@global.net_write_timeout;

Fix

Read results faster (stream them with an unbuffered cursor and process in batches), or fetch less at once.

The password is wrong or was rotated, or its variable is empty in this environment.

Check

psql "host=db.example.internal port=5432 dbname=shop user=app" -c 'SELECT current_user'   # with the password from the secret store: works, so the job gets a different one
grep -A1 'password authentication failed' /var/log/postgresql/postgresql-*.log | tail -4   # on the server (Debian, Ubuntu; managed services have their own log): for a wrong password the DETAIL line names the pg_hba.conf line it matched

Fix

Give the job the current password from the environment or a secret store.

The role does not exist on this server (another environment's server, or never created), has no password, or its VALID UNTIL date has passed.

Check

grep -A1 'password authentication failed' /var/log/postgresql/postgresql-*.log | tail -4   # DETAIL: Role "app" does not exist, has no password assigned, or has an expired password
SELECT rolname, rolvaliduntil, rolpassword IS NULL AS no_password FROM pg_authid WHERE rolname = 'app';   # as a superuser; no row: the role does not exist here

Fix

Connect to the right server, or create the role, set its password, or move VALID UNTIL, as the log says.

A stale PGPASSWORD in the environment wins over ~/.pgpass. ~/.pgpass is ignored when its group or others can read it (it needs mode 0600), or the job runs as another user with another HOME.

Check

env | grep -o '^PG[A-Z]*'   # PGPASSWORD set? Names only, not values
ls -l ~/.pgpass   # must be -rw------- and owned by the job's user

Fix

Unset the stale PGPASSWORD; chmod 600 ~/.pgpass for the job's user.

A password with @, : or / inside a postgresql:// URL that is not percent-encoded, so the URL is cut in the wrong place and the server gets part of it.

Check

Does the password contain @, :, / or %?

Fix

Pass connection settings as keyword arguments, or encode the password with urllib.parse.quote(password, safe="") in the URL.

The server requires scram-sha-256 but the role's password was stored as MD5 before an upgrade, so it can never match.

Check

SELECT rolname, rolpassword LIKE 'SCRAM%' AS scram FROM pg_authid WHERE rolname = 'app';   # as a superuser: false

Fix

Set the role's password again (\password app in psql) with password_encryption = scram-sha-256.

The server is not running, or the host or port (5432) is wrong.

Check

pg_isready -h db.example.internal -p 5432   # accepting connections, rejecting connections, or no response

Fix

Start the server, or use the right host and port.

The server listens on localhost only: listen_addresses defaults to localhost, so connections from other hosts are refused.

Check

psql -c 'SHOW listen_addresses'   # on the server: the value it actually uses
ss -ltnp | grep 5432   # on the server: 127.0.0.1:5432 means local only

Fix

Set listen_addresses to the private interface, add a pg_hba.conf line for the application's network, restart, and allow only those hosts in the firewall.

A firewall or security group drops the traffic, so nothing answers (a time-out, not a refusal).

Check

python3 scripts/port_check.py db.example.internal:5432   # timed out

Fix

Allow port 5432 from the application's hosts only.

The host name does not resolve from where the script runs.

Check

getent hosts db.example.internal   # no output: DNS

Fix

Use a name that resolves there.

No host given, so the client uses the local Unix socket; or localhost inside a container, which is the container itself.

Check

env | grep -E '^PGHOST='   # empty: the socket is used

Fix

Give the host explicitly; in containers, connect to the database's service name, not localhost.

A typo, or the name for another environment.

Check

env | grep -E '^PG(DATABASE|HOST|PORT)='

Fix

Name the right database (dbname=shop).

No database given, so the client uses one named after the user (database "app" does not exist).

Check

Is dbname set in the connection settings?

Fix

Set dbname explicitly.

Case: the name in connection settings is used exactly, but SQL folds unquoted names to lower case, so CREATE DATABASE Shop made shop and dbname=Shop does not find it.

Check

psql -h db.example.internal -U app -d postgres -c '\l'   # the exact names

Fix

Use the name exactly as it is listed.

It exists on another server, cluster or port.

Check

psql -h db.example.internal -U app -d postgres -c 'SELECT inet_server_addr(), inet_server_port()'   # which server you reached

Fix

Fix the host and port, or create it here: createdb -h db.example.internal -O app shop.

The server accepts this client only over TLS (hostssl lines; on Amazon RDS, the rds.force_ssl parameter), and the client connected without it (sslmode=disable, or a driver default).

Check

psql "host=db.example.internal dbname=shop user=app sslmode=require" -c 'SELECT ssl, version FROM pg_stat_ssl WHERE pid = pg_backend_pid()'   # works: the server needed TLS
env | grep -E '^PGSSL'

Fix

Connect with TLS and check the server's certificate: psycopg2.connect(..., sslmode="verify-full", sslrootcert="/etc/ssl/certs/db-ca.pem") (sslmode=require encrypts but does not verify).

There is no pg_hba.conf line for this client's address, user and database at all, or only one for another database or user.

Check

psql -c 'SELECT line_number, type, database, user_name, address, error FROM pg_hba_file_rules'   # on the server, as a superuser: the file as it is now (reload after a change)

Fix

On a server you run, add a hostssl line for the application's network and reload (SELECT pg_reload_conf()).

The client's address is not what you expect (a new subnet, NAT, a container network), so a line that exists does not match.

Check

The host in the message: the address the server saw

Fix

Allow the network the client really comes from, no wider.

The other way round: the client demands TLS and the server has ssl off: server does not support SSL, but SSL was required.

Check

psql -c 'SHOW ssl'   # on the server: off

Fix

Turn on TLS on the server with a certificate; do not lower the client's sslmode.

Thread or process pools in which each worker opens its own connection.

Check

psql -c 'SELECT usename, application_name, count(*) FROM pg_stat_activity GROUP BY 1, 2 ORDER BY 3 DESC'   # as a superuser (the reserved slots let you in): which application holds them

Fix

A small pool per process (psycopg_pool or SQLAlchemy) shared by its threads, and PgBouncer in front when many processes connect.

Connections that are never closed, or left idle in a transaction.

Check

psql -c 'SELECT state, wait_event_type, count(*) FROM pg_stat_activity GROUP BY 1, 2 ORDER BY 3 DESC'   # many idle: leaks or oversized pools; idle in transaction: transactions left open

Fix

Close connections with with; set idle_in_transaction_session_timeout so forgotten transactions end.

More processes × pool size than max_connections (default 100).

Check

psql -c 'SHOW max_connections'

Fix

Size pools from the limit, leaving room for other applications and administrators. Raising max_connections needs a restart and costs memory per connection.

Slow queries or lock waits: each request holds its connection longer, so connections pile up.

Check

The state query above: many active, waiting on Lock

Fix

Fix the query or lock that holds everyone up.

Another connection holds a write transaction longer than this one's timeout: a transaction never committed, or a long batch.

Check

fuser -v jobs.db   # or: lsof jobs.db: which processes have it open
ls -l jobs.db*   # a -journal or -wal file next to it

Fix

Commit promptly (with conn: commits or rolls back the transaction; it does not close the connection); allow a longer wait with sqlite3.connect(path, timeout=30).

Overlapping cron runs, or several threads or processes writing at once.

Check

pgrep -af job.py   # more than one copy

Fix

Stop runs from overlapping (flock -n /run/lock/job.lock python3 job.py) and keep one writer at a time.

A transaction that began by reading and then writes cannot wait for the lock (in WAL mode, its snapshot is out of date), so it fails at once.

Check

sqlite3 jobs.db 'PRAGMA journal_mode;'   # wal
Does it fail immediately, not after the timeout?

Fix

Start write transactions with BEGIN IMMEDIATE (isolation_level="IMMEDIATE") so they wait for the lock up front.

A database browser left open on the file holds a lock.

Check

fuser -v jobs.db   # a GUI tool or sqlite3 shell

Fix

Close it; switch to WAL (PRAGMA journal_mode=WAL) so readers do not block the writer.

The file is on a network file system (NFS, SMB), where locking is unreliable.

Check

findmnt -T jobs.db   # FSTYPE nfs or cifs

Fix

Keep SQLite on local disk, or move to a database server.

Connections or sessions are checked out and never returned: a Session not closed, engine.connect() without with.

Check

print(engine.pool.status())   # checked out at the limit while the database is idle: connections are not returned
create_engine(url, echo_pool="debug")   # logs every checkout and return

Fix

Return connections: with Session(engine) as session: and with engine.connect() as conn:.

More threads or tasks than pool_size + max_overflow need a connection at once.

Check

Count worker threads against 15

Fix

Size pool_size and max_overflow to the real concurrency, within the database's connection limit.

Slow queries or long transactions hold connections, or a session stays open across other slow work (an HTTP call, a sleep).

Check

echo_pool="debug" above: long gaps between checkout and return

Fix

Keep sessions out of slow non-database work, and fix slow queries.

A new engine created per request, each with its own pool.

Check

grep -n 'create_engine' *.py   # called inside a function that runs per request

Fix

Create one engine per process.

The driver is not installed in the interpreter that runs the script.

Check

python3 -c 'import sys; print(sys.executable)'
python3 -m pip show sqlalchemy mysqlclient PyMySQL psycopg2-binary psycopg   # which are missing

Fix

Install the driver into the environment that runs the script (.venv/bin/pip install PyMySQL or "psycopg[binary]").

The URL selects another driver than the one you installed: no driver in the URL means the default, and an upgrade to SQLAlchemy 2.1 changed the PostgreSQL default to psycopg.

Check

python3 -c "import os; from sqlalchemy.engine import make_url; u = make_url(os.environ['DATABASE_URL']); print(u.drivername, u.get_dialect().driver)"   # the driver the URL selects; the password is not printed

Fix

Name the driver in the URL: mysql+pymysql://... or postgresql+psycopg://....

The import name differs from the package: MySQLdb is mysqlclient, mysql.connector is mysql-connector-python, psycopg is psycopg (version 3).

Check

The module name in the message

Fix

Install the package that provides that module.

pip install psycopg2 builds from source and needs pg_config ("pg_config executable not found"); mysqlclient needs the MySQL client headers and a compiler.

Check

The pip install output

Fix

Use a binary wheel (psycopg2-binary, psycopg[binary]) or PyMySQL, or install the build dependencies.

A postgres:// URL: SQLAlchemy 1.4 and later only know postgresql:// ("Can't load plugin: sqlalchemy.dialects:postgres").

Check

The URL's scheme

Fix

Rewrite postgres:// to postgresql://.

The command itself failed; its stderr, printed above the traceback or captured, says why.

Check

Run the exact command from the error in a shell
subprocess.run(..., capture_output=True, text=True) and print .stderr

Fix

Fix the command's cause; catch CalledProcessError only where you can do something useful, and exit non-zero otherwise.

The program is not on PATH for this process (cron: /usr/bin:/bin).

Check

env -i PATH=/usr/bin:/bin sh -c 'command -v kubectl'   # no output: cron cannot find it

Fix

Use the absolute path (/usr/local/bin/kubectl), or set PATH where the job is defined.

A whole command line was passed as one string without a shell, so Python looks for a program named 'kubectl get pods'.

Check

print(args) before the call   # one string instead of a list

Fix

Pass a list of arguments: ['kubectl', 'get', 'pods'].

A stuck remote host or mount (an NFS server that does not answer).

Check

Run the command by hand with `timeout 30`

Fix

Catch subprocess.TimeoutExpired per host, report it, and carry on with the others.

The command is waiting for input or a password prompt that nobody will answer.

Check

ssh -o BatchMode=yes host true   # fails instead of prompting

Fix

Use BatchMode for ssh and pass input explicitly (stdin=subprocess.DEVNULL).

The call is missing await.

Check

python3 -X dev script.py   # shows where the coroutine was created

Fix

Add await (await asyncio.sleep(1)), or schedule it with asyncio.create_task and keep a reference to the task.

The OS or the image moved to Python 3.12 or later, which no longer has distutils, imp, asynchat, asyncore or smtpd.

Check

python3 -V   # 3.12 or later

Fix

Replace the module: version comparison with tuples of ints or packaging.version; imp with importlib.

On 3.13 also cgi, crypt, telnetlib, pipes and others were removed.

Check

python3 -W error::DeprecationWarning script.py   # on the old version: what will break

Fix

pipes.quote becomes shlex.quote; cgi parsing moves to email.message or a framework.

Nothing in the credential chain is set for this process: no AWS_* variables, no profile in ~/.aws, no SSO session, no role.

Check

aws configure list   # where each setting comes from; keys shown masked; <not set> everywhere: this cause
env | grep -o '^AWS_[A-Z_]*'   # which AWS variables are set, without their values

Fix

Give the process an identity the standard way: a role on the instance, task or pod (best), --profile or AWS_PROFILE, or aws sso login. Never paste keys into the script.

cron, systemd and sudo run with another HOME or a clean environment, so ~/.aws and your exported variables are not there.

Check

sudo -u <job-user> env HOME=<job-home> aws sts get-caller-identity   # fails as the job's user

Fix

Configure the profile for the job's user, or set AWS_PROFILE and HOME in the job's definition.

A container on EC2 cannot reach the instance metadata service: IMDSv2 with a hop limit of 1 stops one network hop short of the container.

Check

aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --query 'Reservations[].Instances[].MetadataOptions'   # HttpPutResponseHopLimit 1

Fix

Raise the IMDSv2 hop limit to 2, or better give the container its own role.

The task or pod has no role attached (ECS task role, EKS IRSA or Pod Identity).

Check

aws ecs describe-task-definition --task-definition <name> --query 'taskDefinition.taskRoleArn'   # null: no task role

Fix

Attach a role with the minimum policy the script needs.

No region_name in the code, and no AWS_REGION or AWS_DEFAULT_REGION in this environment.

Check

env | grep -E '^AWS_(DEFAULT_)?REGION='
python3 -c 'import boto3; print(boto3.Session().region_name)'   # None: this cause

Fix

Pass the region explicitly (boto3.client("ec2", region_name=args.region)), or set AWS_REGION where the job is defined.

The region was exported in ~/.bashrc, which cron does not read.

Check

grep -n AWS_ ~/.bashrc ~/.profile

Fix

Set the region in the crontab or unit file, or in the profile's config.

The profile in ~/.aws/config has no region line, or a different profile is used.

Check

aws configure get region   # empty for this profile

Fix

Add region = ... to the profile, or select the right profile.

Temporary credentials (SSO, assume-role, MFA sessions) expire at the end of their session, often after an hour.

Check

aws sts get-caller-identity   # fails with the same error

Fix

Refresh: aws sso login --profile <name>, or assume the role again. Profiles that assume a role (role_arn with source_profile) are refreshed by botocore automatically.

Old AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN values left in the environment win over the refreshed profile.

Check

env | grep -o '^AWS_[A-Z_]*'   # AWS_SESSION_TOKEN set
aws configure list   # Type env for the keys

Fix

Unset the stale AWS_* variables so the profile is used.

The job runs longer than the role's maximum session duration.

Check

aws iam get-role --role-name <role> --query Role.MaxSessionDuration

Fix

Use a profile or instance role that botocore refreshes, split the job, or raise the role's maximum session duration.

InvalidClientTokenId: the access key was deleted or deactivated, belongs to another partition, or is a temporary (ASIA...) key used without its session token.

Check

aws configure list   # which key (masked) and where it comes from
aws iam list-access-keys --user-name "$USER_NAME"   # from a working identity: is the key Active?

Fix

Replace a deleted key; pass the session token with temporary keys.

SignatureDoesNotMatch: the secret key is wrong, or was copied with a space or a line break.

Check

aws configure list   # the source of the secret key

Fix

Set the secret key again from its source, without whitespace.

Stale AWS_* variables in the environment override the profile you meant.

Check

env | grep -o '^AWS_[A-Z_]*'

Fix

Unset the AWS_* variables or fix the profile: use one source of credentials.

The region is not enabled for the account (EC2 answers AuthFailure); a skewed clock gives "Signature expired" instead.

Check

aws account list-regions --region-opt-status-contains ENABLED ENABLED_BY_DEFAULT --query 'Regions[].RegionName'
date -u   # compare with the real time

Fix

Enable the region or use another; sync the clock (chrony or systemd-timesyncd).

The identity's policies do not allow this action, or not on this resource or under this condition (a tag, a region).

Check

aws sts get-caller-identity   # the identity actually used
aws iam simulate-principal-policy --policy-source-arn "$ROLE_ARN" --action-names ec2:StopInstances   # implicitDeny: no policy allows it

Fix

Grant exactly the missing action on the resources it needs (each script's header lists its minimum policy).

A service control policy, a permission boundary or a session policy denies it, even though the identity's policy allows it.

Check

aws sts decode-authorization-message --encoded-message "$MSG" --query DecodedMessage --output text   # EC2's encoded message names the denying policy type

Fix

Change the SCP or boundary with its owner, or run the job in an account where it is allowed.

A resource policy denies it: an S3 bucket policy, a KMS key policy (encrypted volumes and buckets need key permissions too).

Check

CloudTrail: the event's errorCode and errorMessage

Fix

Fix the bucket or key policy, or grant the kms: actions the operation needs.

A different identity is used than you think.

Check

aws sts get-caller-identity   # another role or account than expected

Fix

Select the right profile or role.

Too many requests per second for the account, region and API.

Check

CloudTrail: events with errorCode Throttling or RequestLimitExceeded at that time   # other jobs too?
aws service-quotas list-service-quotas --service-code ec2 --query 'Quotas[].QuotaName' | head

Fix

Fewer and bigger calls (paginators, filters, batch APIs), and jobs spread out in time.

The default (legacy) retry mode gives up after four retries.

Check

grep -c 'reached max retries' job.log

Fix

Create the client with Config(retries={"mode": "adaptive", "max_attempts": 10}): exponential backoff with jitter, and adaptive mode also slows the client down when throttled.

Many threads each retrying on their own make the burst worse.

Check

How many threads call the API at once?

Fix

Share one client between threads and keep the pool small.

A misspelt or non-existent region name, so the host name does not resolve.

Check

python3 -c "import socket; print(socket.getaddrinfo('ec2.eu-west-1.amazonaws.com', 443)[0][4])"   # works for the right region

Fix

Correct the region.

The server has no route to AWS: a private subnet with no NAT gateway and no VPC endpoint for that service.

Check

curl -sS -o /dev/null -w '%{http_code}\n' https://sts.eu-west-1.amazonaws.com/   # times out from this host

Fix

Add a NAT gateway or interface VPC endpoints for the services the job calls.

A proxy is required (HTTPS_PROXY) or is set but unreachable; a firewall blocks port 443.

Check

env | grep -i -o '^[a-z_]*proxy'

Fix

Set HTTPS_PROXY (and NO_PROXY for 169.254.169.254) where a proxy is required.

The service is not offered in that region.

Check

The AWS regional services list for that service

Fix

Use a region that has the service.

A value from argparse or the environment is still a string where the API wants a number or a list.

Check

The message: `Invalid type for parameter MaxResults, value: 50, type: <class 'str'>`

Fix

Convert values once where you read them (type=int in argparse), and pass lists where the API takes lists.

A required parameter is missing: Missing required parameter in input: "Bucket".

Check

aws ec2 stop-instances help   # required parameters

Fix

Pass it.

A parameter name in Python style instead of the API's: Unknown parameter in input: "instance_ids", must be one of: InstanceIds, ....

Check

python3 -c "import boto3; m = boto3.client('ec2', region_name='eu-west-1').meta.service_model; print(list(m.operation_model('StopInstances').input_shape.members))"   # the exact names

Fix

Use the API's exact parameter names (InstanceIds, not instance_ids).

The script calls with DryRun=True but does not catch ClientError; EC2 answers DryRunOperation when the call would succeed and UnauthorizedOperation when it would be refused.

Check

aws ec2 stop-instances --instance-ids "$ID" --dry-run   # the CLI prints the same message

Fix

Catch ClientError and look at e.response["Error"]["Code"]: DryRunOperation means it would succeed (report "would stop"); re-raise or report anything else.

List and describe calls return one page at a time (list_objects_v2 at most 1,000 keys), and the script never follows NextToken, NextContinuationToken or Marker.

Check

aws s3api list-objects-v2 --bucket "$BUCKET" --prefix orders/ --query 'length(Contents)'   # the CLI pages for you: a bigger number
python3 -c "import boto3; print(boto3.client('s3').can_paginate('list_objects_v2'))"

Fix

Use a paginator for every list call: for page in client.get_paginator("list_objects_v2").paginate(Bucket=b): ..., or .search("Contents[]"). Note page.get("Contents", []): an empty page has no Contents key.

Asking for a bigger page only works up to the API's cap.

Check

The API reference's maximum for MaxKeys or MaxResults

Fix

Paginate instead of raising the page size.

The resource takes longer than the waiter's default (EC2 instance waiters poll every 15 seconds, 40 times: 10 minutes).

Check

aws ec2 describe-instances --instance-ids "$ID" --query 'Reservations[].Instances[].[State.Name,StateTransitionReason]'   # still on its way

Fix

Pass WaiterConfig={"Delay": 15, "MaxAttempts": 80}.

The resource is stuck (an instance stuck in stopping, a shutdown script that hangs).

Check

aws ec2 describe-instance-status --instance-ids "$ID" --include-all-instances

Fix

Look at the resource before retrying; catch WaiterError and report which resource did not get there instead of crashing the whole run.

It reached a failure state: the instance was terminated while you waited for running.

Check

The State.Name above: terminated

Fix

Report it as failed; do not retry the wait.

Whole request bodies, records or exception locals are logged "for debugging".

Check

python3 scripts/pii_scanner.py /var/log/app/   # which files and kinds of data
python3 scripts/pii_scanner.py --loose /var/log/app/ | tail -1   # also numbers without a context word

Fix

Log identifiers, not records (customer["id"]); mask at the source; redact what is already written (log_redactor.py --apply on rotated files).

Logs are copied to more systems (shipping, SIEM, support tickets) and kept longer than the data itself.

Check

List every log store the application ships to

Fix

Purge the copies in every log store. Treat a leak to an outside system as a possible breach and follow the incident runbook.

Nobody owns a rule for what may be logged.

Check

Is there a logging rule in the code review checklist?

Fix

Write the rule down and check it in review.

The resource was created before encryption was the default, or restored from an unencrypted snapshot.

Check

python3 scripts/aws/pii_storage_audit.py --all-regions
aws rds describe-db-instances --query 'DBInstances[].[DBInstanceIdentifier,StorageEncrypted]'
aws ec2 get-ebs-encryption-by-default

Fix

Encrypt by copying: an encrypted copy of the snapshot (aws rds copy-db-snapshot --kms-key-id ...), restore from it, switch over, then delete the unencrypted originals. Turn on EBS encryption by default in every region.

Dumps and exports are made outside the managed backup and land wherever there is space.

Check

python3 scripts/pii_scanner.py /srv/dumps   # personal data in plain dump files

Fix

Take backups only through the managed, encrypted path, and delete the stray dumps.

Data is collected for a purpose, but no job erases it when the purpose is served.

Check

python3 scripts/pii_retention.py /srv/exports --days 180   # what is past the period

Fix

Agree a retention period per kind of data with its owner, then enforce it: database deletion jobs, and pii_retention.py --apply for file shares, with the dry-run report reviewed first.

Copies (exports, backups, test environments) are outside every retention rule.

Check

aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET"   # NoSuchLifecycleConfiguration: no rule

Fix

Add S3 lifecycle rules and retention to backups and every copy.

Restoring production is the easiest way to get realistic data.

Check

python3 scripts/pii_scanner.py tests/fixtures/ seed/

Fix

Replace it with synthetic data or an irreversibly masked copy, then delete the copied data everywhere it went (snapshots, dumps, laptops).

Test environments have weaker access control, logging and retention than production.

Check

python3 scripts/aws/pii_storage_audit.py --all-regions   # with test resources tagged too

Fix

Make production backups restorable only into production accounts, and add a masking step as the only path to test.