DevSecOps problems and fixes

A live key from your own account was committed, usually pasted into a config file or script for a quick test. This is the common case.

Check

gitleaks git -v --redact   # commit, file and line, without printing the key
aws sts get-access-key-info --access-key-id <key>   # Account is your account ID: the key is yours
aws iam get-access-key-last-used --access-key-id <key>   # its user and last use; a recent use you cannot explain means it may already be in use
aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<key>   # what it did: management events, last 90 days, this region only; repeat per region

Fix

Rotate first: give the app a new key (better, a role), deactivate the leaked one at once (aws iam update-access-key --status Inactive) and delete it once nothing breaks. Deleting it from git does not revoke it. If CloudTrail shows use you do not recognise, follow "Leaked key already used by an attacker". Then remove it from code, and purge history in a fresh clone: git filter-repo --replace-text replace.txt, re-add origin, push --force, and ask collaborators to re-clone. Forks, pull request refs and caches keep the old commits, which is why rotation comes first.

The key belongs to another AWS account (a vendor's, a customer's, a former employer's). The rule matches the shape, not the owner.

Check

aws sts get-access-key-info --access-key-id <key>   # an Account that is not yours

Fix

Tell that account's owner now, through a channel you trust; only they can revoke it. Then remove it from code and history as above.

It is a documented example key, not a real one. The keys in AWS's documentation end in EXAMPLE and have the same shape.

Check

gitleaks git -v --redact   # then open that file and line: a key ending in EXAMPLE, in docs or a test fixture

Fix

Suppress that one finding: add its Fingerprint to .gitleaksignore with a comment saying why. Not a rule-wide allowlist.

No ignore rule matches .env, so git add . picked it up. /.env only matches the top folder, and .env.local or .env.production need their own pattern.

Check

git check-ignore -v .env   # prints nothing: no rule ignores it
git log --oneline --all -- .env   # empty: only staged, never committed

Fix

git rm --cached .env (your file stays on disk), add .env and .env.* to .gitignore, and keep !.env.example if the template should stay tracked. Nothing left your machine, so nothing needs rotating.

It was forced in with git add -f, which overrides ignore rules.

Check

git check-ignore -v .env   # names a rule, yet the file is staged: it was added with -f

Fix

git rm --cached .env. The existing rule then keeps it out.

It was already committed, and maybe pushed. Unstaging no longer removes it: it is in history.

Check

git log --oneline --all -- .env   # lists commits: it is in history
git branch -r --contains <commit>   # a remote branch listed: it was pushed

Fix

Committed, not pushed: take it out of those commits (git commit --amend, or git reset --soft to before it) before you push. Pushed: rotate every value in it first, then git rm --cached .env, fix .gitignore and purge it in a fresh clone: git filter-repo --path .env --invert-paths; push --force.

Push protection found a secret in one of the commits. Removing it in a newer commit does not help: the older commit is still part of the push.

Check

Read the remote: lines   # GITHUB PUSH PROTECTION, a secret type, commit: and path:
git log --oneline origin/main..HEAD   # every commit this push sends
git show <sha> -- <path>   # the commit that holds it

Fix

In the last commit: remove it, git commit -a --amend, push. Older: git rebase -i to edit that commit, or git reset --soft origin/main and recommit. GitHub did not receive the secret, but rotate it anyway if those commits went anywhere else (another remote, a fork, a CI cache). Do not use the bypass URL for a real secret.

Another ruleset rule failed: signed commits required, restricted file paths, extensions or sizes, or a commit message pattern. GH013 is the code for every ruleset rule, not only secrets.

Check

Read the remote: lines   # a rule such as Commits must have verified signatures
git log --show-signature origin/main..HEAD   # unsigned commits, if signatures are required

Fix

Change the commits to meet the rule (sign them, drop the file, reword the message) instead of asking for a bypass.

A file with a credential was copied in, and a later step deleted it. Deleting in a later RUN only hides the file; the earlier layer still contains it.

Check

trivy image --scanners secret <image>   # the file and the step that added it
docker history --no-trunc <image>   # a COPY of the file, and a later RUN rm of it

Fix

Rotate the credential first: every pulled copy of the image has it, and deleting the tag does not reach them. Then mount it only for the step that needs it: RUN --mount=type=secret,id=<id> with docker build --secret id=<id>,env=VAR. Rebuild, then delete the old image versions from the registry.

The token was passed as ARG or ENV. Build args stay in the image history and ENV in the image config, so it leaks without any file, and a file-only scan may not report it.

Check

trivy image --scanners secret --image-config-scanners secret <image>   # findings in the image config: ENV or history
docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' <image> | cut -d= -f1   # names of baked-in variables, not values
docker history --no-trunc <image>   # build-arg values show in RUN lines: do not paste the output

Fix

Rotate it, use a build secret (above) instead of ARG or ENV, and pass runtime secrets when the container starts, from the platform's secret store.

A local state file was committed, because there is no remote backend and no ignore rule.

Check

git ls-files '*.tfstate*'   # state files tracked by git
grep -c '"password"' terraform/terraform.tfstate   # counts matches without printing values

Fix

Rotate the secrets it holds first. Move state to an encrypted, access-controlled backend (backend "s3" with encrypt = true and locking). git rm --cached the state file, add *.tfstate* to .gitignore, purge it from history.

State is remote, but a copy was pulled for debugging (terraform state pull) or a backup file was committed.

Check

git log --stat --all -- '*.tfstate*'   # who added which file

Fix

Rotate, remove and purge as above; never write state into the working tree.

The secret is in state by design: a password passed as a variable is stored there, whichever backend you use.

Check

grep -n password terraform/*.tf   # a password argument set from a variable

Fix

Let the service manage it (manage_master_user_password for RDS), or read it at run time from a secret store, so state never holds it.

The version pinned in your lock file has the advisory, and a release with the fix exists. The scanner reads the lock file, so the file in the row is where to change it.

Check

osv-scanner scan source -r .   # the row and its fixed version
pip index versions <package>   # the fixed version is available from your index

Fix

Change the constraint at its source (requirements.in, package.json) and regenerate the lock: pip-compile --upgrade-package <package>. Run the tests and the pipeline.

The lock file already has the fix, but the scan read an older image or another manifest, so it reports what shipped before.

Check

docker run --rm <image> pip show <package>   # Version: the one installed in the image you ship

Fix

Rebuild and redeploy through the pipeline; scan the image you ship, not only the source.

The package is a transitive dependency and its parent's range stops the upgrade, so pip-compile cannot resolve the fixed version.

Check

grep -n -A3 <package> requirements.txt   # a '# via' line names the parent

Fix

Upgrade the parent; see "Vulnerable transitive dependency".

The parent caps the child below the fix, and a newer parent release allows it.

Check

grep -n -A3 <package> requirements.txt   # the '# via' line names the parent
pipdeptree -r -p <package>   # every package that requires it, with its range
pip index versions <parent>   # newer parent releases

Fix

Upgrade the parent to a version that allows the fixed child: pip-compile --upgrade-package <parent> --upgrade-package <package>. Run the tests.

No parent release allows the fix yet, so upgrading cannot meet both ranges.

Check

curl -s https://pypi.org/pypi/<parent>/json | python3 -c "import json,sys; print(json.load(sys.stdin)['info']['requires_dist'])"   # the newest release still caps the child below the fix

Fix

Replace the parent, or override the child in a resolver that supports overrides (uv's override-dependencies) and test it. A pip constraints file cannot do this: it only narrows ranges, it never overrides a parent's requirement. Ask the parent's maintainers to widen the range.

Your use does not reach the vulnerable code, or you can block the input that does. Advisories name a function or an input, and many projects never use it.

Check

osv-scanner scan source -r .   # the advisory ID
Read the advisory, then grep your code for the affected function   # no call, or only input you control: not reachable

Fix

Record a scoped exception: osv-scanner.toml [[IgnoredVulns]] with id, ignoreUntil and reason (or .trivyignore.yaml with id, statement and expired_at). Track it; when the exception expires the gate fails again. Not trivy --ignore-unfixed in the gate: it hides every unfixed finding, including the next one that matters.

Your code does reach it with outside input, so the risk is real today.

Check

grep -rn <affected function> app/   # called with data from requests, files or other services

Fix

Mitigate now: validate or block that input, or move to a maintained alternative. Keep the finding open until a fix ships.

A fix exists in a newer release that the advisory database does not list yet.

Check

pip index versions <package>   # releases newer than the advisory's affected range

Fix

Read the release notes; if a later release fixes it, upgrade, and report the missing fixed version to the advisory's source.

The package really is under a licence on your deny list, such as AGPL-3.0, whose terms can require you to publish your own source.

Check

trivy fs --scanners license .   # the package and its licence
pip show -v <package>   # License and classifiers as the package declares them; read its LICENSE file too

Fix

Replace the package, or get a written legal decision before changing the policy.

The metadata is wrong or incomplete: an old classifier, no licence at all (UNKNOWN), or one side of a dual licence. Scanners read metadata, not the licence text.

Check

pip show -v <package>   # differs from the project's LICENSE file, or offers a choice of licences
syft <image> -o cyclonedx-json | grep -i license   # what another tool reads

Fix

Record the real licence with the evidence as a reviewed exception in the scanner's configuration, and ask the project to fix its metadata.

The dependency graph is off for the repository, so there is nothing to compare.

Check

Settings > Advanced Security: Dependency graph   # disabled

Fix

Enable the dependency graph, and run the action on pull_request events: it compares a pull request's base and head.

The repository is private or internal without GitHub Code Security (part of Advanced Security), which the action needs there.

Check

gh api repos/<owner>/<repo> --jq '.visibility, .security_and_analysis'   # private or internal, with Advanced Security or Code Security disabled

Fix

Have an organisation owner or security manager turn on Code Security for the repository.

Typosquat: the name is one letter off a popular package (reqeusts for requests), from a typo in a requirements file or a command.

Check

osv-scanner scan source -r .   # MAL-... ids are known malicious packages
git log -p -S <name> -- 'requirements*'   # the commit that added the misspelled name

Fix

Treat the machine or job that installed it as compromised: rotate every secret it could read, then remove the package and install the intended one.

Dependency confusion: the name is your internal package, but the installer also searched the public index and took a higher version someone published there.

Check

pip config debug   # the index-url and extra-index-url in effect; they can hold a token, do not paste them
pip index versions <internal-name> --index-url https://pypi.org/simple   # found on the public index: confusion

Fix

Rotate what the installing job could read. Then use one index that serves internal names only from inside (a proxy that blocks public copies of them), with no --extra-index-url, and reserve your names on the public index.

The file has ranges like flask>=3.0. They resolve to different versions over time, so scans and SBOMs no longer describe the build.

Check

grep -vE '==|^\s*(#|--hash)' requirements.txt   # requirement lines without an exact pin

Fix

Generate a lock: pip-compile --generate-hashes requirements.in (npm ci with package-lock.json, and so on), and install only from it.

A transitive package is missing from the hashed file, so pip has to fetch it unpinned. The lock was edited by hand, or generated on another Python version or platform.

Check

Read the lines after "These do not:"   # a package that is not in requirements.txt at all: a missing transitive pin

Fix

Regenerate the whole lock with pip-compile --generate-hashes on the Python version CI uses, instead of editing it.

Scorecard reports the same problem as pipCommand not pinned by hash: a pip install in a Dockerfile or workflow names packages directly.

Check

grep -rn 'pip install' Dockerfile .github/workflows   # pip install <name> instead of -r requirements.txt

Fix

Install from the hashed lock file: pip install --require-hashes -r requirements.txt.

Nobody owns triage, so security fixes wait behind routine bumps.

Check

gh pr list --author app/dependabot   # many open, some weeks old

Fix

Merge security fixes first; group routine updates and schedule them.

An auto-merge workflow merges everything, majors included, sometimes before checks finish.

Check

cat .github/workflows/dependabot-automerge.yml   # no update-type condition, or a direct merge without --auto

Fix

Auto-merge only dependabot[bot], only patch and minor (dependabot/fetch-metadata update-type), with gh pr merge --auto so required checks gate it. Leave majors for a person.

Outside input is placed into the SQL text, so the driver sends the attacker's text as part of the query.

Check

semgrep scan --config p/python app/   # file and line of the query
Trace each formatted value back   # one comes from a request, a file or another service: injectable

Fix

Pass values as parameters: cur.execute("... WHERE id = %s", (value,)). The placeholder depends on the driver: %s for psycopg, ? for sqlite3.

The formatted part is an identifier (a table, column or ORDER BY name), which parameters cannot carry.

Check

Look at what is formatted in   # a table or column name, not a value

Fix

Choose it from an allowlist and quote it with psycopg.sql.Identifier.

Only constants are formatted in, so it is a false positive.

Check

Trace each formatted value   # all are literals in the code

Fix

Prefer parameters anyway; otherwise suppress that one line with the reason: # nosemgrep: <rule-id>. See "A finding is a false positive".

Request input is formatted into the HTML without escaping.

Check

semgrep scan --config p/python app/   # the route
curl -s '<url>/search?q=<script>alert(1)</script>'   # the script tag comes back unchanged: exploitable

Fix

Render with a template (Jinja escapes by default), or escape each value with markupsafe.escape.

Only trusted values are formatted in, or the response is not HTML.

Check

curl -sI '<url>/search?q=x'   # Content-Type is not text/html, or nothing user-controlled reaches the string

Fix

Return the right type (jsonify for JSON); if it still matches, suppress that one line with the reason.

The repository is private or internal without GitHub Code Security (part of Advanced Security), which code scanning needs there.

Check

gh api repos/<owner>/<repo> --jq '.visibility, .security_and_analysis'   # private or internal, with Advanced Security or Code Security disabled

Fix

Have an organisation owner or security manager turn on Code Security for the repository, then re-run the workflow.

Code scanning is switched off for the repository, which can happen on public ones too.

Check

Settings > Advanced Security (Code security on older pages)   # code scanning not set up

Fix

Enable code scanning, keep security-events: write on the analysing job only, and re-run the workflow.

The flagged value comes from a fixed source: an allowlist, a constant, a validated enum. The rule cannot see that.

Check

Trace the input to the flagged line   # it never carries outside data: write down why

Fix

Suppress one rule on one line, with the reason above it: # nosemgrep: <rule-id>, #checkov:skip=<ID>:<reason>, a .trivyignore.yaml entry with statement. Ask a second reviewer.

It is test or example data: a fixture, a documented example key.

Check

Look at the path   # tests/, docs/ or examples/

Fix

Mark that line with the reason, or exclude that path for that rule only. Never a bare # nosemgrep, a global rule exclusion, or continue-on-error.

It is not a false positive after all: the safe-looking path has a way round.

Check

Ask a second reviewer to try to break it   # they find an input that reaches it

Fix

Fix the code; see the entry for that rule.

The image has no USER instruction, so it runs as root, often inherited from the base image, and the Pod sets no runAsUser.

Check

docker inspect --format '{{.Config.User}}' <image>   # empty: root by default
kubectl get pod <pod> -o jsonpath='{.spec.securityContext.runAsUser} {.spec.containers[*].securityContext.runAsUser}'   # empty: the Pod does not choose a user

Fix

Add USER 10001 (numeric) to the Dockerfile and make the files the app writes owned by that user; or set runAsUser: 10001 in the securityContext if the image works as that user. Keep runAsNonRoot: true.

The image sets USER 0 or USER root, or the base image does and yours does not override it.

Check

docker inspect --format '{{.Config.User}}' <image>   # 0 or root

Fix

A numeric non-root USER in your Dockerfile, after the steps that need root.

The Dockerfile names the user instead of giving its number.

Check

docker inspect --format '{{.Config.User}}' <image>   # a name, such as app

Fix

USER 10001 in the Dockerfile (create the user with that UID), or runAsUser: 10001 in the securityContext.

A full OS base image ships hundreds of packages the app never uses, each with its own advisories.

Check

trivy image --severity HIGH,CRITICAL --pkg-types os <image>   # most findings are OS packages
docker images <image>   # a size near 1 GB points to a full base

Fix

Multi-stage build: build in a full image, ship from slim, alpine or distroless.

The base image is old: the fixes exist, but this image was built before them.

Check

trivy image --severity HIGH,CRITICAL --ignore-unfixed <image>   # what a rebuild on a current base would fix
docker inspect --format '{{.Created}}' <image>   # built weeks ago

Fix

Rebuild regularly on a current base, pinned by digest and updated by a bot.

The findings are in your app's packages, not the OS.

Check

trivy image --pkg-types library <image>   # Python, npm or Java packages

Fix

Upgrade those dependencies at their source; see "Critical vulnerability with a fixed version".

A floating base tag (python:latest, python:3) moved to a newer Python that the package has no wheels for yet and does not build on.

Check

grep FROM Dockerfile   # a tag without a fixed minor version
docker run --rm <base> python -V   # newer than the package supports

Fix

Pin the version you tested, ideally with its digest: FROM python:3.12-slim@sha256:..., or upgrade the package to a release with wheels for it.

The base changed to slim or alpine, which has no compiler or headers, so the source build cannot run.

Check

Scroll up in the log   # gcc: not found, or Python.h: No such file or directory

Fix

Install the build tools in a build stage only, build the wheels there, and copy them into the final image.

The base is alpine, which uses musl: manylinux wheels do not apply, so pip builds from source unless the package publishes musllinux wheels.

Check

grep FROM Dockerfile   # an alpine image
pip download --only-binary=:all: --platform musllinux_1_2_x86_64 --no-deps <package>   # fails: no musl wheel

Fix

Use a slim (Debian) base, or build the wheels in an alpine build stage with the tools installed.

The vulnerable code is in the package, but your product never runs it.

Check

trivy image --severity HIGH,CRITICAL <image>   # the package and advisory
Show that nothing in the product calls the vulnerable code   # the advisory's function is never reached

Fix

Write an OpenVEX statement: status not_affected, a justification such as vulnerable_code_not_in_execute_path, and an impact_statement. Pass it to the scanner: trivy image --vex vex.openvex.json <image>.

The product does reach it, so it is affected.

Check

Trace calls into the package   # the vulnerable function is used

Fix

Upgrade or remove the package. A not_affected statement here would be false.

The Terraform sets acl = "public-read" or a grant to AllUsers. It only takes effect if ACLs are enabled and the public access block is off, which the same code often changes to make it apply.

Check

checkov -d terraform   # CKV_AWS_20 on the bucket's ACL resource
aws s3api get-public-access-block --bucket <bucket>   # BlockPublicAcls or IgnorePublicAcls false
aws s3control get-public-access-block --account-id 111122223333   # the account-wide setting

Fix

Remove the ACL and set object ownership to BucketOwnerEnforced; set all four public access block settings to true. Share single objects with presigned URLs.

The bucket is also, or instead, public through a bucket policy with Principal "*". This check does not cover that, so passing CKV_AWS_20 is not proof it is private.

Check

aws s3api get-bucket-policy-status --bucket <bucket>   # IsPublic: true is what AWS actually evaluates

Fix

Narrow the policy to named principals or a CloudFront origin access control; BlockPublicPolicy then stops it coming back.

It is meant to be public, such as a static website.

Check

Read the bucket's purpose in the code or its tags   # website hosting, public assets

Fix

Serve it through CloudFront with origin access control and keep the bucket private; if it must be public, skip the check on that resource with the reason.

An ingress rule was opened to everyone for convenience, often while debugging.

Check

checkov -d terraform   # the resource with CKV_AWS_24
aws ec2 describe-security-groups --filters Name=ip-permission.from-port,Values=22 Name=ip-permission.cidr,Values=0.0.0.0/0 --query 'SecurityGroups[].GroupId'   # groups in the account open to everyone on 22

Fix

Remove the rule and use SSM Session Manager, or allow only the VPN range. Moving SSH to another port is not a fix.

The rule opens a port range that includes 22, or all ports, which the check also flags.

Check

grep -n -B2 -A6 from_port terraform/*.tf   # from_port 0 and to_port 65535, or protocol "-1"

Fix

Open only the ports the service needs, from the sources that need them.

encrypted (EBS) or storage_encrypted (RDS) is false or missing, and missing means false.

Check

checkov -d terraform   # the resource
aws ec2 get-ebs-encryption-by-default   # false: new volumes in this region are not encrypted unless asked

Fix

Set encrypted = true / storage_encrypted = true, with a KMS key if you need your own.

The resource already exists unencrypted. Changing the flag in Terraform forces a new resource, which for a database means a new, empty one if applied carelessly.

Check

terraform plan -lock=false   # must be replaced

Fix

Plan first. Snapshot, copy the snapshot with encryption, restore from it and cut over; do not apply a replacement on a live database.

It was a quick fix for an AccessDenied error: widening to * made the error go away.

Check

checkov -d terraform   # the policy
git log -p -S 'Action' -- terraform/   # the commit that widened it, and the error it was fixing

Fix

Name the actions and resources the code needs. IAM Access Analyzer can generate a policy from the role's CloudTrail activity.

It is meant as an admin role, but it is attached to something that is not a person (CI, an app).

Check

aws iam list-entities-for-policy --policy-arn <arn>   # roles and users that have it

Fix

Keep admin for break-glass human roles with MFA; give CI and apps their own narrow roles.

metadata_options sets http_tokens = "optional", or is missing and the AMI or account default allows v1.

Check

checkov -d terraform   # CKV_AWS_79 on the resource
aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,MetadataOptions.HttpTokens]'   # optional: v1 allowed

Fix

metadata_options { http_tokens = "required" } on instances and launch templates. Fix the SSRF itself too: allowlist the hosts the app may fetch.

Old software on the instance still uses v1 (an old SDK or agent), so it was left on.

Check

aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name MetadataNoToken --dimensions Name=InstanceId,Value=<id> --start-time <start> --end-time <end> --period 3600 --statistics Sum   # above 0: something still calls v1

Fix

Update that software, then require tokens.

A debug or tooling sidecar was left in the manifest with privileged: true.

Check

kube-linter lint k8s/   # privileged-container and the container's name
grep -rn 'privileged: true' k8s/

Fix

Remove the sidecar; use kubectl debug for live debugging.

A container mounts /var/run/docker.sock (or containerd's socket) to build or inspect images. Control of the runtime is control of the node.

Check

grep -rn -A3 hostPath k8s/   # docker.sock or containerd.sock

Fix

Build images in CI, or with a rootless builder (BuildKit rootless) that needs no socket.

A node agent (network plugin, storage driver, monitoring) really needs it.

Check

kubectl get ds -A   # a vendor DaemonSet

Fix

Run it in its own namespace with privileged allowed only there, and keep restricted everywhere else.

The Pod spec does not set the fields the restricted profile requires; a field left out counts as not set.

Check

kubectl get ns <namespace> --show-labels   # pod-security.kubernetes.io/enforce=restricted
kubectl label --dry-run=server --overwrite ns <namespace> pod-security.kubernetes.io/enforce=restricted   # warns for each running Pod that would fail

Fix

Fix the Pod, not the namespace: runAsNonRoot: true and a numeric runAsUser, allowPrivilegeEscalation: false, capabilities drop ALL, seccompProfile type RuntimeDefault. Do not lower the namespace label to make it pass.

It came through a Deployment: kubectl only warned (would violate PodSecurity), and the ReplicaSet then failed to create Pods, so the Deployment looks applied but has none.

Check

kubectl get events -n <namespace> --field-selector reason=FailedCreate   # the same violates PodSecurity message

Fix

The same fix in the Deployment's Pod template; the ReplicaSet then creates the Pods.

The image runs as root, so with the fields set it still fails at start (runAsNonRoot).

Check

docker inspect --format '{{.Config.User}}' <image>   # empty, 0 or root

Fix

Build the image with a numeric USER so it can run that way.

No NetworkPolicy exists for the namespace, so every Pod can reach every other Pod, the database included.

Check

kubectl get networkpolicy -n shop   # No resources found
kubectl exec -n other deploy/x -- nc -zv -w 3 postgres.shop 5432   # succeeded: reachable from another namespace

Fix

A default-deny-ingress policy (podSelector: {}, policyTypes: [Ingress]), then explicit allows for each flow: ingress controller, monitoring, app to database.

The policy existed and was deleted, or moved to a folder conftest does not read.

Check

git log --diff-filter=D --oneline -- k8s/networkpolicy.yaml   # the commit that deleted it

Fix

Restore it with git revert of that commit, or move it back to the folder conftest tests.

Policies exist but do nothing: the cluster's network plugin does not enforce them (flannel alone, for example). The API accepts them anyway.

Check

kubectl get networkpolicy -n shop   # a deny policy is listed
kubectl exec -n other deploy/x -- nc -zv -w 3 postgres.shop 5432   # still succeeds: not enforced

Fix

Use a network plugin that enforces NetworkPolicy (Calico, Cilium, or your cloud's option), and test a deny after setting it up.

A forbidden error (cannot get resource ...) was fixed with the biggest role instead of the one permission that was missing.

Check

kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> -n <ns>   # *.* with every verb
kubectl logs deploy/<app> --previous   # the original forbidden message names the verb and resource it needed

Fix

A Role with the exact verbs and resources (and resourceNames), and a RoleBinding in that namespace; delete the cluster-admin binding.

A Helm chart or operator asked for it at install.

Check

kubectl get clusterrolebinding -o wide | grep cluster-admin   # the binding and its subjects

Fix

Use the chart's RBAC values to scope it, or ask the vendor what it needs; do not accept cluster-admin for an app.

A Secret with real values was committed, usually to make a deploy work.

Check

git log --stat -- k8s/secret.yaml   # when it was added
git branch -r --contains <commit>   # pushed: treat the values as exposed
Do not decode the real values to check them.

Fix

Rotate every value first, then remove the file and purge it from history. Sync from a secret store (External Secrets) or commit it encrypted (Sealed Secrets, SOPS).

The values are placeholders or already encrypted (a SOPS file), and the policy only looks at kind.

Check

grep -n -E '^kind:|^sops:' k8s/secret.yaml   # a sops: block: encrypted

Fix

Keep it, and let the policy allow SOPS-encrypted files and SealedSecret objects.

The action is referenced by a tag or branch, as most examples show.

Check

zizmor .github/workflows/   # each unpinned uses: line
git ls-remote https://github.com/<owner>/<action> 'refs/tags/<tag>*'   # the SHA to pin to

Fix

uses: owner/action@<40-char sha> # vX.Y.Z. For an annotated tag, take the SHA on the ^{} line: that is the commit.

A docker:// action or container image is referenced by tag.

Check

grep -n 'docker://' .github/workflows/*.yml   # an image:tag without @sha256:

Fix

Pin the image by digest: docker://image@sha256:...

The SHA is a tag object, not a commit: annotated tags have their own SHA, which git ls-remote shows on the line without ^{}.

Check

git ls-remote https://github.com/actions/setup-python 'refs/tags/<tag>*'   # your SHA is on the line without ^{}: a tag object

Fix

Use the SHA on the ^{} line, which is the commit.

A typo, a shortened SHA, or a SHA copied from another repository.

Check

gh api repos/<owner>/<action>/commits/<sha> --jq .sha   # Not Found or No commit found: not in that repository

Fix

Copy the SHA again from git ls-remote of the right repository, or let Dependabot or Renovate write the pins.

The repository was renamed, deleted or made private.

Check

gh api repos/<owner>/<action> --jq .full_name   # another name, or Not Found

Fix

Point at the current repository, or move to a maintained action, pinned by SHA.

An event field is expanded directly in run:. Quoting does not help: the substitution happens before the shell parses anything.

Check

zizmor .github/workflows/   # the line and the expression

Fix

Pass it through the environment: env: TITLE: ${{ github.event.issue.title }}, then "$TITLE" in the script, quoted.

The same inside actions/github-script's script: input, which runs as JavaScript.

Check

grep -n -A6 github-script .github/workflows/*.yml   # ${{ github.event... }} inside script:

Fix

Read it from context.payload in the script, or pass it through env and process.env.

permissions: write-all at the top or on a job. zizmor reports this as error.

Check

zizmor .github/workflows/   # error[excessive-permissions], uses write-all permissions

Fix

permissions: contents: read at the top; add packages: write or id-token: write only on the job that needs it.

No permissions block, so the token gets the repository default, which may be write for everything. zizmor reports this as warning: default permissions used due to no permissions: block.

Check

gh api repos/<owner>/<repo>/actions/permissions/workflow   # default_workflow_permissions: write

Fix

Add permissions: contents: read to the workflow, and set the repository default to read-only.

The job's GITHUB_TOKEN lacks packages: write, because the workflow sets permissions without it.

Check

grep -n -A4 permissions .github/workflows/ci.yml   # no packages: write on the job that pushes
In the run log, Set up job > GITHUB_TOKEN Permissions   # Packages: read

Fix

Add permissions: packages: write to that job only.

The package already exists and this repository has no write access to it: it was first pushed from another repository or by a person.

Check

The package's settings > Manage Actions access   # this repository missing, or without the Write role

Fix

Add the repository with the Write role there, or connect the package to the repository. Not a personal access token.

The job's permissions lack the one the call needs: issues: write or pull-requests: write to comment, security-events: write to upload SARIF, contents: write to push.

Check

grep -n -A6 permissions .github/workflows/<workflow>.yml   # the permission is missing
In the run log, Set up job > GITHUB_TOKEN Permissions   # what the token actually had

Fix

Add the one permission the step needs to that job, for example permissions: pull-requests: write.

The run comes from a pull request from a fork (or from Dependabot): GitHub makes the token read-only whatever the workflow asks for.

Check

In the run log, Set up job > GITHUB_TOKEN Permissions   # all read although the workflow asks for write

Fix

Do not switch to pull_request_target with a checkout of the pull request's code. Split the work: a read-only job on pull_request, and a separate trusted workflow that comments without running fork code.

A job-level permissions block replaced the workflow-level one, so a permission set at the top is missing in that job.

Check

grep -n permissions .github/workflows/<workflow>.yml   # blocks both at the top and on the job

Fix

List every permission the job needs on the job itself.

The job has no permissions: id-token: write, so the runner gets no way to request a token.

Check

grep -n -B2 -A6 configure-aws-credentials .github/workflows/*.yml   # no id-token: write for that job
In the run log, Set up job > GITHUB_TOKEN Permissions   # no IdToken: write

Fix

Add id-token: write (with contents: read) to that job. Check the role's trust policy names this repository and branch.

id-token: write is set for the workflow, but the job has its own permissions block, which replaces the workflow's.

Check

grep -n -A4 permissions .github/workflows/<workflow>.yml   # a job-level block without id-token

Fix

Add id-token: write to the job's block.

The job is in a reusable workflow, and the calling job did not grant id-token: write; a called workflow cannot have more than its caller.

Check

grep -n -B4 -A4 'uses: ./.github/workflows' .github/workflows/*.yml   # the calling job's permissions

Fix

Grant id-token: write on the calling job too.

The trust policy's sub condition names another repository, branch or environment. A job with environment: prod sends repo:<owner>/<repo>:environment:prod, not a ref, and a pull request sends repo:<owner>/<repo>:pull_request.

Check

aws iam get-role --role-name <role> --query Role.AssumeRolePolicyDocument   # the sub and aud conditions
Compare token.actions.githubusercontent.com:sub with the job   # repository, ref, or the environment if the job has one

Fix

Make the condition match exactly: StringEquals token.actions.githubusercontent.com:sub = repo:<owner>/<repo>:ref:refs/heads/main (or :environment:<name>). Do not widen sub to repo:<owner>/* or * to make it pass.

The aud condition is not sts.amazonaws.com, or the action was given another audience.

Check

aws iam get-role --role-name <role> --query Role.AssumeRolePolicyDocument   # token.actions.githubusercontent.com:aud is something else

Fix

Set aud to sts.amazonaws.com and leave the action's audience at its default.

The account has no IAM OIDC provider for GitHub, or the trust policy's Federated ARN names another account. The message then says No OpenIDConnect provider found.

Check

aws iam list-open-id-connect-providers   # no token.actions.githubusercontent.com

Fix

Create the IAM OIDC provider for https://token.actions.githubusercontent.com with audience sts.amazonaws.com.

role-to-assume is the wrong ARN: another account, a typo, or a deleted role.

Check

grep -n role-to-assume .github/workflows/*.yml   # compare account ID and name with aws iam get-role

Fix

Fix the ARN; keep it in one variable rather than in many workflows.

A debug step printed the secret transformed (base64, reversed, split, URL-encoded). Masking only matches the exact value.

Check

gh run view <id> --log | grep -n -A1 base64   # the step and the encoded line; it is the secret, do not paste it

Fix

Rotate the secret first and update it where CI reads it. Then remove the step and delete the affected runs: gh run delete <id>.

A value derived from a secret (a token fetched at run time, a field of a JSON secret) was never registered for masking.

Check

grep -n add-mask .github/workflows/*.yml   # nothing before the value is used

Fix

Rotate it, register it with ::add-mask:: as soon as it is created, and never print it.

A pull_request_target (or workflow_run) workflow checks out the pull request's code and runs it. That trigger runs with secrets and a write token, so fork code gets them.

Check

zizmor .github/workflows/   # dangerous-triggers
grep -n -A12 pull_request_target .github/workflows/*.yml   # a checkout with ref: github.event.pull_request.head.sha or head.ref

Fix

Use on: pull_request for anything that runs pull request code; keep pull_request_target for metadata only (labels, comments), with no checkout of the pull request.

Script injection: an untrusted ${{ github.event... }} is expanded inside run:.

Check

zizmor .github/workflows/   # template-injection

Fix

Pass it through env: and use "$VAR"; see "zizmor: code injection via template expansion".

Shared CI runners share addresses, and every job downloads the database again.

Check

grep -rn -B2 -A6 'actions/cache' .github/workflows   # no cache for Trivy's cache directory

Fix

Cache the database between runs.

The scan pulls from ghcr.io only: an older Trivy, or --db-repository (TRIVY_DB_REPOSITORY) set to ghcr.io. Recent versions try mirror.gcr.io first and fall back to ghcr.io.

Check

trivy --version
grep -rn 'db-repository\|TRIVY_DB_REPOSITORY' .github/workflows   # pinned to ghcr.io

Fix

Use mirrors, in order: --db-repository mirror.gcr.io/aquasec/trivy-db:2,public.ecr.aws/aquasecurity/trivy-db:2, or upgrade Trivy.

A fresh runner has an empty cache, and the job skips the download.

Check

ls ~/.cache/trivy/db   # No such file or directory: no database

Fix

Restore the cache before the scan, or download from a mirror on the first run.

A cache is restored, but to another path than Trivy reads (--cache-dir or TRIVY_CACHE_DIR; ~/.cache/trivy by default).

Check

grep -rn 'cache-dir\|TRIVY_CACHE_DIR\|path:' .github/workflows   # the restored path and Trivy's path differ

Fix

Restore the cache to the path Trivy reads, or pass that path with --cache-dir.

The cache key missed, so nothing was restored.

Check

In the run log, the cache step   # Cache not found for input keys

Fix

Add restore-keys so yesterday's database is restored when today's is missing.

The job pulls without logging in, and shared CI runners share addresses, so other people's pulls count against the same limit.

Check

grep -rn 'FROM\|image:' Dockerfile .github/workflows   # images from Docker Hub
grep -rn docker/login-action .github/workflows   # no login in the job that pulls

Fix

Log in before pulling: docker/login-action with a Docker Hub access token kept as a secret (not the account password).

The job logs in, but after the pull or in another job, or the account's own limit is used up.

Check

The order of steps in the run log   # the login comes after the failing pull, or is missing in this job

Fix

Log in first in every job that pulls. For heavy use, pull from a mirror (mirror.gcr.io/library/<image> or public.ecr.aws/docker/library/<image>, pinned by digest) and cache the build so unchanged layers are not pulled again.

The CI secret is stale, usually right after the token was rotated, so pip sends a revoked token.

Check

gh secret list   # when each secret was last updated
aws secretsmanager describe-secret --secret-id <id> --query '[LastChangedDate,LastRotatedDate]'   # changed after the CI secret was set

Fix

Update the CI secret with the current value: gh secret set NAME, which reads it from stdin, so it is not on the command line.

The token is empty because this run cannot read the secret: a pull request from a fork, a Dependabot run (which gets Dependabot secrets, not Actions secrets), or an environment secret in a job without that environment.

Check

In the job: test -n "$PIP_TOKEN" && echo set || echo empty   # empty: the secret did not reach the job

Fix

Give that kind of run the secret (add it as a Dependabot secret, or set the job's environment), or do not run the step there.

The credentials are right but sent wrongly: the wrong user name for that index, or another index URL than the one holding the token.

Check

pip config debug   # the index-url in effect; it can hold a token, do not paste it

Fix

Use the user name the index documents for tokens, and set the URL in one place (PIP_INDEX_URL built from the secret).

The image was never signed: built on a laptop, or in a job without a signing step.

Check

cosign tree <image>   # no signatures listed

Fix

Sign in CI (keyless) and deploy what CI built.

The tag now points to a rebuild that was not signed, or one digest was signed and another verified (a multi-arch index against one platform's image).

Check

docker buildx imagetools inspect <image>   # the digest the tag resolves to now; compare with the digest CI signed (in the CI log)

Fix

Deploy and verify by digest, and sign the digest your deployments use.

The signature was stored in another repository (COSIGN_REPOSITORY), so cosign does not look for it there.

Check

grep -rn COSIGN_REPOSITORY .github/workflows   # set when signing

Fix

Set the same COSIGN_REPOSITORY when verifying, and give the verifier read access to it.

The trusted workflow signed it, but from another ref: a tag build signs as @refs/tags/v1.4.0, a main build as @refs/heads/main, and you asked for the other.

Check

Read got subjects   # the same workflow, a different ref

Fix

Name it exactly: --certificate-identity https://github.com/<owner>/<repo>/.github/workflows/release.yml@refs/tags/<tag>, or --certificate-identity-regexp anchored to that workflow for all tags, with --certificate-oidc-issuer https://token.actions.githubusercontent.com.

Another workflow or repository signed it, for example a test workflow or a fork.

Check

Read got subjects   # another workflow file or repository

Fix

Do not deploy it: rebuild and sign it in the release workflow. Never widen the identity to .* to make verification pass.

A reusable workflow did the signing, so the certificate names that workflow, not the one that called it.

Check

Read got subjects   # a workflow path in another repository, or a shared workflow

Fix

Trust the reusable workflow's identity, pinned to its ref.

The image has no signature from the identity the policy trusts: built outside CI, or a tag that moved to an unsigned rebuild.

Check

kubectl get clusterpolicy   # the verifyImages policy
cosign verify --certificate-identity <identity from the policy> --certificate-oidc-issuer https://token.actions.githubusercontent.com <image>   # fails here too: not signed by it

Fix

Deploy an image built and signed by CI; do not switch the policy to Audit.

Kyverno could not read the registry (a private registry without credentials for Kyverno). The message then says UNAUTHORIZED or failed to fetch, and the image may be signed.

Check

cosign verify ... <image>   # passes from your machine but not in the cluster: Kyverno's registry access

Fix

Give Kyverno pull credentials for that registry (imageRegistryCredentials in the policy); not a weaker policy.

No SBOM is generated in the pipeline.

Check

grep -rn 'syft\|sbom' .github/workflows   # no SBOM step

Fix

syft <image> -o cyclonedx-json=sbom.cdx.json in the release job, then attest it (below).

The SBOM was generated but not attached, or attached with another predicate type (SPDX) or to another digest.

Check

cosign tree <image>   # no attestations on this digest
cosign verify-attestation --type cyclonedx --certificate-identity <workflow-ref> --certificate-oidc-issuer https://token.actions.githubusercontent.com <image>   # none of the attestations matched the predicate type: another type

Fix

cosign attest --type cyclonedx --predicate sbom.cdx.json <image@digest>, in the job that pushes the image.

Branch protection was turned off or relaxed: no required reviews, admins exempt (enforce_admins false), or force pushes allowed.

Check

gh api repos/<owner>/<repo>/branches/main/protection   # Branch not protected, or the weak settings
git log --oneline origin/main   # commits that reached main without a pull request

Fix

Revert unreviewed changes with git revert, keeping the record. Restore the policy: gh api -X PUT repos/<owner>/<repo>/branches/main/protection --input branch-protection.json, with enforce_admins true.

main is protected by a ruleset, not classic protection. The classic API then says Branch not protected, and Scorecard may score 0 when its token cannot read the rules.

Check

gh api repos/<owner>/<repo>/rules/branches/main   # the ruleset rules that apply to main

Fix

If the rules require reviews and block force pushes, nothing needs restoring; run Scorecard with a token that can read them.

A tag was re-pushed (a rebuild, or someone pushing over it), and Pods deployed by tag pulled the new image when they restarted.

Check

kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'   # different digests for the same tag
cosign triangulate <image>   # where its signature lives; compare digests with the one CI signed

Fix

Deploy by digest or by an immutable per-commit tag. Keep Kyverno's verifyDigest and mutateDigest on (both default to true).

The registry allows tags to be overwritten.

Check

aws ecr describe-repositories --repository-names <repo> --query 'repositories[].imageTagMutability'   # MUTABLE

Fix

Turn on tag immutability where the registry supports it (ECR IMMUTABLE).

You are in your working clone, with local branches, changes or reflog history.

Check

git status   # changes or untracked files
git reflog | head   # more than the single clone entry

Fix

Work in a fresh mirror clone: git clone --mirror <url>, run filter-repo there, and keep the old clone as a backup.

It was a fresh clone, but something touched it afterwards (a fetch, a checkout of another branch).

Check

git reflog | head   # entries after the clone

Fix

Clone again, or add --force once you have a backup and are sure.

filter-repo ran in this clone and removed origin.

Check

git remote -v   # no origin listed

Fix

Check the rewrite, then git remote add origin <url> and git push --force --all (and --tags). Tell collaborators to re-clone. If you removed a secret, rotate it first: forks, pull request refs and old clones still hold the commits.

The remote has another name, or this repository never had one.

Check

git remote -v   # a remote under another name, or none, without a filter-repo run

Fix

Push to that name, or add the remote.

The app or proxy does not send the header at all.

Check

curl -sI https://<staging>/   # no Content-Security-Policy line

Fix

Set the headers in the app (after_request) or at the ingress, once, for every response.

The headers are set, but only on some responses: not on redirects, error pages or static files served by another layer.

Check

curl -sI https://<staging>/<a URL from the ZAP report>   # the header is missing on that URL only

Fix

Set them at the layer every response passes through, such as the ingress or proxy.

HSTS is only checked on HTTPS, and ZAP scanned plain HTTP or another host than you tested.

Check

Read the -t target in the ZAP command   # http://, or another host

Fix

Scan the HTTPS URL users reach.

Advisories are published after release; a build-time scan could not have known.

Check

grype sbom:./sbom.cdx.json   # the package and its fixed-in version
cosign verify-attestation --type cyclonedx ... <running image>   # the SBOM belongs to the image that is actually running

Fix

Upgrade, release through the pipeline, and confirm the running image's digest changed.

The SBOM is from an older image than the one running, so the finding may already be fixed.

Check

kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'   # compare with the SBOM's image digest

Fix

Scan the SBOM attached to the running digest.

The key was found and used, and the attacker created persistence: users, keys, roles, policies, instances, often in regions you do not watch.

Check

aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<key>   # what it did: management events, last 90 days, this region
aws cloudtrail lookup-events --region <other-region> --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<key>   # repeat for every region

Fix

Deactivate the key. Remove what it created, in every region. Delete the backdoor user: delete its keys, detach its policies, then delete-user.

The key was used to get temporary credentials (GetSessionToken, AssumeRole). Those stay valid until they expire, even after the key is deactivated.

Check

aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<key> | grep -E 'GetSessionToken|AssumeRole|GetFederationToken'   # sessions issued with the key

Fix

On that user, deny actions to sessions issued before now (an aws:TokenIssueTime condition), and revoke active sessions on any role it assumed.

Managed or inline policies are still on the user.

Check

aws iam list-attached-user-policies --user-name <user>   # managed policies
aws iam list-user-policies --user-name <user>   # inline policies

Fix

detach-user-policy for each managed policy, delete-user-policy for each inline one.

Access keys or group memberships remain (the message then says must delete access keys first, or must remove users from group first).

Check

aws iam list-access-keys --user-name <user>
aws iam list-groups-for-user --user-name <user>

Fix

delete-access-key, remove-user-from-group.

Other credentials remain: a console password, MFA devices, signing certificates, SSH keys, service-specific credentials.

Check

aws iam get-login-profile --user-name <user>   # NoSuchEntity: no console password
aws iam list-mfa-devices --user-name <user>

Fix

delete-login-profile, deactivate-mfa-device (and delete a virtual one), and so on, then delete-user.