DevOps Nimbus · Docker

Docker problems and fixes

66 problems people meet with Docker, each with the exact error, why it happens, how to confirm it, the fix and how to prevent it. Then learn it for good: break and fix each one yourself in the browser playground.

Docker itself

Cannot connect to the Docker daemon

The error

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

What it means

The docker command is only a client: it sends every request to the Docker daemon (dockerd) over a socket. unix:///var/run/docker.sock is the address it tried, and nothing answered there. Either dockerd is not running, or the client is looking in the wrong place.

Causes, most likely first

  1. dockerd failed at start, most often because /etc/docker/daemon.json is not valid JSON or sets an option that is also given as a dockerd flag. The service exits at once, so the socket has no listener.

    Check

    systemctl status docker docker.socket   # failed or inactive: the daemon is down
    sudo journalctl -u docker --since '10 min ago'   # the reason, such as 'invalid character' in daemon.json
    sudo dockerd --validate --config-file /etc/docker/daemon.json   # Engine 23.0+: 'configuration OK', or the exact problem

    Fix

    Fix what the journal names (for example the JSON in /etc/docker/daemon.json), then sudo systemctl start docker.

  2. The service was stopped, or never enabled after install. Nothing is wrong with it, it is just not running.

    Check

    systemctl is-enabled docker; systemctl is-active docker   # disabled / inactive with no error in the journal

    Fix

    sudo systemctl enable --now docker

  3. DOCKER_HOST or the current context points at another socket or host (a rootless daemon, a remote host, Docker Desktop's context). The client asks there, and that daemon is not answering.

    Check

    docker context show; echo "DOCKER_HOST=${DOCKER_HOST:-unset}"   # anything but default and unset: the client is not using /var/run/docker.sock

    Fix

    docker context use default, or unset DOCKER_HOST. For rootless Docker, start it with systemctl --user start docker and set DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock.

  4. On macOS or Windows, the daemon runs inside Docker Desktop's VM. If Docker Desktop is not running, or its VM is still starting, there is no daemon to answer.

    Check

    docker context show   # desktop-linux or default on a Mac or PC
    docker version   # a Client section but no Server section: no daemon answered

    Fix

    Start Docker Desktop and wait until it says the engine is running.

Prevent it

Validate daemon.json before restarting (dockerd --validate on Engine 23.0+, or python3 -m json.tool) and change it through configuration management; alert when the docker unit fails.

Now learn it: break and fix it yourself in Cannot connect to the daemon, then take Level 16 · Troubleshooting and cleanup. Open in DevOps Nimbus Docker.

permission denied on /var/run/docker.sock

The error

permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

What it means

A daemon is running, but your user may not open its socket, /var/run/docker.sock. The socket belongs to root and the docker group, with permissions srw-rw----, so only those two may connect.

Causes, most likely first

  1. You are not in the docker group, so the socket's group permission does not apply to you.

    Check

    ls -l /var/run/docker.sock   # owner root, group docker, srw-rw----
    getent group docker   # your user name is missing from the list

    Fix

    sudo usermod -aG docker $USER, then log out and back in. The docker group is root-equivalent: grant it deliberately, or use rootless Docker.

  2. You were added to the group, but your shell started before that. Groups are read at login, so this shell still runs without it.

    Check

    id -nG   # docker missing here while getent group docker lists you: the session is older than the change

    Fix

    newgrp docker for this shell, or log out and back in.

  3. The socket's group is not docker (a custom group, or a package that sets it differently), so being in docker does not help.

    Check

    stat -c '%G' /var/run/docker.sock   # a group other than docker

    Fix

    Join the group the socket really uses, or set it in the daemon's config ("group" in daemon.json). Never chmod 666 the socket: that gives every local user root.

Prevent it

Remember that the docker group is root-equivalent: grant it deliberately, or use rootless Docker. Never chmod 666 the socket.

Now learn it: break and fix it yourself in Permission denied on the socket, then take Level 00 · Your first container. Open in DevOps Nimbus Docker.

docker-compose: command not found

The error

docker-compose: command not found

What it means

The shell found no program called docker-compose. That is the name of the old Python Compose v1, which is end of life. Compose v2 is normally installed as a docker CLI plugin and run as docker compose, with a space.

Causes, most likely first

  1. Scripts and docs written for Compose v1, on a host that only has the v2 plugin. The plugin answers to docker compose, not docker-compose.

    Check

    docker compose version   # prints a v2 version: Compose is installed, only the old name is missing

    Fix

    Use docker compose (with a space): docker compose up -d. For a script you cannot change yet, a docker-compose wrapper that runs exec docker compose "$@".

  2. Compose is not installed at all. On Linux, the distribution's docker package often comes without the Compose plugin.

    Check

    docker compose version   # docker: 'compose' is not a docker command
    docker info --format '{{range .ClientInfo.Plugins}}{{.Name}} {{end}}'   # compose is not listed

    Fix

    Install the plugin: sudo apt-get install docker-compose-plugin from Docker's repository, or your distribution's Compose v2 package.

Prevent it

Update scripts and CI to docker compose. Compose v1 stopped receiving updates in July 2023, and v2 differs in small ways (container names use - instead of _).

Now learn it: break and fix it yourself in Compose: wrong host, then take Level 10 · Compose. Open in DevOps Nimbus Docker.

fork: Resource temporarily unavailable

The error

fork() failed while spawning "worker process" (11: Resource temporarily unavailable)

What it means

The program asked the kernel for a new process or thread (fork), and the kernel said no. 11 is EAGAIN: a limit on processes is reached, so try again later. The limit can be the container's, your user's, or the whole host's.

Causes, most likely first

  1. A container leaks processes or threads (or runs a fork bomb) with no --pids-limit, until the host runs out of process IDs. Every program on the host then fails the same way.

    Check

    docker stats --no-stream   # PIDS column: one container far above the rest is the leak
    ps -eLf | wc -l; cat /proc/sys/kernel/pid_max   # the host's count near pid_max

    Fix

    docker rm -f <leaking-container>, re-create it with --pids-limit 200, and fix the leak in the app.

  2. The container's own cgroup pids limit is too low for what the app really needs, so it hits the limit while the host is fine.

    Check

    docker exec <name> cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max   # cgroup v2: current at max means this limit

    Fix

    docker update --pids-limit <higher> <name>, and set the same pids_limit in the compose file.

  3. A per-user process limit (ulimit -u, nproc). It counts every process of that UID, across all containers that run as the same UID.

    Check

    docker exec <name> sh -c 'ulimit -u'   # a small number here, with pids.max not reached: nproc

    Fix

    Raise it for that container (--ulimit nproc=...), or run unrelated containers as different UIDs.

Prevent it

Set --pids-limit (pids_limit in Compose) on every container, so one leak cannot take the host down.

Now learn it: break and fix it yourself in Out of processes, then take Level 12 · Limits and OOM. Open in DevOps Nimbus Docker.

Slow or timing out at a CPU limit

The error

curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received

What it means

The request did not get an answer in time (curl's error 28 is a timeout). Nothing crashed. A common reason in containers is a CPU limit: the kernel lets the container use only its share of CPU time in each period and pauses it for the rest, so everything is slow.

Causes, most likely first

  1. --cpus (or a Compose cpus limit) is set below the real load, or a traffic peak goes beyond what the limit was sized for. The kernel throttles the container whenever it uses up its quota.

    Check

    docker stats --no-stream   # CPU % pinned at the limit (100% per CPU allowed)
    docker inspect <name> --format '{{.HostConfig.NanoCpus}} {{.HostConfig.CpuQuota}} {{.HostConfig.CpuPeriod}}'   # all 0: no CPU limit, so not this cause
    docker exec <name> cat /sys/fs/cgroup/cpu.stat   # cgroup v2: nr_throttled rising between two reads confirms it

    Fix

    docker update --cpus 1 <name> (takes effect at once), then set the same value in the compose file so it survives a re-create.

  2. There is no limit, but the host itself is out of CPU (other containers, a noisy neighbour). The app is just as slow, but nothing throttles it.

    Check

    uptime; top -bn1 | head -15   # load above the CPU count, with nr_throttled not rising: the host is short of CPU

    Fix

    Move or limit the other workloads, or add capacity. Raising this container's limit will not help.

Prevent it

Size CPU limits from measured load with headroom, and alert on throttling, not only on errors.

Now learn it: break and fix it yourself in Throttled CPU, then take Level 12 · Limits and OOM. Open in DevOps Nimbus Docker.

Container will not stay up

Exited (0) right after it starts

The error

Exited (0) 2 seconds ago

What it means

The container's main process (PID 1) finished with exit code 0, which means success. A container lives exactly as long as PID 1, so it stopped with it. Nothing failed; the program simply did not keep running.

Causes, most likely first

  1. The server was started as a daemon (nginx without daemon off, service x start). It forks into the background and the foreground process returns, so PID 1 ends.

    Check

    docker inspect <name> --format '{{.Path}} {{.Args}}'   # service ... start, or nginx without -g 'daemon off;'
    docker logs <name>   # usually empty or a one-line start message

    Fix

    Run the program in the foreground (nginx -g 'daemon off;', or the image's default CMD). docker rm <name> and run it again with that command.

  2. A shell with no terminal attached: docker run -d alpine runs sh, which reads an empty input and exits at once.

    Check

    docker inspect <name> --format '{{.Path}} {{.Args}} tty={{.Config.Tty}} stdin={{.Config.OpenStdin}}'   # sh with tty=false stdin=false

    Fix

    Give it the command you really want to run. For an interactive shell, docker run -it alpine sh.

  3. The command ends with &, so the shell starts the program in the background and exits, taking the container with it.

    Check

    docker inspect <name> --format '{{.Args}}'   # a command ending in &

    Fix

    Remove the & and run the program in the foreground; never background the main process.

Prevent it

Use the image's default command when you can; never background the main process.

Now learn it: break and fix it yourself in Exits immediately, then take Level 02 · Why did it exit?. Open in DevOps Nimbus Docker.

Exited (1) or another non-zero code

The error

Exited (1) 3 seconds ago

What it means

The main process started and then exited with code 1, a general failure. Docker did its part; the app gave up. The reason is almost always in its last log lines.

Causes, most likely first

  1. A dependency it cannot reach: a name that does not resolve, or a database that is not ready yet. Many apps exit on the first failed connection.

    Check

    docker logs --tail 50 <name>   # ENOTFOUND, ECONNREFUSED, could not translate host name

    Fix

    Fix the address or network (see the enotfound and econnrefused-localhost entries), or wait for the dependency with a healthcheck.

  2. Missing files: a mount over the app, a wrong WORKDIR, or a missing module. The interpreter starts but cannot load the code.

    Check

    docker logs --tail 50 <name>   # Cannot find module, No such file
    docker inspect <name> --format '{{json .Mounts}}'   # a mount over the app's folder

    Fix

    See the module-hidden entry; fix the mount or WORKDIR and re-create the container.

  3. No permission to write where it needs to, so the app fails while opening a file.

    Check

    docker logs --tail 50 <name>   # EACCES or Permission denied with a path

    Fix

    See the eacces-mount entry.

  4. Missing configuration, such as a required environment variable, makes the app stop on purpose.

    Check

    docker logs --tail 50 <name>   # a message naming the setting
    docker exec <name> printenv <VAR> >/dev/null && echo set || echo missing   # needs the container running; prints no value

    Fix

    Provide the setting (-e, --env-file, or environment: in Compose) and re-create the container.

Prevent it

Make apps fail with a clear message on bad configuration, and check required variables at start.

Now learn it: break and fix it yourself in Name not found · Mount hides the app · Permission denied · Restarting forever, then take Level 02 · Why did it exit?. Open in DevOps Nimbus Docker.

Exited (137), OOMKilled true

The error

Exited (137)

What it means

137 is 128 + 9: the main process was killed with signal 9, SIGKILL, which cannot be caught. OOMKilled true means the kernel did it because the container went over its memory limit. With OOMKilled false, someone or something else sent the kill.

Causes, most likely first

  1. The --memory limit is below what the app really uses. When the container's memory reaches it, the kernel's OOM killer kills a process in it.

    Check

    docker inspect <name> --format '{{.State.OOMKilled}} {{.HostConfig.Memory}}'   # true and a non-zero limit
    sudo dmesg -T | grep -i 'memory cgroup out of memory'   # a line naming the process

    Fix

    docker update --memory 512m --memory-swap 512m <name>, then docker start <name>, and set the same in the compose file.

  2. A memory leak grows until it hits the limit. It looks like the cause above, but at any limit it only takes longer.

    Check

    docker stats <name>   # MEM USAGE climbing steadily under constant load
    docker inspect <name> --format '{{.RestartCount}}'   # repeated kills at regular intervals

    Fix

    Find the leak (a heap profile). Raising the limit only delays it.

  3. There is no container limit, but the host (or Docker Desktop's VM) ran out of memory, and the kernel chose this process to kill.

    Check

    sudo dmesg -T | grep -i -E 'out of memory|killed process'   # 'Out of memory' without 'memory cgroup': the whole host

    Fix

    Give the host or VM more memory, or set limits on the other containers so one cannot starve the rest.

  4. OOMKilled is false: docker kill, or docker stop after its 10-second grace period, sent the SIGKILL.

    Check

    docker events --since 1h --filter container=<name>   # kill or stop events, and when

    Fix

    Find who or what stopped it; for slow stops see the stop-slow entry.

Prevent it

Size limits from measured use under load (docker stats), with headroom, and alert on restarts.

Now learn it: break and fix it yourself in OOMKilled, then take Level 12 · Limits and OOM. Open in DevOps Nimbus Docker.

docker stop takes 10 seconds, then Exited (137)

The error

Exited (137) 1 second ago   (after docker stop)

What it means

docker stop sends SIGTERM, waits 10 seconds by default, then sends SIGKILL. Taking the full 10 seconds and ending in 137 (128 + 9) means the process never acted on SIGTERM and was killed, cutting off in-flight work.

Causes, most likely first

  1. The app runs as PID 1 and has no SIGTERM handler. The kernel ignores signals to PID 1 that it has no handler for, so SIGTERM does nothing.

    Check

    time docker stop <name>   # about 10 s, then 137
    docker exec <name> ps -o pid,comm   # the app itself is PID 1 (needs ps in the image)

    Fix

    docker run --init ... (tini becomes PID 1 and forwards SIGTERM), or handle SIGTERM in the app and close cleanly.

  2. Shell-form CMD (CMD node server.js) makes /bin/sh -c the PID 1, and the shell may not pass the signal on to the app.

    Check

    docker inspect <name> --format '{{.Path}} {{.Args}}'   # /bin/sh -c ... means shell form

    Fix

    Use exec form: CMD ["node", "server.js"].

  3. A wrapper script starts the app without exec, so the script stays PID 1 and the app never gets the signal.

    Check

    docker exec <name> ps -o pid,comm   # PID 1 is sh or bash, the app has another PID

    Fix

    End the script with exec <app> so the app replaces the shell.

  4. The app stops on another signal (nginx stops gracefully on SIGQUIT) and the image sets no STOPSIGNAL.

    Check

    docker inspect <name> --format '{{.Config.StopSignal}}'   # empty, while the app's docs name another signal

    Fix

    STOPSIGNAL in the Dockerfile, or stop_signal in Compose.

  5. The app does handle SIGTERM, but draining takes longer than the grace period.

    Check

    docker logs --tail 20 <name>   # a shutdown message that starts but does not finish

    Fix

    Raise the grace period: docker stop -t 30, or stop_grace_period in Compose.

Prevent it

Test shutdown in CI: docker stop must finish well inside its timeout with exit code 0 or 143.

Now learn it: break and fix it yourself in docker stop takes 10 seconds, then take Level 11 · Healthchecks and restarts. Open in DevOps Nimbus Docker.

exec format error

The error

exec /usr/local/bin/docker-entrypoint.sh: exec format error

What it means

The kernel was asked to run the file as a program and did not recognise its format. Usually the image's binaries are for another CPU architecture than this host (arm64 on amd64, or the reverse). It can also be a script with no #! line. The container exits at once, usually with 255.

Causes, most likely first

  1. An image built on an Arm laptop (Apple silicon) runs on an amd64 server, or the other way round. The binaries are valid, just for another CPU.

    Check

    docker image inspect <image> --format '{{.Os}}/{{.Architecture}}'   # arm64 on an amd64 host, or the reverse
    docker info --format '{{.OSType}}/{{.Architecture}}'   # the host: x86_64 is amd64, aarch64 is arm64

    Fix

    docker build --platform linux/amd64 -t <image> . for the host you run on.

  2. A single-platform image was pushed under a tag others expect to be multi-platform, so each machine pulls the one platform there is.

    Check

    docker buildx imagetools inspect <image>   # only one Platform listed

    Fix

    Publish both: docker buildx build --platform linux/amd64,linux/arm64 --push .

  3. The entrypoint is a script without a #! first line (or with a byte-order mark before it), started in exec form. Without #! the kernel does not know which interpreter to use.

    Check

    docker run --rm --entrypoint sh <image> -c 'head -c 4 /usr/local/bin/docker-entrypoint.sh | od -c'   # must start with # !; 357 273 277 is a byte-order mark

    Fix

    Add #!/bin/sh as the first line, save without a byte-order mark, and rebuild.

Prevent it

Build release images in CI for every platform you run on.

Now learn it: break and fix it yourself in exec format error, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

executable file not found in $PATH

The error

exec: "nodejs": executable file not found in $PATH: unknown

What it means

The container runtime (runc) tried to start the program named in the command, "nodejs" here, and found no such file in any directory on PATH. The container never really ran, and Docker reports exit code 127, the shell's code for command not found.

Causes, most likely first

  1. A typo in ENTRYPOINT or the command (nodejs instead of node). The name runc looks for does not exist.

    Check

    docker inspect <name> --format '{{json .Path}} {{json .Args}}'   # the exact name it tried

    Fix

    Fix the name in the Dockerfile and rebuild, or override it: docker run --entrypoint node <image> server.js

  2. The tool is not in the image at all (curl or bash in an alpine image, anything in distroless).

    Check

    docker run --rm --entrypoint sh <image> -c 'command -v node'   # nothing printed: not in the image (or not on PATH)

    Fix

    Install it in the image, or use a tool the image has.

  3. Exec form with the whole command in one string: CMD ["npm start"] looks for a single program called 'npm start', space included.

    Check

    docker inspect <name> --format '{{json .Path}}'   # one string with a space in it

    Fix

    Split it into separate items: CMD ["npm", "start"].

  4. ENV PATH was replaced instead of extended, so the directory with the program is no longer searched.

    Check

    docker image inspect <image> --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^PATH='   # /usr/local/bin or /usr/bin missing

    Fix

    Extend it: ENV PATH="/app/bin:$PATH".

Prevent it

Run the image once in CI (a smoke test) before it is published.

Now learn it: break and fix it yourself in Executable not found, then take Level 02 · Why did it exit?. Open in DevOps Nimbus Docker.

exec: permission denied (entrypoint script)

The error

exec: "/app/start.sh": permission denied: unknown

What it means

runc found the entrypoint file, /app/start.sh, but the kernel refused to execute it. Docker reports exit code 126, the shell's code for found but not executable. Usually the file has no executable bit, and then nobody, not even root, may run it.

Causes, most likely first

  1. The script has no executable bit: it was committed without +x, or created on Windows. COPY keeps the file's mode from the build context, so the image has it without x too.

    Check

    docker run --rm --entrypoint ls <image> -l /app/start.sh   # -rw-r--r--, with no x
    git ls-files -s start.sh   # 100644 in git: the bit was never committed

    Fix

    chmod +x start.sh and git update-index --chmod=+x start.sh, then rebuild; or COPY --chmod=755 start.sh /app/ (needs BuildKit, the default builder since Engine 23.0).

  2. The file is on a mount that forbids execution: a noexec tmpfs or volume, or a bind mount that replaced the image's copy with a host file without x.

    Check

    docker inspect <name> --format '{{json .Mounts}} {{json .HostConfig.Tmpfs}}'   # a mount over /app, or noexec on it

    Fix

    Keep executables out of that mount, mount it without noexec, or chmod +x the host file.

  3. The path is a directory, not a file, so there is nothing to execute.

    Check

    docker run --rm --entrypoint ls <image> -ld /app/start.sh   # a leading d

    Fix

    Point ENTRYPOINT at the script inside that folder.

Prevent it

Store the bit in git (git update-index --chmod=+x start.sh) and set it in the Dockerfile.

Now learn it: break and fix it yourself in Entrypoint permission denied, then take Level 03 · Your first Dockerfile. Open in DevOps Nimbus Docker.

node: bad option (arguments replaced CMD)

The error

node: bad option: --verbose

What it means

node received --verbose as its only argument and rejected it. Anything after the image name in docker run replaces the image's CMD entirely, so node started without server.js and with just your flag.

Cause

  1. docker run <image> --flag, expecting the flag to be appended to the default command. It replaces CMD instead, and only ENTRYPOINT (here node) stays.

    Check

    docker inspect <image> --format '{{.Config.Entrypoint}} {{.Config.Cmd}}'   # ENTRYPOINT node, CMD server.js
    docker inspect <name> --format '{{.Path}} {{.Args}}'   # node --verbose, with server.js gone

    Fix

    Give the whole command: docker run <image> node server.js --verbose, or use the app's environment setting instead.

Prevent it

Put the fixed program in ENTRYPOINT and only defaults in CMD, so arguments add to it.

Now learn it: break and fix it yourself in Arguments replace CMD, then take Level 02 · Why did it exit?. Open in DevOps Nimbus Docker.

Restarting (1) over and over

The error

Restarting (1) 5 seconds ago

What it means

The container has a restart policy (always, unless-stopped or on-failure), and its process keeps exiting, here with code 1, so Docker keeps starting it again. The number in brackets is the last exit code. The policy hides the crash; it does not fix it.

Causes, most likely first

  1. Missing configuration, or a dependency that is down, makes the app exit on every start.

    Check

    docker logs --tail 30 <name>   # the same error at the end of each run
    docker inspect <name> --format '{{.RestartCount}} {{.HostConfig.RestartPolicy.Name}}'   # a growing count

    Fix

    Fix the cause from the logs, then re-create the container with the right settings.

  2. A crash on start: a bug, a bad file, or a resource limit (exit 137 means it was killed, see oom-137).

    Check

    docker inspect <name> --format '{{.State.ExitCode}} {{.State.OOMKilled}}'   # 137 and true: memory

    Fix

    Fix the crash; to debug, stop the loop with docker update --restart no <name> and read the logs.

Prevent it

Alert on restart counts, not only on containers that are down.

Now learn it: break and fix it yourself in Restarting forever, then take Level 11 · Healthchecks and restarts. Open in DevOps Nimbus Docker.

unable to find user

The error

unable to find user app: no matching entries in passwd file

What it means

Docker tried to run the container as the user app, looked it up in the image's /etc/passwd, and found no such line. USER in the image or --user names an account that does not exist inside the image.

Causes, most likely first

  1. The user is created in another build stage. Only the final stage's files reach the image, so its /etc/passwd lacks the user.

    Check

    docker run --rm --entrypoint grep <image> app /etc/passwd   # nothing: the name is not in the image
    grep -n -E 'FROM|useradd|adduser|USER' Dockerfile   # the user created before the last FROM

    Fix

    Create the user in the final stage, or use a numeric UID:GID, which needs no passwd entry.

  2. --user (or user: in Compose) overrides the image's USER with a name the image does not have.

    Check

    docker inspect <name> --format '{{.Config.User}}'; docker image inspect <image> --format '{{.Config.User}}'   # they differ: the override

    Fix

    Use a name the image has, or a numeric UID such as --user 10001:10001.

  3. A mount over /etc (or over /etc/passwd) hides the image's passwd file.

    Check

    docker inspect <name> --format '{{json .Mounts}}'   # a destination of /etc or /etc/passwd

    Fix

    Remove that mount, or mount only the file you need.

Prevent it

Prefer numeric USER 10001:10001 in images; it also makes file ownership predictable.

Now learn it: break and fix it yourself in Permission denied, then take Level 09 · Users and permissions. Open in DevOps Nimbus Docker.

exec ...: no such file or directory (entrypoint)

The error

exec /docker-entrypoint.sh: no such file or directory

What it means

The kernel could not find something it needs to start the program. That is the file itself, the interpreter its #! first line names, or, for a binary, its loader. The message names the entrypoint even when the missing piece is the interpreter.

Causes, most likely first

  1. The file is not at that path: a wrong path in ENTRYPOINT, or a mount over its folder hides it.

    Check

    docker run --rm --entrypoint ls <image> -l /docker-entrypoint.sh   # No such file: missing from the image
    docker inspect <name> --format '{{json .Mounts}}'   # a mount over its folder

    Fix

    Fix the path in ENTRYPOINT, or stop mounting over it.

  2. Windows line endings: the first line reads #!/bin/sh\r, and there is no program called /bin/sh\r.

    Check

    docker run --rm --entrypoint sh <image> -c 'head -1 /docker-entrypoint.sh | od -c'   # \r before \n means Windows endings

    Fix

    sed -i 's/\r$//' docker-entrypoint.sh, then rebuild.

  3. The script starts with #!/bin/bash in an image without bash (alpine has only /bin/sh).

    Check

    docker run --rm --entrypoint sh <image> -c 'ls -l /bin/sh /bin/bash'   # /bin/bash missing

    Fix

    Start the script with #!/bin/sh, or install bash in the image.

  4. A binary built against glibc is copied into an alpine (musl) or scratch image, where its loader (/lib64/ld-linux-x86-64.so.2) does not exist.

    Check

    file <binary>   # on the build machine: dynamically linked, interpreter /lib64/ld-linux...

    Fix

    Build it static (for Go, CGO_ENABLED=0), or use a base image with glibc.

Prevent it

Add *.sh text eol=lf to .gitattributes, so scripts keep Unix line endings on every checkout.

Now learn it: break and fix it yourself in Executable not found, then take Level 03 · Your first Dockerfile. Open in DevOps Nimbus Docker.

Cannot create or start it

Conflict. The container name is already in use

The error

Conflict. The container name "/web" is already in use by container

What it means

Docker refused to create a container called web because a container with that name already exists. Names are unique on a daemon, and a stopped container still holds its name until it is removed. The rest of the message gives the ID of the container that has it.

Causes, most likely first

  1. The old container exited but was never removed, so its name is still taken.

    Check

    docker ps -a --filter name=web   # a line with Exited (...): the old one

    Fix

    docker rm web (or docker rename web web-old to keep it), then run the new one.

  2. A script runs docker run again without --rm or a cleanup step, so every second run collides with the first.

    Check

    docker ps -a --filter name=web --format '{{.Status}} {{.CreatedAt}}'   # created by the previous run of the script

    Fix

    Remove or rename the old container in the script before docker run, or let Compose replace it.

Prevent it

Deploy scripts remove or rename the old container first; Compose does this for you.

Now learn it: break and fix it yourself in Name already in use, then take Level 00 · Your first container. Open in DevOps Nimbus Docker.

port is already allocated

The error

Bind for 0.0.0.0:8080 failed: port is already allocated

What it means

Docker tried to publish host port 8080 and something already holds it. "port is already allocated" means Docker itself has given that port to another container. "bind: address already in use" means another program on the host has it.

Causes, most likely first

  1. An older container still publishes the port, perhaps from another Compose project. Docker will not give the same host port to two containers.

    Check

    docker ps --filter publish=8080   # a container listed: that one holds it

    Fix

    docker stop <old-container>, then docker start <name>.

  2. A local service on the host (another web server) listens on that port, so Docker's proxy cannot bind it.

    Check

    sudo ss -ltnp 'sport = :8080'   # a process that is not docker-proxy

    Fix

    Stop that service, or publish another host port: -p 8081:80.

  3. A docker-proxy left behind after a daemon crash holds the port with no container behind it.

    Check

    sudo ss -ltnp 'sport = :8080'   # docker-proxy, while docker ps --filter publish=8080 lists nothing

    Fix

    sudo systemctl restart docker

Prevent it

Replace containers in one step (stop old, start new), or put a reverse proxy in front.

Now learn it: break and fix it yourself in Port already allocated, then take Level 06 · Ports. Open in DevOps Nimbus Docker.

container is not running (docker exec)

The error

Error response from daemon: container 3f2a... is not running

What it means

docker exec starts an extra process inside a running container. This container is stopped (or between restarts), so there is no running container to join.

Causes, most likely first

  1. The container exited. Its main process ended, so the container stopped with it.

    Check

    docker ps -a --filter name=<name>   # Exited (n)
    docker logs <name>   # why it stopped

    Fix

    Read docker logs and fix the cause. To look around the image, docker run --rm -it --entrypoint sh <image>.

  2. It is in a restart loop and caught between crashes (the message says is restarting).

    Check

    docker inspect <name> --format '{{.State.Status}} {{.RestartCount}}'   # restarting, and a growing count

    Fix

    See the restarting entry.

Prevent it

Debug from logs first; exec is for running containers.

Now learn it: break and fix it yourself in Mount hides the app, then take Level 02 · Why did it exit?. Open in DevOps Nimbus Docker.

Minimum memory limit allowed is 6MB

The error

Minimum memory limit allowed is 6MB.

What it means

Docker refuses memory limits too small for any process to run. 6MB is the lowest limit it accepts, and the value it got was below that.

Cause

  1. A unit typo: a number without a unit is bytes, so --memory 512 means 512 bytes, not megabytes.

    Check

    history | grep -- '--memory'   # a number with no b, k, m or g after it

    Fix

    Give a unit: --memory 512m.

Prevent it

Always write units on memory flags.

Now learn it: break and fix it yourself in OOMKilled, then take Level 12 · Limits and OOM. Open in DevOps Nimbus Docker.

Memory limit should be smaller than already set memoryswap limit

The error

Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time

What it means

docker update was asked to raise the memory limit above the container's existing memory-plus-swap limit (--memory-swap). The memory limit can never be larger than memory plus swap, so Docker refuses.

Cause

  1. --memory was raised without --memory-swap. When the container was created with only --memory, the swap limit defaulted to twice that old memory limit, and the new value is above it.

    Check

    docker inspect <name> --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'   # MemorySwap smaller than the new --memory

    Fix

    Set both together: docker update --memory 512m --memory-swap 512m <name>.

Prevent it

Set both flags together whenever you change a limit.

Now learn it: break and fix it yourself in OOMKilled, then take Level 12 · Limits and OOM. Open in DevOps Nimbus Docker.

No such container

The error

Error response from daemon: No such container: web

What it means

The daemon you are talking to has no container with that name or ID. Either it was never there, it has been removed, or you are asking a different daemon.

Causes, most likely first

  1. It ran with --rm and was removed when it exited.

    Check

    docker ps -a --format '{{.Names}}\t{{.Status}}'   # the name is not listed at all

    Fix

    Start it again without --rm if you need it to stay after it exits.

  2. A typo, or the ID of a container that was already removed.

    Check

    docker ps -a --format '{{.Names}}\t{{.Status}}'   # a similar name

    Fix

    Use the exact name from docker ps -a.

  3. Compose v2 names containers <project>-<service>-1 (v1 used <project>_<service>_1), not just <service>.

    Check

    docker compose ps   # the real container names of this project

    Fix

    Use docker compose exec <service> ... inside the project, or the full container name.

  4. Another Docker context or host is active, so the container exists on a different daemon.

    Check

    docker context ls   # the context marked * is the one in use

    Fix

    docker context use <the right one>, or unset DOCKER_HOST.

Prevent it

Give containers a fixed --name in scripts, and use docker compose commands with service names inside a Compose project.

Now learn it: break and fix it yourself in Name already in use, then take Level 00 · Your first container. Open in DevOps Nimbus Docker.

invalid reference format

The error

docker: invalid reference format: repository name must be lowercase.

What it means

Docker read a word on the command line as an image name, and that word is not a valid name[:tag]. Here it contains capital letters, which image names may not. The word it read is often not the one you meant as the image.

Causes, most likely first

  1. Capital letters in the image name: image names must be lowercase (myapp, not MyApp).

    Check

    set -x   # then run the script again: the image name exactly as the shell expanded it

    Fix

    Use lowercase image names.

  2. An empty variable left a trailing colon: shop:$TAG with TAG unset gives shop:, which is not valid.

    Check

    echo "TAG=${TAG:-unset}"   # unset or empty

    Fix

    Give variables a default: ${TAG:-latest}, or fail when unset: ${TAG:?}.

  3. A space in a path, as in -v $(pwd):/app in a folder with a space, so the part after the space is read as the image.

    Check

    pwd   # a space in the path

    Fix

    Quote paths: -v "$(pwd)":/app.

  4. A dash copied from a web page (– instead of -), so the flag is not recognised and is read as the image name.

    Check

    printf '%s' '<the flag>' | od -c | head -2   # 342 200 223 is an en dash

    Fix

    Retype the flags by hand.

Prevent it

Quote every variable in scripts and set tags explicitly, so an empty or spaced value cannot become part of the image name.

Now learn it: break and fix it yourself in pull access denied, then take Level 01 · Images and tags. Open in DevOps Nimbus Docker.

the input device is not a TTY

The error

the input device is not a TTY

What it means

-t asks Docker for a terminal (TTY), and Docker can only give one when its own input is a terminal. Here the input is not: a CI job, a cron job, a pipe or a redirect.

Causes, most likely first

  1. docker run -it or docker exec -it in a CI pipeline or cron job, where there is no terminal.

    Check

    tty   # prints 'not a tty' in that environment

    Fix

    Drop -t in scripts and CI; keep -i only if you pipe input in.

  2. Piping into it: cat dump.sql | docker exec -it db psql. The input is the pipe, not a terminal.

    Check

    set -x   # then run the script again: a | before docker, or a < after it, means its input is not a terminal

    Fix

    Use -i without -t: docker exec -i db psql -U app < dump.sql.

  3. Git Bash (mintty) on Windows, which Docker does not see as a terminal.

    Check

    echo $TERM; uname -s   # MINGW or MSYS in Git Bash

    Fix

    Prefix the command: winpty docker exec -it db sh, or use PowerShell or Windows Terminal.

Prevent it

Use -it only when you type into the container yourself; scripts and CI use -i or nothing.

Now learn it: break and fix it yourself in Exits immediately, then take Level 02 · Why did it exit?. Open in DevOps Nimbus Docker.

Cannot reach it

curl: (7) Couldn't connect to server

The error

curl: (7) Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to server

What it means

curl's error 7: it could not open a TCP connection at all. Nothing listens on localhost port 8080 from where curl runs, so the connection was refused at once.

Causes, most likely first

  1. The container is not running, so nothing is behind the port.

    Check

    docker ps -a --filter name=<name>   # Exited: it is not running

    Fix

    Read docker logs <name> and fix that first.

  2. The port was never published (-p). EXPOSE in the Dockerfile only documents a port; it does not open it on the host.

    Check

    docker port <name>   # empty: nothing published

    Fix

    Re-create the container with -p 8080:<container-port>.

  3. It was published to another host port than the one you call.

    Check

    docker port <name>   # 80/tcp -> 0.0.0.0:8081, for example

    Fix

    Call that port, or re-create with the port you want.

  4. localhost is not where the daemon runs: DOCKER_HOST or a context points at a remote host or VM, so the published port is there.

    Check

    docker context show   # not default (or desktop-linux): the ports are on that host

    Fix

    Connect to that host's address, not localhost.

Prevent it

Keep run options in a compose file so they are not forgotten.

Now learn it: break and fix it yourself in Port not published · Exits immediately, then take Level 06 · Ports. Open in DevOps Nimbus Docker.

curl: (56) Connection reset by peer

The error

curl: (56) Recv failure: Connection reset by peer

What it means

curl's error 56: the connection opened, then the other side closed it before answering. With Docker that usually means the host port is published and Docker accepted the connection, but nothing answered on the container side of it.

Causes, most likely first

  1. The port mapping points at the wrong container port: -p 8080:8080 when the app listens on 80 inside. Docker forwards to a port where nothing listens.

    Check

    docker port <name>   # 8080/tcp -> ...
    docker logs <name>   # the app says it listens on 80

    Fix

    Publish to the port the app uses: -p 8080:80.

  2. The app listens on 127.0.0.1 inside the container instead of 0.0.0.0, so traffic arriving from Docker's network is not accepted.

    Check

    docker exec <name> wget -qO- localhost:<port>   # works inside the container, while curl from the host is reset
    docker exec <name> netstat -ltn   # 127.0.0.1:<port> instead of 0.0.0.0:<port> (needs netstat in the image)

    Fix

    Make the app listen on 0.0.0.0. To keep it private, restrict the host side instead: -p 127.0.0.1:8080:80.

  3. The app is still starting, or crashed right after the connection.

    Check

    docker logs <name>   # still starting, or a crash

    Fix

    Wait for its healthcheck before sending traffic, or fix the crash.

Prevent it

Configure servers to bind 0.0.0.0 in containers; restrict exposure with -p 127.0.0.1:... instead.

Now learn it: break and fix it yourself in Port not published · Listening on 127.0.0.1, then take Level 06 · Ports. Open in DevOps Nimbus Docker.

getaddrinfo ENOTFOUND / bad address

The error

Error: getaddrinfo ENOTFOUND cache

What it means

getaddrinfo is the call that turns a name into an address, and ENOTFOUND means no address came back for cache. The container asked DNS for that name and nobody knew it.

Causes, most likely first

  1. Both containers are on the default bridge, which has no DNS for container names. Only user-defined networks get Docker's embedded DNS.

    Check

    docker exec <name> cat /etc/resolv.conf   # not 127.0.0.11: no Docker DNS, so the default bridge
    docker inspect <name> --format '{{json .NetworkSettings.Networks}}'   # only bridge

    Fix

    docker network create app-net, then docker network connect app-net <each container>.

  2. They are on different user-defined networks, and a name only resolves for containers on the same network.

    Check

    docker inspect <name> --format '{{json .NetworkSettings.Networks}}'; docker inspect <other> --format '{{json .NetworkSettings.Networks}}'   # no network in common

    Fix

    Connect both to one network: docker network connect <network> <container>.

  3. The target container is stopped, so its name is no longer registered in Docker's DNS.

    Check

    docker ps -a --filter name=<other>   # Exited

    Fix

    Start it, and fix why it stopped.

  4. The name is wrong: in Compose, use the service name (cache), not a container name from another project.

    Check

    docker compose config --services   # the names Compose registers in DNS

    Fix

    Use the service name, or add an alias on the shared network.

  5. An outside name (a database host, an API) fails. Containers use the host's DNS servers, which the daemon passes on, and those cannot resolve it or are not reachable from containers.

    Check

    docker exec <name> getent hosts <outside-name>   # nothing, while the same lookup works on the host

    Fix

    Fix the host's DNS, or set "dns" in /etc/docker/daemon.json to resolvers the containers can reach.

Prevent it

Put services that talk to each other on one user-defined network (Compose does it by default).

Now learn it: break and fix it yourself in Name not found · Different networks, then take Level 07 · Networks and DNS. Open in DevOps Nimbus Docker.

ECONNREFUSED 127.0.0.1

The error

Error: connect ECONNREFUSED 127.0.0.1:5432

What it means

The app tried to connect to 127.0.0.1 port 5432 and was refused: nothing listens there. Inside a container, localhost is the container itself, not the host and not other containers.

Causes, most likely first

  1. A host name copied from a laptop where everything ran locally. In the container, localhost points at the container, where the database is not running.

    Check

    docker exec <name> printenv DB_HOST   # localhost or 127.0.0.1 (prints one variable, not the secrets next to it)

    Fix

    Point it at the other container's name (DB_HOST=db) on a shared network.

  2. The other service runs in its own container and must be reached by name over a network both containers share.

    Check

    docker network inspect <network> --format '{{range .Containers}}{{.Name}} {{end}}'   # both containers listed

    Fix

    Put both on one user-defined network and use the service name.

  3. The service runs on the host itself, which is not localhost from inside a container.

    Check

    ss -ltn   # the service listens on the host

    Fix

    Use host.docker.internal (see the host-internal entry).

Prevent it

Never default service addresses to localhost in container configuration.

Now learn it: break and fix it yourself in localhost is the container · Compose: wrong host · Compose: started too early, then take Level 07 · Networks and DNS. Open in DevOps Nimbus Docker.

host.docker.internal not found on Linux

The error

Error: getaddrinfo ENOTFOUND host.docker.internal

What it means

The container could not resolve host.docker.internal, the name for the machine running Docker. Docker Desktop defines it for every container; Docker Engine on Linux does not, unless you ask for it.

Causes, most likely first

  1. Configuration written on Docker Desktop is run on a Linux server, where the name was never added.

    Check

    docker inspect <name> --format '{{.HostConfig.ExtraHosts}}'   # empty

    Fix

    docker run --add-host=host.docker.internal:host-gateway ... (Engine 20.10+), or extra_hosts: ["host.docker.internal:host-gateway"] in Compose.

  2. The name resolves, but the host service listens only on 127.0.0.1, so containers still cannot connect (ECONNREFUSED or a timeout instead of this error).

    Check

    docker exec <name> getent hosts host.docker.internal   # an address such as 172.17.0.1
    ss -ltn   # 127.0.0.1:<port> only

    Fix

    Make the host service listen on the docker0 bridge address as well, and allow that port from the bridge in the host firewall.

Prevent it

Add the extra host in the compose file so it works the same everywhere.

Now learn it: break and fix it yourself in Reaching the host, then take Level 07 · Networks and DNS. Open in DevOps Nimbus Docker.

A published port is open despite ufw

The error

0.0.0.0:5432->5432/tcp

What it means

docker ps shows the port published on 0.0.0.0, every interface. Docker adds its own iptables rules for published ports, and they are evaluated before ufw's. Traffic to a published port is forwarded to the container, so ufw's INPUT rules never see it.

Causes, most likely first

  1. -p 5432:5432 (or ports: "5432:5432" in Compose) publishes on every interface unless an address is given.

    Check

    docker ps --format '{{.Names}} {{.Ports}}'   # 0.0.0.0: or [::]: is open on every interface; 127.0.0.1: is local only

    Fix

    Re-create the container without -p (reach it over a Docker network), or with -p 127.0.0.1:5432:5432.

  2. A ufw (or firewalld) rule cannot override Docker's own rules, because the traffic is forwarded before ufw's rules for the host see it.

    Check

    sudo iptables -t nat -L DOCKER -n   # the DNAT rule for the port
    sudo iptables -L DOCKER-USER -n   # empty: nothing filters published ports
    sudo ufw status   # 5432 denied, yet reachable

    Fix

    If it must be published but only to some sources, filter it in the DOCKER-USER chain, which Docker leaves to you. Not "iptables": false in daemon.json: it breaks container networking instead of securing it.

Prevent it

Publish only public services; review docker ps PORTS in security checks.

Now learn it: break and fix it yourself in Database open to the internet, then take Level 15 · Container security. Open in DevOps Nimbus Docker.

network not found

The error

Error response from daemon: network app-net not found

What it means

The container asked to join the network app-net, and this daemon has no network with that name.

Causes, most likely first

  1. The network was never created, or docker network prune removed it.

    Check

    docker network ls   # app-net is not listed

    Fix

    docker network create app-net

  2. Compose prefixes networks with the project name: shop_default, not default. A docker run --network that uses the short name finds nothing.

    Check

    docker network ls   # shop_default or <project>_app-net
    docker compose config   # the networks section shows the names Compose uses

    Fix

    Use the name docker network ls shows.

  3. An external network in compose.yaml that nobody has created yet. Compose does not create external networks; it expects them.

    Check

    docker compose config | grep -n -A2 external   # external: true on that network

    Fix

    Create it first, or drop external: true and let Compose create it.

Prevent it

Create shared networks in the same script or Compose file that uses them, and let Compose own the networks it needs.

Now learn it: break and fix it yourself in Different networks, then take Level 07 · Networks and DNS. Open in DevOps Nimbus Docker.

Files and permissions

Cannot find module after mounting a folder

The error

Error: Cannot find module 'express'

What it means

Node looked for the package express in node_modules and did not find it. In a container that has mounts, the usual reason is that a mount replaced what the image had at that path, so the files the build installed are hidden.

Causes, most likely first

  1. A bind mount of your project over /app hides the image's /app/node_modules. Your host folder has no node_modules (or one for another platform), and that is what the app now sees.

    Check

    docker inspect <name> --format '{{json .Mounts}}'   # a bind mount with Destination /app
    docker run --rm --entrypoint ls <image> /app/node_modules/express   # present without the mount: the mount hides it

    Fix

    Add a volume for the inner path so the image's copy shows through: -v /app/node_modules.

  2. A named volume from an older version holds stale files at that path, and a volume that already has content is not refreshed from the image.

    Check

    docker inspect <name> --format '{{json .Mounts}}'   # a volume at /app or /app/node_modules, created before the dependency was added

    Fix

    Remove that volume (after checking it holds nothing you need) so it is filled again from the image.

  3. No mount at all: the module is really missing from the image (not in package.json, or a devDependency left out by npm ci --omit=dev).

    Check

    docker run --rm --entrypoint ls <image> /app/node_modules/express   # No such file: the image lacks it
    grep -n '"express"' package.json   # under devDependencies, or not there

    Fix

    Add it to dependencies and rebuild.

Prevent it

Mount only what you edit (src/), or use Compose watch for development.

Now learn it: break and fix it yourself in Mount hides the app, then take Level 08 · Volumes and bind mounts. Open in DevOps Nimbus Docker.

EACCES: permission denied on a mount

The error

Error: EACCES: permission denied, open '/data/uploads/.write-test'

What it means

EACCES is the kernel's permission denied. The app tried to create /data/uploads/.write-test and the kernel refused. On a mounted folder, the check is made by UID number on the host's files, and on some systems also by SELinux.

Causes, most likely first

  1. The host folder belongs to root (or another UID) and the app runs as a non-root UID. Names do not matter, only numbers: user app in the image may be UID 1000 or 10001.

    Check

    docker exec <name> id   # the UID the container really runs as
    ls -ln <folder>   # the owner's number differs from that UID

    Fix

    sudo chown -R <uid>:<gid> <folder>, or run the container with --user <owner-uid>. Not chmod 777: it hides the cause and lets every user on the host write there.

  2. SELinux (Fedora, RHEL and similar): the folder's label does not allow containers to use it, even when the UIDs match.

    Check

    getenforce; ls -Zd <folder>   # Enforcing, and a label without container_file_t
    sudo ausearch -m avc -ts recent   # an avc denied line for the path

    Fix

    Add :Z to the bind mount (-v /data/uploads:/data/uploads:Z), or :z when several containers share it. Not setenforce 0.

  3. Rootless Docker or userns-remap: container UIDs map to other UIDs on the host, so the owner you see inside is not the owner on disk.

    Check

    docker info --format '{{.SecurityOptions}}'   # name=rootless or name=userns

    Fix

    chown to the mapped UID (see /etc/subuid), or use a named volume.

  4. The mount is read-only, or an NFS export squashes the UID, so writes fail whatever the owner.

    Check

    docker inspect <name> --format '{{range .Mounts}}{{.Destination}} rw={{.RW}} {{end}}'   # rw=false
    findmnt -T <folder>   # nfs, with root_squash on the server

    Fix

    Mount it read-write, or set the owner on the NFS server.

Prevent it

Give images a fixed numeric UID and create host folders with that owner; use named volumes where the host does not need the files.

Now learn it: break and fix it yourself in Permission denied, then take Level 09 · Users and permissions. Open in DevOps Nimbus Docker.

Read-only file system

The error

mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)

What it means

Error 30 is EROFS: the app wrote to a file system that is mounted read-only. Here nginx tried to create its cache folder. Usually the container runs with --read-only and that path is outside its writable mounts.

Causes, most likely first

  1. --read-only (read_only: true in Compose) makes the whole root file system read-only, and the app writes caches, pid files or temp files there.

    Check

    docker inspect <name> --format '{{.HostConfig.ReadonlyRootfs}}'   # true
    docker logs <name>   # the path it tried to write

    Fix

    Add --tmpfs (or a volume) for each path the app writes: --tmpfs /var/cache/nginx --tmpfs /var/run.

  2. The path is on a volume or bind mount marked :ro.

    Check

    docker inspect <name> --format '{{range .Mounts}}{{.Destination}} rw={{.RW}} {{end}}'   # rw=false on that path

    Fix

    Mount that path read-write, or point the app's writes elsewhere.

  3. The host file system itself went read-only: after disk errors, the kernel remounts it read-only to protect it.

    Check

    sudo dmesg -T | grep -i 'remount.*read-only'   # the host disk failed

    Fix

    Check and repair the disk before anything else; do not just restart.

Prevent it

Test images with --read-only in CI so writable paths are known.

Now learn it: break and fix it yourself in Read-only file system, then take Level 15 · Container security. Open in DevOps Nimbus Docker.

Operation not permitted (capabilities)

The error

chown("/var/cache/nginx/client_temp", 101) failed (1: Operation not permitted)

What it means

Error 1 is EPERM: the kernel did not allow the operation. Here nginx, running as root in the container, tried to chown a folder to UID 101. Root in a container only has the Linux capabilities Docker gives it, and this operation needs one it lacks.

Causes, most likely first

  1. --cap-drop removed a capability it needs. chown needs CHOWN; switching user needs SETUID and SETGID.

    Check

    docker inspect <name> --format '{{.HostConfig.CapDrop}} {{.HostConfig.CapAdd}}'   # ALL, or that capability, dropped

    Fix

    Add back exactly what the logs ask for: --cap-add CHOWN --cap-add SETUID --cap-add SETGID. Not --privileged: it gives every capability and device to fix one.

  2. The container does not run as root at all (--user, or USER in the image), so it has no capabilities to chown.

    Check

    docker exec <name> grep CapEff /proc/1/status   # 0000000000000000: no capabilities
    docker inspect <name> --format '{{.Config.User}}'   # a non-root user

    Fix

    Set the ownership at build time (COPY --chown) so the app does not chown at start.

  3. The path is on a mount that refuses chown from containers: NFS with root squash, or a share from the host.

    Check

    docker inspect <name> --format '{{json .Mounts}}'   # the path is on a bind mount or NFS volume

    Fix

    Set the owner on the server or host side.

  4. Rootless Docker or userns-remap, where the target UID is not mapped into the container's user namespace.

    Check

    docker info --format '{{.SecurityOptions}}'   # name=rootless or name=userns

    Fix

    Use a UID inside the mapped range, or a named volume.

Prevent it

Prefer images that run as non-root (nginx-unprivileged), which need no capabilities.

Now learn it: break and fix it yourself in Capabilities dropped, then take Level 15 · Container security. Open in DevOps Nimbus Docker.

Data gone after re-creating a container

The error

(integer) 0

What it means

The data the app wrote is not there after the container was re-created. It lived in the old container's writable layer, or in a volume the new container does not use.

Causes, most likely first

  1. No volume at all: the data was in the container's writable layer and was deleted with docker rm.

    Check

    docker inspect <name> --format '{{json .Mounts}}'   # [] : nothing mounted

    Fix

    Restore from a backup, and re-create with a named volume: -v redis-data:/data.

  2. The image's VOLUME gave each container its own anonymous volume, so the new container got a new, empty one. The old one may still exist.

    Check

    docker image inspect <image> --format '{{json .Config.Volumes}}'   # /data listed
    docker volume ls   # long 64-character names: anonymous volumes

    Fix

    Mount the old anonymous volume into the new container to recover, then use a named volume.

  3. Compose prefixes volumes with the project name, so a new folder name or -p made a new, empty volume (shop_data instead of app_data).

    Check

    docker volume ls   # an older <project>_data next to the new one
    docker compose config --volumes   # the volume names this project expects, before the prefix

    Fix

    Keep the project name fixed (name: in compose.yaml), or mark the volume external with its real name.

  4. docker compose down -v, or docker volume prune, deleted the volume.

    Check

    docker volume ls   # no volume with the old name at all

    Fix

    Restore from a backup. Use down -v only when you mean to delete data.

  5. The volume is mounted at another path than the one the app writes to.

    Check

    docker inspect <name> --format '{{range .Mounts}}{{.Destination}} {{end}}'   # not the app's data folder (redis writes /data)

    Fix

    Mount the volume at the path the app writes to.

Prevent it

Always name volumes for data; back them up.

Now learn it: break and fix it yourself in Data gone after re-create, then take Level 08 · Volumes and bind mounts. Open in DevOps Nimbus Docker.

not a directory: mount a directory onto a file

The error

error mounting "/home/app/nginx.conf" to rootfs at "/etc/nginx/nginx.conf": ... not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)?

What it means

A bind mount tried to put a directory on a file, or a file on a directory, and the kernel cannot do that. Usually the host file did not exist the first time the container was created, so Docker created an empty directory with its name.

Causes, most likely first

  1. The host path was missing or misspelt the first time. docker run -v creates a directory for a missing source, and that directory now sits where the file should be.

    Check

    ls -ld /home/app/nginx.conf   # drwx...: a directory where the file should be

    Fix

    rmdir /home/app/nginx.conf, put the real file there, then docker rm <name> and start it again.

  2. The command ran from another folder, so $(pwd) pointed somewhere else, or the path in -v or compose.yaml is wrong.

    Check

    docker inspect <name> --format '{{json .Mounts}}'   # the Source path Docker really used

    Fix

    Fix the path in -v or compose.yaml.

Prevent it

Use --mount type=bind,source=...,target=... in scripts: it fails when the source does not exist instead of creating a directory.

Now learn it: break and fix it yourself in Mount hides the app, then take Level 08 · Volumes and bind mounts. Open in DevOps Nimbus Docker.

volume is in use

The error

Error response from daemon: remove pgdata: volume is in use - [3f2a1b9c8d7e]

What it means

docker volume rm refused to remove pgdata because a container, even a stopped one, still refers to it. The ID in brackets is that container.

Causes, most likely first

  1. A stopped container still has the volume attached. Stopping a container does not detach its volumes.

    Check

    docker ps -a --filter volume=pgdata   # the container from the error, often Exited

    Fix

    Back up anything you need first (the data is gone once the volume is removed), then docker rm <container> and docker volume rm pgdata.

  2. A Compose project was stopped (docker compose stop), not removed, so its containers still exist.

    Check

    docker compose ps -a   # the project's containers, Exited

    Fix

    docker compose down, then remove the volume if you really mean to.

Prevent it

Remove a project with docker compose down, and use down -v only when you mean to delete its data.

Now learn it: break and fix it yourself in Data gone after re-create, then take Level 08 · Volumes and bind mounts. Open in DevOps Nimbus Docker.

Security findings

Image scan finds critical or high vulnerabilities

The error

✗ Detected 7 vulnerable packages with a total of 7 vulnerabilities

What it means

A scanner compared the packages in the image with lists of known vulnerabilities (CVEs) and found matches. Most come from the base image, often in packages the app never uses.

Causes, most likely first

  1. A full base image ships compilers, git, imagemagick and many other packages, and each one adds its own CVEs.

    Check

    docker scout quickview <image>   # how much comes from the base image
    docker scout cves <image>   # the packages; many you did not install yourself

    Fix

    Move to a slim, alpine or distroless base.

  2. The base was not rebuilt after fixes were published, so the image keeps the old, vulnerable package versions.

    Check

    docker image inspect <image> --format '{{.Created}}'   # weeks or months old
    docker scout cves <image>   # most findings have a fixed version

    Fix

    Rebuild with --pull to pick up fixed packages, re-scan, and gate on critical and high.

Prevent it

Scan in CI with an exit code, rebuild images on a schedule, and keep bases small.

Now learn it: break and fix it yourself in Scan gate fails, then take Level 15 · Container security. Open in DevOps Nimbus Docker.

docker.sock mounted into a container

The error

-v /var/run/docker.sock:/var/run/docker.sock

What it means

The container has the host's Docker socket. Whatever runs in it can use the full Docker API: start a privileged container, mount the host's root, and own the host. A :ro mount does not limit API calls; it only stops the socket file from being replaced.

Cause

  1. Dashboards, CI agents or log shippers are given the socket for convenience, with no limit on what they can ask the daemon.

    Check

    docker ps --filter volume=/var/run/docker.sock   # every container that has it
    docker inspect <name> --format '{{json .Mounts}}'   # the socket as Source

    Fix

    Remove the socket mount, or give the container a socket proxy that allows only the API calls it needs, over a network (DOCKER_HOST=tcp://socket-proxy:2375).

Prevent it

Treat socket access as root on the host: review it like any privileged access, and alert on new mounts of it.

Now learn it: break and fix it yourself in Docker socket in a container, then take Level 15 · Container security. Open in DevOps Nimbus Docker.

Images and builds

The image is far bigger than the app

The error

nimbus/shop   1.0   1.62GB

What it means

The image is much bigger than the app it runs. Image size is the sum of every layer, and a file deleted in a later step still takes space in the layer that added it.

Causes, most likely first

  1. COPY . . without a .dockerignore sends .git, node_modules and logs into the image.

    Check

    docker build --progress=plain .   # transferring context: hundreds of MB
    docker history <image>   # a large COPY layer

    Fix

    Add a .dockerignore.

  2. A full base image (node:22 is over 1 GB) brings a whole operating system and build tools.

    Check

    docker history <image>   # most of the size in the first layers, from FROM

    Fix

    Use a slim or alpine base.

  3. Build tools and dev dependencies are shipped in the final image.

    Check

    docker history <image>   # a large RUN npm ci or apt-get install build-essential layer

    Fix

    Use a multi-stage build: build in one stage, copy only production files into the final one.

  4. A file deleted in a later RUN still takes space in the earlier layer that created it.

    Check

    docker history --no-trunc <image>   # a large layer, then a later rm of the same files

    Fix

    Delete it in the same RUN that created it, or keep it out of the final stage.

Prevent it

Track image size in CI and fail on large jumps.

Now learn it: break and fix it yourself in 1.6 GB image, then take Level 05 · Small images. Open in DevOps Nimbus Docker.

Every build reinstalls dependencies

The error

 => [4/4] RUN npm ci   (never CACHED)

What it means

The build cache reuses a step only when the step and everything it copies are unchanged. When one step's input changes, that step and every step after it run again. Here the dependency install is one of those steps on every build.

Causes, most likely first

  1. COPY . . comes before the install step, so any change to any file breaks the cache for the install.

    Check

    docker build --progress=plain .   # the COPY . . step is the first one without CACHED
    grep -n -E 'COPY|RUN' Dockerfile   # COPY . . above RUN npm ci

    Fix

    COPY package*.json ./ then RUN npm ci, and only then COPY . .

  2. A file that changes on every build (a version file, a build date) is copied early, or an ARG used before the step changes every build (a commit SHA).

    Check

    docker build --progress=plain .   # the first step without CACHED copies that file or uses that ARG

    Fix

    Copy or use that value as late in the Dockerfile as possible.

  3. --no-cache was passed, or the build cache was pruned.

    Check

    docker buildx du   # little or no cache

    Fix

    Drop --no-cache from routine builds, and prune less often.

  4. Another builder: a buildx builder with the docker-container driver keeps its own cache, separate from the default one.

    Check

    docker buildx ls   # the builder marked * is the one in use

    Fix

    Build with the same builder each time (docker buildx use <name>).

Prevent it

Order instructions from least to most often changed; in CI, reuse cache with --cache-from.

Now learn it: break and fix it yourself in Cache busted on every change, then take Level 04 · Layers and the build cache. Open in DevOps Nimbus Docker.

Cache export is not supported for the docker driver

The error

ERROR: Cache export is not supported for the docker driver.

What it means

--cache-to asked the builder to export its cache, and the default docker driver cannot do that with the classic image store. CI runners also start every job with an empty cache, so without an exported cache every step runs again.

Causes, most likely first

  1. Building in CI with the default docker driver and the classic image store, which cannot export cache.

    Check

    docker buildx ls   # DRIVER docker on the builder marked *
    docker info --format '{{.DriverStatus}}'   # no io.containerd.snapshotter: the classic image store

    Fix

    docker buildx create --name ci --driver docker-container --use, or turn on the containerd image store, with which the docker driver can export cache.

  2. No --cache-from, or a cache nobody has exported yet, so there is nothing to import.

    Check

    docker buildx build --progress=plain ...   # no 'importing cache manifest' line

    Fix

    docker buildx build --cache-from type=registry,ref=<image>:buildcache --cache-to type=registry,ref=<image>:buildcache,mode=max .

Prevent it

Export and import the cache in every pipeline (registry, or type=gha on GitHub Actions).

Now learn it: break and fix it yourself in CI rebuilds everything, then take Level 04 · Layers and the build cache. Open in DevOps Nimbus Docker.

COPY failed: not found

The error

failed to calculate checksum of ref ...: "/dist": not found

What it means

COPY asked for /dist, and the build context (the files the client sent to the builder) does not contain it. What is on your disk does not matter, only what was sent. BuildKit reports this as failed to calculate checksum ... not found; the legacy builder as COPY failed.

Causes, most likely first

  1. A .dockerignore pattern excludes it, so it is never sent.

    Check

    grep -n dist .dockerignore   # a matching pattern

    Fix

    Remove the pattern from .dockerignore, or add !dist.

  2. The build was started from another folder, or -f points elsewhere. The context is the last argument (.), not the Dockerfile's folder.

    Check

    docker build --progress=plain .   # transferring context: a tiny size means the wrong folder

    Fix

    Build from the right folder, or pass the context explicitly.

  3. The path starts outside the context (../file), which a build can never reach.

    Check

    grep -n 'COPY \.\./' Dockerfile   # a path going up

    Fix

    Move the file into the context, or build from a parent folder with -f.

  4. The file is made by a step that has not run: dist/ only exists after npm run build.

    Check

    ls -ld dist   # missing on disk

    Fix

    Build it in a stage of the Dockerfile (RUN npm run build) instead of relying on the host.

  5. COPY --from=<stage> names a path that stage never created.

    Check

    grep -n 'COPY --from' Dockerfile   # then check that the stage creates that path

    Fix

    Copy the path that stage really writes.

Prevent it

Keep .dockerignore next to the Dockerfile and review it with the Dockerfile.

Now learn it: break and fix it yourself in COPY: not found, then take Level 03 · Your first Dockerfile. Open in DevOps Nimbus Docker.

dockerfile parse error: unknown instruction

The error

dockerfile parse error on line 2: unknown instruction: COPPY

What it means

The Dockerfile parser read line 2 and its first word, COPPY, is not a Dockerfile instruction. Every line (after joining lines that end in \) must start with one.

Causes, most likely first

  1. A typo (COPPY, RUNN).

    Check

    sed -n '2p' Dockerfile   # the line the error names

    Fix

    Fix the word.

  2. A missing \ at the end of a continued RUN line, so the next line is read as a new instruction.

    Check

    sed -n '1,2p' Dockerfile | cat -A   # the line before must end in \$

    Fix

    Add the missing \.

  3. A heredoc (RUN <<EOF) built with the legacy builder, which does not understand heredocs, so their lines are read as instructions.

    Check

    docker buildx version; echo "DOCKER_BUILDKIT=${DOCKER_BUILDKIT:-unset}"   # no buildx, or DOCKER_BUILDKIT=0: the legacy builder

    Fix

    Build with BuildKit (the default since Engine 23.0): install the buildx plugin, and do not set DOCKER_BUILDKIT=0.

Prevent it

Lint Dockerfiles in CI (hadolint, or the X-ray on this site).

Now learn it: break and fix it yourself in COPY: not found, then take Level 03 · Your first Dockerfile. Open in DevOps Nimbus Docker.

did not complete successfully: exit code

The error

process "/bin/sh -c npm ci" did not complete successfully: exit code: 1

What it means

A RUN step's command exited with a non-zero code (1 here), so the build stopped. The code is the command's own; the reason is in the step's output just above this line.

Causes, most likely first

  1. A package install cannot authenticate (E401) or reach the network (DNS, or a proxy the build does not have).

    Check

    docker build --progress=plain --no-cache .   # E401, ETIMEDOUT or ENOTFOUND above the error
    docker build --progress=plain --build-arg HTTPS_PROXY=$HTTPS_PROXY .   # passes now: the build needed the proxy

    Fix

    Private packages: a secret mount (see the secret-in-image entry), never an ARG with the token. Network: pass the proxy as a build argument (HTTPS_PROXY is predefined).

  2. npm ci without a package-lock.json, or one out of step with package.json.

    Check

    git ls-files package-lock.json   # nothing: not committed

    Fix

    Commit the lock file, generated from the same package.json.

  3. A command that is not in the base image (exit code 127).

    Check

    docker run --rm -it <base-image> sh   # try the command by hand

    Fix

    Install it first in the Dockerfile, or use a base that has it.

  4. The base image moved to a new version, so a package or path is different.

    Check

    docker image inspect <base-image> --format '{{.Created}}'   # newer than the last good build

    Fix

    Pin the base image to a version or digest.

Prevent it

Commit lock files; build in CI with the same base images.

Now learn it: break and fix it yourself in Token in a layer, then take Level 03 · Your first Dockerfile. Open in DevOps Nimbus Docker.

A secret found in an image

The error

//npm.pkg.github.com/:_authToken=...

What it means

A credential is stored in the image. Anything copied into a layer, set with ENV, or passed as a build argument stays in the image, and anyone who can pull it can read it.

Causes, most likely first

  1. COPY .npmrc or .env (or COPY . . with no .dockerignore entry for them). A later rm does not help: the file stays in the earlier layer.

    Check

    docker run --rm --entrypoint ls <image> -la /root/.npmrc /app/.env   # the file is still there
    grep -n -E 'npmrc|\.env' .dockerignore   # nothing: they are sent

    Fix

    Revoke the token first. Then RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci, with docker build --secret id=npmrc,src=.npmrc .

  2. ARG or ENV with a token. Build arguments are recorded in docker history; ENV is in the image config.

    Check

    docker history --no-trunc --format '{{.CreatedBy}}' <image> | grep -i -c -E 'token|password|secret'   # a count, not the values; above 0: read those lines privately
    docker image inspect <image> --format '{{range .Config.Env}}{{println .}}{{end}}' | cut -d= -f1   # variable names only

    Fix

    Revoke the token, then pass it as a build secret instead of ARG or ENV.

Prevent it

Scan images for secrets in CI; keep credential files in .dockerignore.

Now learn it: break and fix it yourself in Token in a layer, then take Level 15 · Container security. Open in DevOps Nimbus Docker.

failed to read dockerfile: no such file

The error

ERROR: failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory

What it means

docker build looked for a file named Dockerfile (or the one -f names) and did not find it. BuildKit says failed to read dockerfile; the legacy builder says unable to evaluate symlinks in Dockerfile path.

Causes, most likely first

  1. The build ran from another folder, so . is not the project and has no Dockerfile.

    Check

    pwd; ls -la   # no Dockerfile here

    Fix

    cd to the folder with the Dockerfile, or pass its path with -f.

  2. The file has another name (Dockerfile.dev, or Dockerfile.txt saved by an editor).

    Check

    ls -la | grep -i dockerfile   # the real name

    Fix

    Rename it to Dockerfile, or pass its real name with -f.

  3. -f points at a path that does not exist.

    Check

    ls -l path/to/Dockerfile   # No such file

    Fix

    Fix the path: docker build -f path/to/Dockerfile -t <image> path/to/context.

Prevent it

Keep the Dockerfile at the root of the build context, and pass -f and the context explicitly in scripts and CI.

Now learn it: break and fix it yourself in COPY: not found, then take Level 03 · Your first Dockerfile. Open in DevOps Nimbus Docker.

unable to delete image (must be forced)

The error

Error response from daemon: conflict: unable to delete 5f1c0b7a9e3d (must be forced) - image is being used by stopped container 3f2a1b9c8d7e

What it means

Docker would not delete the image because a container, even a stopped one, was made from it. (must be forced) means -f would remove it anyway; (cannot be forced) means a running container uses it.

Causes, most likely first

  1. Stopped containers made from the image were never removed.

    Check

    docker ps -a --filter ancestor=<image>   # Exited containers

    Fix

    docker rm <container> (the one named in the error), then docker rmi <image>.

  2. A running container uses it, and then it cannot be forced at all.

    Check

    docker ps --filter ancestor=<image>   # Up ...

    Fix

    Stop and remove that container first, if it is no longer needed.

  3. Several tags point at the same image ID, and docker rmi <id> refuses until you remove the tags by name.

    Check

    docker image ls --no-trunc | grep <id>   # more than one REPOSITORY:TAG with that ID

    Fix

    docker rmi <repo>:<tag> for each tag.

Prevent it

Run short-lived containers with --rm, and clean up with docker system df and the prune commands instead of deleting images one by one.

Now learn it: break and fix it yourself in No space left on device, then take Level 16 · Troubleshooting and cleanup. Open in DevOps Nimbus Docker.

Registries and upgrades

pull access denied, repository does not exist

The error

pull access denied for ngnix, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

What it means

The registry answered the pull for ngnix with denied. Registries give the same answer for a repository that does not exist and for a private one you may not read, so they do not reveal private names. The message cannot tell you which; the checks below can.

Causes, most likely first

  1. A typo in the image name (ngnix instead of nginx): no such repository exists.

    Check

    docker search <name>   # Docker Hub only: the right spelling

    Fix

    Correct the name.

  2. No registry or namespace in the name, so it resolves to Docker Hub's library: shop means docker.io/library/shop, which is not your image.

    Check

    echo <image>   # no registry.example.internal/ or <user>/ in front

    Fix

    Write the full name: registry.example.internal/team/shop:1.0.

  3. A private repository, and you are not logged in, or logged in as an account without access.

    Check

    jq -r '(.auths // {} | keys[]), (.credsStore // empty)' ~/.docker/config.json   # that registry is not listed (registry names only, no credentials)

    Fix

    docker login <registry> with an account that can read it.

  4. sudo docker pull uses root's login (/root/.docker/config.json), not yours, so it pulls anonymously.

    Check

    docker buildx imagetools inspect <image>   # works as you while sudo docker pull fails: root's login is the cause

    Fix

    Run docker without sudo (docker group), or log in as root too.

Prevent it

Pin images in one place (compose file, variables) instead of retyping them.

Now learn it: break and fix it yourself in pull access denied, then take Level 01 · Images and tags. Open in DevOps Nimbus Docker.

manifest unknown

The error

manifest for nginx:1.72 not found: manifest unknown: manifest unknown

What it means

The registry found the repository nginx but no tag 1.72 in it. A manifest is the registry's record of one image version, and there is none under that tag (or digest).

Causes, most likely first

  1. A tag that was never pushed (the CI push failed, or went to another repository), or a version typo.

    Check

    docker buildx imagetools inspect <repo>:<another-known-tag>   # works: the repository is there and this tag is not

    Fix

    Use a tag that exists, or the digest.

  2. The tag was deleted by a registry retention or cleanup policy.

    Check

    docker buildx imagetools inspect <image>@sha256:<digest>   # the digest you deployed is gone too

    Fix

    Push it again from CI, and exclude release tags from cleanup policies.

  3. The right tag is in another repository: the image was pushed under a different path or registry.

    Check

    grep -rn 'docker push\|tags:' .github/ Makefile 2>/dev/null   # where CI really pushes

    Fix

    Pull from the path CI pushes to.

Prevent it

Promote images by digest between environments.

Now learn it: break and fix it yourself in pull access denied, then take Level 01 · Images and tags. Open in DevOps Nimbus Docker.

x509: certificate signed by unknown authority

The error

tls: failed to verify certificate: x509: certificate signed by unknown authority

What it means

Docker connected to the registry over HTTPS, and the certificate it was given is signed by a certificate authority (CA) this machine does not trust. The connection stops before any image is pulled. Often it is a company's own CA or a proxy that inspects TLS.

Causes, most likely first

  1. An internal registry uses a private CA, which is not in the system's or Docker's trust store.

    Check

    openssl s_client -connect <registry>:443 -servername <registry> -showcerts </dev/null 2>/dev/null | grep -E '^ *[0-9] s:|i:'   # issuer: your company's CA
    ls /etc/docker/certs.d/   # no folder for that registry

    Fix

    sudo mkdir -p /etc/docker/certs.d/<registry>[:port], then sudo cp ca.crt /etc/docker/certs.d/<registry>[:port]/ca.crt (read per pull, no restart). Or add it to the system store (update-ca-certificates) and restart Docker.

  2. A corporate proxy re-signs HTTPS traffic with its own CA, so even public registries look untrusted.

    Check

    openssl s_client -connect registry-1.docker.io:443 -servername registry-1.docker.io </dev/null 2>/dev/null | grep 'i:'   # an issuer that is your proxy, not a public CA

    Fix

    Install the proxy's CA in the system store and restart Docker (Docker Desktop: in the macOS keychain or Windows certificate store, then restart it).

  3. The registry serves only its own certificate without the intermediate, so the chain stops short of a trusted root.

    Check

    openssl s_client -connect <registry>:443 -servername <registry> -showcerts </dev/null 2>/dev/null | grep -c 's:'   # 1: only the leaf certificate is sent

    Fix

    Fix the registry to serve the full chain.

  4. The CA is in certs.d under the wrong name. A registry on a port needs the port in the folder name.

    Check

    ls /etc/docker/certs.d/   # registry.example.internal while you pull from registry.example.internal:5000

    Fix

    Rename the folder to match exactly, with :port.

Prevent it

Install the company CA with the machine image or configuration management, and monitor the registry's chain. Never use insecure-registries to get past it: it turns verification off.

Now learn it: break and fix it yourself in x509: unknown authority, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

unauthorized (private registry)

The error

Head "https://ghcr.io/v2/kitchen/shop/manifests/2.1": unauthorized

What it means

The registry needs credentials for this image and got none it accepts. The Head request to /v2/.../manifests/2.1 is Docker asking for that image version, and unauthorized is the registry's answer.

Causes, most likely first

  1. Not logged in to that registry. Credentials are stored per registry host name, so a login to Docker Hub does nothing for ghcr.io.

    Check

    jq -r '(.auths // {} | keys[]), (.credsStore // empty)' ~/.docker/config.json   # the registry is not listed (names only, never the stored credentials)

    Fix

    printf '%s' "$TOKEN" | docker login <registry> -u <user> --password-stdin, then docker pull <image>.

  2. An expired token, or one without read access to the package (on GHCR a token needs read:packages).

    Check

    docker pull <image> 2>&1 | tail -1   # right after a fresh login: still unauthorized means the token lacks read access

    Fix

    Create a token with read access to that package and log in again.

  3. sudo docker pull: the CLI then reads root's config, not yours.

    Check

    sudo jq -r '.auths // {} | keys[]' /root/.docker/config.json   # the registry is missing for root

    Fix

    Run docker without sudo, or log in as root too.

  4. A credential helper (credsStore) cannot return the credentials: Docker Desktop's helper on a server, or a locked keychain.

    Check

    docker-credential-<helper> list   # an error, or the registry missing (URLs and user names only)
    echo "DOCKER_CONFIG=${DOCKER_CONFIG:-unset}"   # set: the CLI reads a different config file

    Fix

    Fix or remove credsStore in config.json, then log in again.

Prevent it

Use read-only pull tokens per machine or CI job, and a credential helper instead of plain config.

Now learn it: break and fix it yourself in Private registry, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

toomanyrequests: pull rate limit

The error

toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit

What it means

Docker Hub limits how many images can be pulled in a period. Unauthenticated pulls are counted per IP address, so every machine behind the same address shares one allowance, and it is used up.

Causes, most likely first

  1. Shared CI runners or NAT gateways pull anonymously from one address, and together they use the allowance.

    Check

    TOKEN=$(curl -s 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull' | jq -r .token); curl -sI -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit   # this address's limit and what is left, without printing the token

    Fix

    printf '%s' "$DOCKERHUB_TOKEN" | docker login -u <user> --password-stdin with a Docker Hub access token, as the user that pulls.

  2. Pulling on every job instead of caching, so even a modest pipeline repeats the same pulls.

    Check

    grep -rn 'docker pull\|--pull always' .github/ 2>/dev/null   # a pull in every job

    Fix

    Cache images on the runner, or use a pull-through mirror (registry-mirrors in daemon.json).

  3. Logged in, but the pull runs as another user (sudo, or a CI step without the login), so it is anonymous.

    Check

    jq -r '.auths // {} | keys[]' ~/.docker/config.json   # https://index.docker.io/v1/ missing for the user that pulls

    Fix

    Log in as the user that pulls, in the same CI job.

  4. Authenticated, and the account's own limit is used up (the message then does not say unauthenticated).

    Check

    jq -r '.auths // {} | keys[]' ~/.docker/config.json   # https://index.docker.io/v1/ listed for the user that pulls: the account's own limit applies

    Fix

    Mirror the base images into your own registry, or use an account with a higher limit.

Prevent it

Authenticate CI pulls and mirror base images into your own registry.

Now learn it: break and fix it yourself in toomanyrequests, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

Running an old version of :latest

The error

shop: ok (version 2.3)

What it means

The container is running an older version than the registry has under the same tag. docker run uses the local image for a tag and never checks the registry for a newer one, and an existing container keeps the image it was created from.

Causes, most likely first

  1. The tag moved in the registry after you last pulled, so the local copy is old.

    Check

    docker image inspect <image> --format '{{.RepoDigests}}'; docker buildx imagetools inspect <image> | grep Digest   # different: the registry has a newer one

    Fix

    docker pull <image> (or docker compose pull), then re-create the container, or docker run --pull always.

  2. You pulled, but the running container was not re-created. docker restart keeps the old image.

    Check

    docker inspect <name> --format '{{.Image}}'; docker image inspect <image> --format '{{.Id}}'   # different: the container runs an older image than the local tag

    Fix

    docker compose up -d (it re-creates when the image changed), or docker rm -f and run again.

  3. docker compose up -d does not pull an image it already has.

    Check

    docker compose images   # the image ID it uses

    Fix

    docker compose pull before docker compose up -d, or pull_policy: always.

Prevent it

Deploy immutable version tags or digests, never :latest.

Now learn it: break and fix it yourself in Old image, new tag, then take Level 01 · Images and tags. Open in DevOps Nimbus Docker.

does not match the detected host platform

The error

WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested

What it means

The image has nothing for this host's CPU architecture: here it is linux/arm64 and the host is linux/amd64. With this warning it still starts and then fails with exec format error; with no matching manifest it is not pulled at all.

Causes, most likely first

  1. A single-platform image built on an Arm laptop and pushed for everyone.

    Check

    docker buildx imagetools inspect <image>   # only linux/arm64 listed

    Fix

    docker build --platform linux/amd64 -t <image> ., or build every platform with docker buildx build --platform linux/amd64,linux/arm64 --push .

  2. --platform, DOCKER_DEFAULT_PLATFORM or platform: in Compose asks for a platform the image does not have.

    Check

    echo "DOCKER_DEFAULT_PLATFORM=${DOCKER_DEFAULT_PLATFORM:-unset}"   # set to another platform
    grep -n platform compose.yaml   # a platform: line on the service

    Fix

    Remove the forced platform.

  3. The publisher builds that image for amd64 only, and this host is arm64 (or the reverse).

    Check

    docker buildx imagetools inspect <image>   # one platform, not yours

    Fix

    Use another image, or run it under emulation (--platform linux/amd64 with QEMU installed), which is much slower.

Prevent it

Publish multi-platform images from CI with docker buildx.

Now learn it: break and fix it yourself in exec format error, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

database files are incompatible with server

The error

FATAL:  database files are incompatible with server

What it means

PostgreSQL started and found a data directory made by another major version. A major version (16, 17) cannot open another one's files; the DETAIL line under the error names both versions.

Causes, most likely first

  1. The image tag moved to a new major (postgres:latest, or an edited version), while the volume still holds the old major's data.

    Check

    docker run --rm -v <volume>:/data:ro alpine:3.20 cat /data/PG_VERSION   # the major that made the data
    docker inspect <name> --format '{{.Config.Image}}'   # the major the container runs

    Fix

    Go back to the major that owns the data. Upgrade deliberately: dump and restore, or pg_upgrade, after a backup.

  2. Another project's or an older volume is mounted by mistake.

    Check

    docker inspect <name> --format '{{json .Mounts}}'   # the volume name is not the one you expected

    Fix

    Mount the right volume.

Prevent it

Pin database images to a major version and upgrade them on purpose.

Now learn it: break and fix it yourself in Database files are incompatible, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

denied: requested access to the resource is denied (push)

The error

The push refers to repository [docker.io/library/shop]
denied: requested access to the resource is denied

What it means

The registry refused the push. The first line shows where Docker tried to push, docker.io/library/shop here: the image name decides the registry and the namespace. You are not logged in, or that namespace is not yours to write to.

Causes, most likely first

  1. The image is tagged without your namespace: shop:1.0 pushes to docker.io/library/shop, which only Docker may write to.

    Check

    docker image ls --format '{{.Repository}}:{{.Tag}}'   # no namespace or registry in front

    Fix

    docker tag shop:1.0 <your-user>/shop:1.0, then docker push <your-user>/shop:1.0.

  2. Not logged in to that registry, or logged in as another account.

    Check

    jq -r '(.auths // {} | keys[]), (.credsStore // empty)' ~/.docker/config.json   # the registry is missing (names only, no credentials)

    Fix

    docker login <registry> as the account that owns the namespace.

  3. The repository belongs to an organization or team that has not given you write access.

    Check

    docker buildx imagetools inspect <your-org>/shop:1.0   # readable while push is denied: you have read but not write

    Fix

    Ask the organization to grant write access, with a token that can write packages.

Prevent it

Tag images with the full registry and namespace from the start (registry.example.internal/team/shop:1.0), and log in with a token in CI.

Now learn it: break and fix it yourself in Private registry, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

Cannot reach the registry (timeout or DNS)

The error

Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

What it means

The Docker daemon tried to reach the registry and got no answer in time. DNS, a proxy or a firewall is in the way. The daemon makes this connection, not your shell, so your shell's proxy variables do not apply to it.

Causes, most likely first

  1. The network needs an HTTP proxy, and the daemon has not been given one (or has a wrong one, or NO_PROXY misses an internal registry).

    Check

    curl -sI https://registry-1.docker.io/v2/   # HTTP 401 from your shell: reachable (with your shell's proxy, if set)
    docker info --format 'http={{.HTTPProxy}} https={{.HTTPSProxy}} no={{.NoProxy}}'   # the daemon's proxy (passwords masked); empty while curl needs one: the cause

    Fix

    Give the daemon the proxy: "proxies": {"https-proxy": "http://proxy.example.internal:3128", "no-proxy": "registry.example.internal"} in /etc/docker/daemon.json (Engine 23.0+), or a systemd drop-in with Environment="HTTPS_PROXY=...". Then sudo systemctl daemon-reload && sudo systemctl restart docker. On Docker Desktop, set it in its settings.

  2. The host cannot resolve the registry's name (lookup ... server misbehaving).

    Check

    getent hosts registry-1.docker.io   # nothing: DNS

    Fix

    Fix the host's resolver, then pull again.

  3. A firewall blocks outbound HTTPS from the host.

    Check

    curl -sI --max-time 5 https://registry-1.docker.io/v2/   # also times out, with no proxy involved

    Fix

    Allow outbound 443 to the registry, or use a mirror inside the network.

Prevent it

Set the daemon's proxy (or a registry mirror) as part of host setup, and check that docker info shows it.

Now learn it: break and fix it yourself in toomanyrequests, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

server gave HTTP response to HTTPS client

The error

Error response from daemon: Get "https://registry.example.internal:5000/v2/": http: server gave HTTP response to HTTPS client

What it means

Docker talks to registries over HTTPS, and this one answered in plain HTTP. Docker will not send a pull over plain HTTP unless that registry is explicitly allowed.

Causes, most likely first

  1. A local or test registry (the registry:2 image) was started without TLS.

    Check

    curl -sI http://registry.example.internal:5000/v2/   # HTTP/1.1 200 or 401 over plain HTTP

    Fix

    Best: give the registry a TLS certificate. For a throwaway test registry only: "insecure-registries": ["registry.example.internal:5000"] in /etc/docker/daemon.json, then sudo systemctl restart docker.

  2. The wrong port: the registry's HTTP port instead of its HTTPS one.

    Check

    curl -sI https://registry.example.internal/v2/   # an answer on 443: use that port

    Fix

    Pull through the HTTPS port.

Prevent it

Serve every registry over TLS, and keep insecure-registries for throwaway test machines. Registries on 127.0.0.0/8 are already allowed over HTTP.

Now learn it: break and fix it yourself in x509: unknown authority, then take Level 14 · Registries and platforms. Open in DevOps Nimbus Docker.

Compose and operations

The variable is not set. Defaulting to a blank string

The error

WARN[0000] The "DB_PASSWORD" variable is not set. Defaulting to a blank string.

What it means

compose.yaml uses ${DB_PASSWORD}, and Compose found no value for it in your shell or in the .env file it reads. It went on with an empty string, which usually breaks the service later.

Causes, most likely first

  1. A fresh checkout without its .env file, or a typo in the variable name.

    Check

    grep -c '^DB_PASSWORD=' .env   # 0, or no such file (prints no value)
    docker compose config --no-interpolate | grep -n DB_PASSWORD   # the exact name the file uses

    Fix

    Create .env next to compose.yaml (and keep it out of git), or export the variable.

  2. .env is not in the project directory (the folder of the first -f file), or --env-file points elsewhere, so Compose never reads it.

    Check

    docker compose ls   # CONFIG FILES: the folder Compose reads .env from
    ls -la <that folder>/.env   # No such file: .env is somewhere else

    Fix

    Run from the project directory, or pass --env-file.

  3. The variable is only in env_file:. That sets it inside the container, but does not fill ${X} in compose.yaml.

    Check

    grep -n -A2 env_file compose.yaml   # the file that holds it

    Fix

    It already reaches the container, so drop the ${DB_PASSWORD} reference from compose.yaml, or also put it in .env.

  4. sudo docker compose: sudo drops variables you exported.

    Check

    printenv DB_PASSWORD >/dev/null && echo set || echo 'not set in this shell'   # set here, but you run Compose with sudo

    Fix

    Run Compose without sudo, or put the value in .env.

  5. A $ inside a value (a password like pa$word) is read as the start of a variable name, so Compose warns about a variable you never meant.

    Check

    grep -n '\$[A-Za-z]' compose.yaml   # a $ inside a value, where you did not mean a variable

    Fix

    Escape it as $$ in compose.yaml.

Prevent it

Use ${X:?X must be set} in compose files so a missing value stops the start.

Now learn it: break and fix it yourself in Compose: variable not set, then take Level 10 · Compose. Open in DevOps Nimbus Docker.

dependency failed to start

The error

dependency failed to start: container orders-db-1 exited (1)

What it means

depends_on makes Compose wait for a service before starting the ones that need it. Here the dependency, orders-db-1, exited with code 1 (or never became healthy), so Compose did not start the services that depend on it.

Causes, most likely first

  1. The dependency crashed on start.

    Check

    docker compose ps -a   # Exited (n) on the dependency
    docker compose logs <service>   # why it crashed

    Fix

    Fix the dependency named in the message, then docker compose up -d.

  2. Its healthcheck never passes (the message then ends in is unhealthy).

    Check

    docker inspect <container> --format '{{json .State.Health}}'   # the check's own output and exit codes

    Fix

    Fix the check (see the unhealthy entry).

  3. The healthcheck gives up before a slow start finishes: start_period, interval or retries too small.

    Check

    docker compose logs <service>   # still starting when it was marked unhealthy

    Fix

    Raise start_period in the healthcheck rather than removing depends_on.

Prevent it

Give every dependency a healthcheck that tests real readiness.

Now learn it: break and fix it yourself in Compose: variable not set · Compose: started too early, then take Level 11 · Healthchecks and restarts. Open in DevOps Nimbus Docker.

(unhealthy) in docker ps

The error

Up 5 minutes (unhealthy)

What it means

The container is running, but its healthcheck command keeps failing, so Docker marks it unhealthy. The healthcheck is a command Docker runs inside the container at intervals; exit code 0 is healthy.

Causes, most likely first

  1. The check uses a tool the image does not have (curl in alpine), so it fails every time, however healthy the app is.

    Check

    docker inspect <name> --format '{{json .State.Health}}'   # Output: curl: not found

    Fix

    Use a tool the image has (wget -qO- http://localhost:3000/ || exit 1), rebuild and re-create.

  2. It checks the wrong port or path.

    Check

    docker exec <name> <the check command>; echo $?   # not 0, while the app works on its real port

    Fix

    Point the check at what the app serves.

  3. The check's timeout is shorter than the endpoint takes to answer.

    Check

    docker inspect <name> --format '{{json .State.Health}}'   # Health check exceeded timeout
    docker inspect <name> --format '{{json .Config.Healthcheck}}'   # the timeout in use

    Fix

    Raise timeout, or check a lighter endpoint.

  4. The app really is stuck, and the check is right.

    Check

    docker exec <name> <the check command>; echo $?   # fails, and the app does not answer on its port either

    Fix

    Fix the app; the logs usually show where it hangs.

Prevent it

Run the healthcheck command with docker exec before baking it into the image.

Now learn it: break and fix it yourself in Unhealthy, then take Level 11 · Healthchecks and restarts. Open in DevOps Nimbus Docker.

configured logging driver does not support reading

The error

Error response from daemon: configured logging driver does not support reading

What it means

docker logs reads a container's logs from the local log files Docker keeps. This container's logging driver keeps none: it sends logs elsewhere or nowhere, and the dual-logging cache that would keep a local copy is off.

Causes, most likely first

  1. --log-driver none: the logs are thrown away.

    Check

    docker inspect <name> --format '{{.HostConfig.LogConfig.Type}}'   # none

    Fix

    Re-create with --log-driver local (or json-file with max-size).

  2. A remote driver (syslog, gelf, awslogs) with dual logging turned off (cache-disabled), or a daemon older than Engine 20.10, which has no dual logging.

    Check

    docker inspect <name> --format '{{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}}'   # a remote driver with cache-disabled true
    docker version --format '{{.Server.Version}}'   # below 20.10

    Fix

    Keep the remote driver and remove cache-disabled, so docker logs reads the local cache (Engine 20.10+); or read the logs where the driver sends them.

Prevent it

Keep a readable local driver with rotation; ship logs from there.

Now learn it: break and fix it yourself in No logs to read, then take Level 13 · Logs and disk space. Open in DevOps Nimbus Docker.

no space left on device

The error

failed to copy files: copy file range failed: no space left on device

What it means

A write failed because the file system it went to is full, or out of inodes (the slots for files). For Docker that is usually the disk holding /var/lib/docker: images, build cache, volumes, or logs.

Causes, most likely first

  1. Old images and build cache from many builds fill /var/lib/docker.

    Check

    df -h /var/lib/docker   # 100% used
    docker system df   # large RECLAIMABLE for images and build cache

    Fix

    docker builder prune -f (and docker buildx prune for other builders), docker image prune -f.

  2. Container logs with no size limit grow until the disk is full.

    Check

    sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -3   # a log of many GB

    Fix

    See the logs-disk entry.

  3. Volumes that are no longer needed, or an app that filled its own volume.

    Check

    docker system df -v   # the large volumes, and whether a container uses them

    Fix

    Look at volumes one by one; never prune them blindly.

  4. Out of inodes, not bytes: many small files (build cache, node_modules) use up the slots for files.

    Check

    df -i /var/lib/docker   # IUse% 100% while df -h still shows space

    Fix

    Prune build cache and images; if it keeps happening, move /var/lib/docker to a file system with more inodes.

  5. Docker Desktop: its VM's virtual disk reached the size set in its settings, while the Mac or PC still has space.

    Check

    docker info --format '{{.OperatingSystem}}'   # Docker Desktop
    docker system df   # the space Docker uses inside the VM

    Fix

    Prune inside it, or raise the disk size in Docker Desktop's settings.

Prevent it

Schedule docker builder prune on build hosts, rotate logs, and alert on disk and inode use of /var/lib/docker.

Now learn it: break and fix it yourself in No space left on device · Logs fill the disk, then take Level 13 · Logs and disk space. Open in DevOps Nimbus Docker.

Container logs filling the disk

The error

/var/lib/docker/containers/<id>/<id>-json.log   14G

What it means

One container's log file has grown to many gigabytes. The default json-file driver keeps every line forever unless it is told to rotate.

Causes, most likely first

  1. No max-size or max-file on the container or in daemon.json.

    Check

    docker inspect <name> --format '{{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}}'   # json-file with {}: this container does not rotate

    Fix

    Re-create with --log-opt max-size=10m --log-opt max-file=3 (or --log-driver local).

  2. log-opts were added to daemon.json later. They apply only to containers created after the restart, so older ones still grow.

    Check

    docker inspect <name> --format '{{.Created}}'   # created before daemon.json changed

    Fix

    Re-create the old containers so they pick up the defaults.

  3. A chatty app logs every request, so even rotated logs fill the disk quickly.

    Check

    docker logs --since 1m <name> | wc -l   # thousands of lines a minute

    Fix

    Lower the app's log level. If the disk is full right now, sudo truncate -s 0 <log path> frees the space in place; rm does not, because dockerd keeps the file open.

Prevent it

Set log-opts in /etc/docker/daemon.json so every new container rotates.

Now learn it: break and fix it yourself in Logs fill the disk, then take Level 13 · Logs and disk space. Open in DevOps Nimbus Docker.