DevOps Nimbus Ansible

Ansible errors and how to fix them

68 errors and symptoms Ansible users meet, each with the exact message, what it means, every cause that produces it with a check that proves it and the fix for that cause, and how to prevent it. Then learn it for good: break and fix each one yourself in the browser playground. Generated from the same data as the interactive reference.

UNREACHABLE: Permission denied (publickey)

Error

fatal: [web-04]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: root@10.0.1.14: Permission denied (publickey).", "unreachable": true}

What it means

Ansible reached the host's SSH server, but the server accepted none of the keys offered for that user. root@10.0.1.14 is the user Ansible logged in as and the address it used; (publickey) means key login is the only method the server allows, so no password was tried.

Causes, most likely first

  1. Ansible logs in as the wrong user. An ansible_user in the inventory beats -u on the command line, so a stale value wins and that user has no copy of your key.

    Check
    ansible-inventory --host <host>   # ansible_user: does it match the user in the message?
    Fix

    Correct ansible_user in the inventory, group_vars or host_vars; set it once per group rather than per host.

  2. The right user, but the public key is not in that user's ~/.ssh/authorized_keys on the host, so the server rejects every key offered.

    Check
    ssh -i ~/.ssh/<key> -o IdentitiesOnly=yes <user>@<address>   # also denied: the key is not installed for that user
    Fix

    Install the public key for that user (ssh-copy-id, or ansible.posix.authorized_key run from a login that still works).

  3. Ansible offers a different key than you think. ansible_ssh_private_key_file, ~/.ssh/config and ssh-agent all add keys, and a key file that is missing or passphrase-protected without an agent is silently skipped.

    Check
    ansible <host> -m ansible.builtin.ping -vvvv   # the 'Offering public key' lines name each key tried; yours is missing or never offered
    ssh-add -l   # the agent's key fingerprints (no private key is printed)
    Fix

    Point ansible_ssh_private_key_file at the right key for that group, load it into the agent, and add -o IdentitiesOnly=yes to ansible_ssh_common_args when the agent holds many keys.

  4. The key is installed but sshd refuses to use it: the user's ~/.ssh or authorized_keys has the wrong owner or is writable by others (StrictModes), or it is a DSA key, which OpenSSH 7.0 and later reject by default.

    Check
    sudo journalctl -u ssh -n 20   # on the host (sshd on RHEL): 'Authentication refused: bad ownership or modes' means permissions
    Fix

    On the host: make the user own ~/.ssh, then chmod 700 ~/.ssh and chmod 600 ~/.ssh/authorized_keys; replace a DSA key with an ed25519 key.

Prevent it

Now learn it: break and fix it yourself in Permission denied (publickey), then take Level 02 · Getting in. Interactive version

UNREACHABLE: Connection timed out

Error

fatal: [db-02]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.0.2.21 port 22: Connection timed out", "unreachable": true}

What it means

Ansible's SSH client sent a connection request to 10.0.2.21 port 22 and nothing answered before the timeout. No SSH conversation started, so users, keys and sudo are not involved yet: this is an address, network or firewall problem.

Causes, most likely first

  1. The address is wrong (a typo or an old IP), so the packets go to a machine that is not there.

    Check
    ansible-inventory --host <host>   # ansible_host: is it the host's current address?
    Fix

    Correct ansible_host, or use a dynamic inventory plugin so cloud addresses stay current.

  2. A firewall or security group drops port 22 from the control node. A dropped packet gets no reply, so SSH waits until it times out.

    Check
    nc -vz -w 5 <address> 22   # times out: dropped on the way; 'refused': the host answered, see Connection refused
    Fix

    Allow port 22 from the control node's address only (security group or host firewall), not from 0.0.0.0/0.

  3. SSH listens on another port, and nothing answers on 22.

    Check
    ansible-inventory --host <host>   # no ansible_port, but the host uses a custom port
    Fix

    Set ansible_port for that host or group in the inventory.

  4. The host is only reachable through a bastion. Your own ~/.ssh/config may have a ProxyJump that the CI runner does not.

    Check
    ansible <host> -m ansible.builtin.ping -vvvv   # the ssh command line: no ProxyJump or ProxyCommand where one is needed
    Fix

    Set ansible_ssh_common_args: '-o ProxyJump=<user>@<bastion>' for that group in the inventory.

  5. The host is down or has no route from here.

    Check
    ssh -v <user>@<address>   # stops at 'Connecting to': nothing answered at all
    Fix

    Start the host or fix the route; then ansible.builtin.wait_for_connection before the next play.

Prevent it

Now learn it: break and fix it yourself in Connection timed out, then take Level 01 · Know your fleet. Interactive version

UNREACHABLE: Connection refused

Error

fatal: [web-06]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.0.1.16 port 22: Connection refused", "unreachable": true}

What it means

The host answered, but with a refusal: nothing accepts connections on that port. Unlike a timeout, the network path works; the problem is on the host or in the port number.

Causes, most likely first

  1. sshd is stopped, not installed, or still starting after a reboot, so the kernel refuses the connection.

    Check
    sudo systemctl status ssh   # on the host, from its console (sshd on RHEL): inactive or failed
    Fix

    Start it: sudo systemctl enable --now ssh (sshd on RHEL), or wait for it after a reboot with ansible.builtin.wait_for_connection.

  2. SSH listens on another port and ansible_port is not set, so Ansible knocks on 22.

    Check
    ansible-inventory --host <host>   # ansible_port missing or wrong
    nc -vz <address> <port>   # 'succeeded' on the real port
    Fix

    Set ansible_port with the host in the inventory.

  3. A host firewall rejects (rather than drops) the connection, which also shows as refused.

    Check
    sudo firewall-cmd --list-services   # on the host (RHEL family): ssh missing; 'sudo ufw status' on Ubuntu
    Fix

    Allow SSH from the control node's address in the host firewall.

  4. ansible_host points at another machine that does not run SSH.

    Check
    ansible-inventory --host <host>   # ansible_host: is it the address you expect?
    Fix

    Correct ansible_host.

Prevent it

Now learn it: break and fix it yourself in Connection timed out, then take Level 01 · Know your fleet. Interactive version

REMOTE HOST IDENTIFICATION HAS CHANGED

Error

fatal: [web-03]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @\r\n... Host key verification failed.", "unreachable": true}

What it means

The host sent an SSH host key that differs from the one stored for its address in known_hosts. SSH stops because it cannot tell a rebuilt host from an attacker in the middle; it never got as far as logging in.

Causes, most likely first

  1. The host was rebuilt or reinstalled and generated a new host key. This is the usual reason, but confirm it before trusting the new key.

    Check
    ssh-keygen -F <address>   # the stored key (grep can miss it: Debian and Ubuntu hash known_hosts)
    ssh-keyscan <address> 2>/dev/null | ssh-keygen -lf -   # the fingerprint offered now: matches the console or cloud boot log means a genuine rebuild
    Fix

    After the fingerprint matches: ssh-keygen -R <address> (and -R <name> if stored by name), then add the new key with ssh-keyscan <address> >> ~/.ssh/known_hosts.

  2. A different machine now has that address (a reused IP, or a load balancer address shared by several hosts).

    Check
    ansible-inventory --host <host>   # ansible_host: is this still the right machine?
    Fix

    Point the inventory at the right address; for shared addresses, connect to each host's own address.

  3. Someone is intercepting the connection. If the fingerprint does not match what the console shows, assume this.

    Check
    ssh-keyscan <address> 2>/dev/null | ssh-keygen -lf -   # does not match the console: do not connect
    Fix

    Do not replace the key; stop and investigate the network path.

Prevent it

Now learn it: break and fix it yourself in Host key changed after a rebuild, then take Level 02 · Getting in. Interactive version

Host key verification failed (new host)

Error

fatal: [web-05]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Host key verification failed.", "unreachable": true}

What it means

SSH has no stored key for this host and would normally ask you to confirm the fingerprint. Ansible cannot answer that prompt, so SSH gives up. Nothing is wrong with the host; it is unknown to this control node.

Causes, most likely first

  1. The host is new, or the CI runner starts with an empty known_hosts.

    Check
    ssh-keygen -F <address>   # no output: SSH has never stored this host's key
    Fix

    Compare ssh-keyscan <address> 2>/dev/null | ssh-keygen -lf - with the fingerprint from the host's console, then append ssh-keyscan <address> to ~/.ssh/known_hosts.

  2. The key is stored under another name. SSH looks up the address it connects to (ansible_host), not the inventory name.

    Check
    ssh-keygen -F <name>   # found by name but not by address
    Fix

    Store the key under the address Ansible uses, or under both.

  3. Ansible runs as another user (sudo ansible-playbook, a CI service account), so it reads a different ~/.ssh/known_hosts.

    Check
    whoami; echo ~   # the user and home directory Ansible runs with
    Fix

    Run Ansible as the user whose known_hosts holds the keys, or write that user's file from a reviewed list.

Prevent it

Now learn it: break and fix it yourself in Host key changed after a rebuild, then take Level 02 · Getting in. Interactive version

Could not resolve hostname

Error

fatal: [db-03]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname db-03: Name or service not known", "unreachable": true}

What it means

SSH tried to turn the name db-03 into an address and DNS (or /etc/hosts) had no answer. No connection was attempted.

Causes, most likely first

  1. The host has no ansible_host, so SSH uses the inventory name, and that name is not in DNS.

    Check
    ansible-inventory --host <host>   # no ansible_host
    Fix

    Set ansible_host to the address, or add the name to DNS.

  2. A typo in ansible_host or in the inventory name.

    Check
    getent hosts <name>   # the exact name from the message; no output means it does not resolve here
    Fix

    Correct the name.

  3. The name resolves only on another network or machine: a VPN, a private DNS zone, or an entry in your /etc/hosts or ~/.ssh/config that the CI runner does not have.

    Check
    ssh -G <name> | grep -i '^hostname'   # what ~/.ssh/config turns the name into on this machine
    Fix

    Put the address in the inventory, or give the runner the same DNS (VPN, resolver).

Prevent it

Now learn it: break and fix it yourself in Connection timed out, then take Level 01 · Know your fleet. Interactive version

Missing sudo password

Error

fatal: [web-03]: FAILED! => {"msg": "Missing sudo password"}

What it means

The task uses become, sudo on the host asked for a password, and Ansible had none to give. The SSH login itself worked.

Causes, most likely first

  1. No become password was supplied: neither -K nor ansible_become_password for this host. A vaulted value in a group the host is not in counts as none.

    Check
    ansible <host> -m ansible.builtin.debug -a 'msg={{ ansible_become_password is defined }}'   # False: no password reaches this host (the value is not printed)
    Fix

    Run with -K (--ask-become-pass), or set ansible_become_password in Vault for the group the host is in.

  2. sudo on that host requires a password for ansible_user, where you expected NOPASSWD.

    Check
    ssh <user>@<address> sudo -n true   # 'a password is required': sudo wants a password for this user
    Fix

    Give the password as above, or add a reviewed NOPASSWD rule for a dedicated automation user.

  3. A NOPASSWD rule exists but lists specific commands. Ansible runs /bin/sh -c ... through sudo, so a command-limited rule never matches and sudo asks for a password.

    Check
    ssh -t <user>@<address> sudo -l   # NOPASSWD for some commands only, not ALL
    Fix

    Use a become password, or a rule Ansible can use for the automation user; sudo command lists do not work with Ansible's wrapper.

Prevent it

Now learn it: break and fix it yourself in Missing sudo password, then take Level 02 · Getting in. Interactive version

Incorrect sudo password

Error

fatal: [web-03]: FAILED! => {"msg": "Incorrect sudo password"}

What it means

Ansible sent a password to sudo and sudo rejected it. The login worked; the password, or the account sudo checks it against, is wrong.

Causes, most likely first

  1. The password typed at -K, or in ansible_become_password, is wrong for this host.

    Check
    ssh -t <user>@<address> sudo -v   # type the same password by hand: rejected here too means the password is wrong
    Fix

    Use the right password; store it per group in Vault when hosts differ.

  2. Hosts have different passwords, and a value meant for one group reaches this host through variable precedence.

    Check
    ansible <host> -m ansible.builtin.debug -a 'msg={{ ansible_become_password | length }}'   # the length of the password this host gets, not the password
    Fix

    Set ansible_become_password in the group_vars of the group that owns that password.

  3. sudo is configured to ask for root's or the target user's password (rootpw, targetpw), not the login user's.

    Check
    sudo grep -rE 'rootpw|targetpw' /etc/sudoers /etc/sudoers.d   # on the host, as an admin: a match explains it
    Fix

    Give the password sudo really asks for, or change that sudoers setting for the automation user.

  4. The account is locked or its password expired, often after repeated failures across many hosts (pam_faillock).

    Check
    sudo faillock --user <user>   # on the host, as an admin: recent failures mean a lock
    sudo chage -l <user>   # 'Password expires' in the past
    Fix

    Fix the password first, then unlock with faillock --user <user> --reset.

Prevent it

Now learn it: break and fix it yourself in Missing sudo password, then take Level 02 · Getting in. Interactive version

Timeout waiting for privilege escalation prompt

Error

fatal: [db-01]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}

What it means

Ansible started sudo (or another become method) and waited for either its password prompt or the success marker; neither came before the timeout. 12s is the connection timeout (10s by default) plus 2 seconds.

Causes, most likely first

  1. sudo asks for a password Ansible does not send, or prints a prompt Ansible does not recognise, so Ansible keeps waiting.

    Check
    ansible <host> -m ansible.builtin.command -a whoami --become -vvvv   # the sudo command and what came back before the timeout
    ssh <user>@<address> 'sudo -n true'   # 'a password is required': sudo wants a password
    Fix

    Give the password with -K or ansible_become_password from Vault, or allow NOPASSWD for the automation user.

  2. sudo is slow to start because the host cannot resolve its own hostname, so it waits on DNS.

    Check
    ssh <user>@<address> 'getent hosts $(hostname)'   # no output: the host cannot resolve its own name
    Fix

    Add the host's own name to /etc/hosts.

  3. The host is overloaded and answers after the timeout.

    Check
    ssh <user>@<address> uptime   # a very high load average
    Fix

    Raise the wait: timeout = 30 in the [defaults] section of ansible.cfg (or ANSIBLE_TIMEOUT=30).

  4. The become method or user is wrong for this host (su where sudo is used, or a user that does not exist).

    Check
    ansible-inventory --host <host>   # ansible_become_method and ansible_become_user
    Fix

    Set the method and user that host really uses.

Prevent it

Now learn it: break and fix it yourself in Missing sudo password, then take Level 02 · Getting in. Interactive version

you must install the sshpass program

Error

fatal: [web-07]: FAILED! => {"msg": "to use the 'ssh' connection type with passwords or pkcs11_provider, you must install the sshpass program"}

What it means

Ansible was asked to log in with an SSH password. Up to ansible-core 2.18 it does that with the sshpass program, which is not installed on the control node. From 2.19 the default is password_mechanism = ssh_askpass, which needs no sshpass; there the message reads to use the password_mechanism=sshpass, you must install the sshpass program and appears only when sshpass was chosen.

Causes, most likely first

  1. A password login was requested on purpose, and sshpass is missing (ansible-core 2.18 or earlier, or password_mechanism = sshpass).

    Check
    command -v sshpass   # on the control node: no output means not installed
    ansible --version   # the ansible-core version decides whether sshpass is needed
    Fix

    Better: log in with an SSH key. Otherwise install sshpass on the control node (sudo apt install sshpass, or sudo dnf install sshpass; on RHEL it comes from AppStream or EPEL depending on the release), or on 2.19+ remove the password_mechanism = sshpass setting.

  2. ansible_password (or ansible_ssh_pass) is set somewhere you did not expect, such as group_vars, so Ansible tries a password login.

    Check
    grep -rlE 'ansible_password|ansible_ssh_pass' inventory group_vars host_vars   # file names only, so no password is printed
    Fix

    Remove the password variable and use a key, or move it into Vault if you really need it.

Prevent it

Now learn it: break and fix it yourself in Permission denied (publickey), then take Level 02 · Getting in. Interactive version

Failed to set permissions on the temporary files

Error

fatal: [db-01]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chmod: invalid mode: 'A+user:postgres:rx:allow'\n...}). For information on working around this, see https://docs.ansible.com/ansible-core/2.18/playbook_guide/playbooks_privilege_escalation.html#risks-of-becoming-an-unprivileged-user"}

What it means

Ansible copied the module to a temporary file owned by the login user and needed to let the become user (here postgres) read it. It tries setfacl first, then fallbacks such as chmod +a (the macOS form, which is the A+user:... error you see), and every method failed.

Causes, most likely first

  1. The host has no setfacl (the acl package), so the first and safest method fails and the fallbacks do too.

    Check
    ansible <host> -m ansible.builtin.command -a 'which setfacl'   # rc 1 and no path: acl is not installed
    Fix

    Install the acl package on the host (for example in the base role).

  2. Pipelining is off, so Ansible needs a temporary module file at all. Copy and template still transfer files even with pipelining.

    Check
    ansible-config dump | grep -i pipelining   # ANSIBLE_PIPELINING(default) = False
    Fix

    Set pipelining = True under [ssh_connection] (needs sudo without requiretty), and still install acl for copy and template.

Prevent it

Now learn it: break and fix it yourself in Becoming an unprivileged user fails, then take Level 02 · Getting in. Interactive version

MODULE FAILURE: sudo parse error in /etc/sudoers.d

Error

fatal: [web-01]: FAILED! => {"changed": false, "module_stderr": ">>> /etc/sudoers.d/monitoring: syntax error near line 1 <<<\nsudo: parse error in /etc/sudoers.d/monitoring near line 1\nsudo: no valid sudoers sources found, quitting\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

What it means

sudo found a syntax error in /etc/sudoers.d/monitoring and refuses to run at all, for every user. Ansible's become uses sudo, so every become task on that host fails until the file is repaired.

Cause

  1. A file with a syntax error was written to /etc/sudoers.d without validation.

    Check
    ssh <user>@<address> sudo -n true   # the same parse error
    visudo -c   # from the console or a root session: names the file and line
    Fix

    From the console or a root session, repair it with visudo -f /etc/sudoers.d/<file>, or remove the file.

Prevent it

Now learn it: break and fix it yourself in A sudoers typo locks everyone out, then take Level 10 · Keeping secrets. Interactive version

Could not match supplied host pattern / skipping: no hosts matched

Error

[WARNING]: Could not match supplied host pattern, ignoring: webservers

PLAY [Message of the day] ****
skipping: no hosts matched

What it means

The play's hosts: pattern (here webservers) matched no host in the inventory Ansible loaded, so the play was skipped. A warning, not an error: the run ends green although nothing ran.

Causes, most likely first

  1. The play names a group that does not exist, or with different case (group names are case-sensitive).

    Check
    ansible-inventory --graph   # the real group names
    Fix

    Use the real group name in hosts:, or add the group to the inventory.

  2. Ansible used another inventory than you meant (-i, or the inventory setting of another ansible.cfg).

    Check
    ansible --version   # config file = the ansible.cfg in use
    ansible-playbook <playbook> --list-hosts   # the hosts each play would target
    Fix

    Pass the right -i, or run from the project directory so its ansible.cfg is used.

  3. The group lives in an inventory source that failed to parse, so it is missing.

    Check
    ansible-inventory --graph 2>&1 | grep -i 'unable to parse'   # an earlier warning about that source
    Fix

    Fix that inventory source (see Unable to parse ... as an inventory source).

Prevent it

Now learn it: break and fix it yourself in Play matched no hosts, then take Level 01 · Know your fleet. Interactive version

--limit leaves us with no hosts to target

Error

[WARNING]: Could not match supplied host pattern, ignoring: web01
ERROR! Specified inventory, host pattern and/or --limit leaves us with no hosts to target.

What it means

After applying --limit, the set of hosts was empty, so ansible-playbook stopped before running anything. The warning above it names the part of the limit that matched nothing.

Causes, most likely first

  1. A typo in the host or group given to --limit.

    Check
    ansible-inventory --graph   # the names that exist
    Fix

    Use a name from the inventory; preview with --list-hosts.

  2. --limit matches inventory names, not addresses, so limiting by an IP fails unless the IP is the inventory name.

    Check
    ansible-inventory --host <name>   # works only for an inventory name
    Fix

    Limit by the inventory name or a group.

  3. The limit and the play's hosts: do not overlap: the host exists but is not in the play's group.

    Check
    ansible-playbook <playbook> --limit <host> --list-hosts   # the play's host list is empty
    Fix

    Limit to a host in the play's group, or fix the host's group membership.

Prevent it

Now learn it: break and fix it yourself in --limit typo runs nothing, then take Level 11 · Change only what you mean. Interactive version

Unable to parse ... as an inventory source

Error

[WARNING]: Unable to parse /home/ops/project/inventory/hostz.ini as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available

What it means

Ansible tried every enabled inventory plugin on the file and none could read it, so it fell back to the implicit localhost only. Every host pattern then matches nothing.

Causes, most likely first

  1. The path given with -i, or the inventory setting in ansible.cfg, does not exist.

    Check
    ls -l <inventory path>   # 'No such file or directory'
    Fix

    Point -i or ansible.cfg's inventory at the real file.

  2. Ansible read a different ansible.cfg than you expected, with another inventory path.

    Check
    ansible --version   # config file = which ansible.cfg was read
    Fix

    Run from the project directory, or set ANSIBLE_CONFIG to the project's ansible.cfg.

  3. The file exists but no plugin accepts it: a YAML syntax error, YAML content in an .ini file, or a plugin config (such as aws_ec2.yml) whose collection is missing or whose name does not end the way the plugin requires.

    Check
    ansible-inventory -i <path> --graph -vvv   # each plugin's reason for rejecting the file
    Fix

    Fix the syntax, or install the plugin's collection and name the file as its documentation requires (for example ending in aws_ec2.yml).

Prevent it

Now learn it: break and fix it yourself in Play matched no hosts, then take Level 01 · Know your fleet. Interactive version

Ansible is being run in a world writable directory

Error

[WARNING]: Ansible is being run in a world writable directory (/mnt/project), ignoring it as an ansible.cfg source. For more information see https://docs.ansible.com/ansible/devel/reference_appendices/config.html#cfg-in-world-writable-dir

What it means

Ansible refuses to load an ansible.cfg from a directory every user can write to, because anyone could plant settings there. It falls back to ~/.ansible.cfg or /etc/ansible/ansible.cfg, so the project's inventory, roles path or remote user are silently not used.

Cause

  1. The project sits on a shared mount where every directory is world writable (Vagrant shares, WSL's /mnt/c, some network file systems).

    Check
    ls -ld .   # drwxrwxrwx means world writable
    ansible --version   # config file = shows which ansible.cfg is really in use
    Fix

    Remove world write with chmod o-w ., or move the project into your home directory on a Linux file system. If neither is possible, name the file explicitly: export ANSIBLE_CONFIG=$PWD/ansible.cfg.

Prevent it

Now learn it: break and fix it yourself in Play matched no hosts, then take Level 01 · Know your fleet. Interactive version

A task did not run with --tags

Error

TASK [Run database migrations] does not appear in the output of ansible-playbook deploy.yml --tags deploy

What it means

With --tags, Ansible runs only tasks that carry one of those tags (or always) and skips the rest without a skipping line. A task that does not appear at all was filtered out by tags.

Causes, most likely first

  1. The task has no tag, so --tags deploy leaves it out silently.

    Check
    ansible-playbook <playbook> --tags <tag> --list-tasks   # the task is not listed
    Fix

    Tag the task, or tag it always when every run needs it.

  2. Tags on include_tasks apply to the include statement only, not to the tasks inside it.

    Check
    ansible-playbook <playbook> --list-tags   # the included tasks have no tags of their own
    Fix

    Use import_tasks, or apply: { tags: [...] } on the include_tasks.

  3. The task is tagged never, which runs only when that tag is asked for by name.

    Check
    grep -rn -B3 'never' roles <playbook>   # tags: [never, ...] on the task
    Fix

    Ask for its tag explicitly, or remove never.

Prevent it

Now learn it: break and fix it yourself in --tags skips a critical task, then take Level 11 · Change only what you mean. Interactive version

mapping values are not allowed in this context

Error

ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
JSON: Expecting value: line 1 column 1 (char 0)

Syntax Error while loading YAML.
  mapping values are not allowed in this context

What it means

The YAML parser found a key: value where it expected a plain value, so it cannot tell which mapping the line belongs to. The line and column it names are where it gave up; the mistake is often on the line above. On ansible-core 2.19 and later it reads YAML parsing failed: Mapping values are not allowed in this context.

Causes, most likely first

  1. A line is indented deeper than its sibling keys, so YAML reads it as part of the previous value.

    Check
    ansible-playbook <playbook> --syntax-check   # the line and column
    cat -n <playbook>   # compare that line's indent with the line above
    Fix

    Line up the module with name: of its task, and its options two spaces deeper.

  2. A plain value contains : (colon and space), which YAML reads as a new key.

    Check
    grep -n ': .*: ' <playbook>   # values with a second ': '
    Fix

    Quote the value: msg: "Status: ok".

Prevent it

Now learn it: break and fix it yourself in YAML indentation error, then take Level 04 · Your first playbook. Interactive version

found unacceptable key (unhashable type: 'AnsibleMapping')

Error

Syntax Error while loading YAML.
  found unacceptable key (unhashable type: 'AnsibleMapping')
...
We could be wrong, but this one looks like it might be an issue with
missing quotes. Always quote template expression brackets when they
start a value.

What it means

A value starts with {{ without quotes, so YAML reads { ... } as an inline dictionary and the inner { x } as a dictionary used as a key, which is not allowed. On ansible-core 2.19 and later it reads This may be an issue with missing quotes around a template block.

Cause

  1. A value that begins with {{ is not quoted.

    Check
    ansible-playbook <playbook> --syntax-check   # the line and column
    grep -nE ':\s+\{\{' <file>   # unquoted values starting with {{
    Fix

    Quote the whole value: dest: "{{ app_dir }}/releases".

Prevent it

Now learn it: break and fix it yourself in Unquoted Jinja breaks YAML, then take Level 06 · Files that fit each host. Interactive version

did not find expected key

Error

Syntax Error while loading YAML.
  did not find expected key

What it means

Inside a mapping (or a list) the parser reached a line at an indentation it did not expect. On ansible-core 2.19 and later a list version reads did not find expected '-' indicator.

Causes, most likely first

  1. A line is indented by an odd amount, between two levels.

    Check
    cat -n <file>   # the named line does not line up with its siblings
    ansible-playbook <playbook> --syntax-check
    Fix

    Make the line's indentation match its siblings.

  2. A list item (-) where a key was expected, or a key where a list item was expected.

    Check
    cat -n <file>   # a '-' at a level that holds keys, or the reverse
    Fix

    Make each level all keys or all list items.

Prevent it

Now learn it: break and fix it yourself in YAML indentation error, then take Level 04 · Your first playbook. Interactive version

found character that cannot start any token

Error

Syntax Error while loading YAML.
  found character that cannot start any token

What it means

The YAML parser met a character that no YAML token can start with. On ansible-core 2.19 and later a tab gives YAML parsing failed: Tabs are usually invalid in YAML.

Causes, most likely first

  1. A tab character in indentation. YAML allows only spaces there.

    Check
    grep -nP '\t' <file>   # every line with a tab
    Fix

    Replace the tabs with spaces.

  2. A plain value starts with @ or a backtick, which YAML reserves.

    Check
    grep -nE ':\s+[@`]' <file>   # values starting with @ or `
    Fix

    Quote the value: user: "@ops".

Prevent it

Now learn it: break and fix it yourself in YAML indentation error, then take Level 04 · Your first playbook. Interactive version

'x' is not a valid attribute for a Play

Error

ERROR! 'when' is not a valid attribute for a Play

What it means

Ansible read a keyword it does not accept at that level: here when directly on a play. The name in quotes is the keyword, and the last word says where it was found (Play, Task, Block, Role).

Causes, most likely first

  1. A task keyword (when, loop, register) used at play level.

    Check
    ansible-playbook <playbook> --syntax-check   # the line it points at
    Fix

    Move the keyword to a task or block.

  2. A tasks file (such as roles/web/tasks/main.yml) run as a playbook. Each task is read as a play, so the module name is reported: 'debug' is not a valid attribute for a Play.

    Check
    head -5 <file>   # a list of tasks with no hosts:
    Fix

    Run the playbook that uses the role, or wrap the tasks in a play with hosts: and tasks:.

  3. A play-level - include: other.yml, removed in ansible-core 2.16, gives 'include' is not a valid attribute for a Play.

    Check
    grep -n '^- include:' <playbook>
    Fix

    Use - import_playbook: other.yml.

  4. A misspelt keyword (become_users, gather_fact) or a task indented out of its tasks: list.

    Check
    cat -n <playbook>   # the keyword's spelling and indent
    Fix

    Fix the spelling or indentation.

Prevent it

Now learn it: break and fix it yourself in YAML indentation error, then take Level 04 · Your first playbook. Interactive version

the field 'hosts' is required but was not set

Error

ERROR! the field 'hosts' is required but was not set

What it means

Ansible read an item as a play (it has play keywords such as name:, vars: or tasks:), but it has no hosts:, so Ansible does not know where to run it.

Causes, most likely first

  1. The play has no hosts: line, often lost when copying or splitting a playbook.

    Check
    head -5 <file>   # a play with name: and tasks: but no hosts:
    Fix

    Add hosts: <group> to the play.

  2. hosts: is indented under another key, such as vars:, so it is not a key of the play.

    Check
    cat -n <file>   # hosts: deeper than name: and tasks:
    Fix

    Indent hosts: at the same level as the play's name:.

Prevent it

Now learn it: break and fix it yourself in YAML indentation error, then take Level 04 · Your first playbook. Interactive version

conflicting action statements

Error

ERROR! conflicting action statements: ansible.builtin.apt, ansible.builtin.service

What it means

One task has two keys that Ansible reads as modules, so it cannot tell which to run. The two names follow the colon.

Causes, most likely first

  1. A missing - name: (or -) between two tasks merged them into one.

    Check
    cat -n <file>   # two modules under one dash
    Fix

    Split them into two tasks, each starting with - name:.

  2. A module's option is indented at the module's level, so Ansible reads the option as a second module: conflicting action statements: debug, msg.

    Check
    cat -n <file>   # the second name is an option of the first module
    Fix

    Indent the options two spaces under the module.

Prevent it

Now learn it: break and fix it yourself in YAML indentation error, then take Level 04 · Your first playbook. Interactive version

no module/action detected in task

Error

ERROR! no module/action detected in task.

What it means

A task has keywords such as name: but no module, so there is nothing to run.

Cause

  1. The module line is missing, or indented out of the task so it belongs to something else.

    Check
    cat -n <file>   # the task's keys: is there a module at the same level as name:?
    Fix

    Add the module at the same indentation as name:.

Prevent it

Now learn it: break and fix it yourself in YAML indentation error, then take Level 04 · Your first playbook. Interactive version

couldn't resolve module/action

Error

ERROR! couldn't resolve module/action 'community.general.ufw'. This often indicates a misspelling, missing collection, or incorrect module path.

What it means

While loading the playbook, Ansible looked for a module or action named community.general.ufw in every collection path and found none. Nothing has run yet.

Causes, most likely first

  1. The collection that provides it is not installed on this control node (common on a fresh CI runner).

    Check
    ansible-galaxy collection list community.general   # not listed
    Fix

    Add the collection to requirements.yml and run ansible-galaxy collection install -r requirements.yml.

  2. The collection is installed, but where this ansible-core does not look: another virtualenv, another user's home, or a collections_path set in an ansible.cfg that was not loaded.

    Check
    ansible --version   # which Python and config file this ansible uses
    ansible-config dump | grep -i collections_path   # the paths really searched
    Fix

    Install into one of the searched paths, or set collections_path in the project's ansible.cfg.

  3. A typo, or a short name (ufw) for a module in a collection that is not installed.

    Check
    ansible-doc <module>   # 'module not found'
    Fix

    Use the full name from the collection's documentation.

  4. The module moved to another collection in a newer release (for example the Docker modules moved from community.general to community.docker).

    Check
    ansible-doc -l | grep <module>   # the module under another collection name
    Fix

    Use the new full name and install that collection.

Prevent it

Now learn it: break and fix it yourself in couldn't resolve module/action, then take Level 09 · Other people's code. Interactive version

Unsupported parameters for module

Error

fatal: [web-01]: FAILED! => {"changed": false, "msg": "Unsupported parameters for (ansible.builtin.apt) module: package_name. Supported parameters include: allow_change_held_packages, ... upgrade (default-release, install-recommends, package, pkg, update-cache)."}

What it means

The module received an option it does not have (here package_name). The message lists every option this installed version accepts; names in brackets are aliases.

Causes, most likely first

  1. A misspelt or invented option.

    Check
    ansible-doc <module>   # the options this version accepts
    Fix

    Use an option name ansible-doc lists.

  2. An option from another module, or from a newer collection version than the one installed. The docs you read online may be for a newer release.

    Check
    ansible-galaxy collection list <namespace.collection>   # the installed version
    ansible-doc <module>   # describes the installed version
    Fix

    Upgrade the collection in requirements.yml, or use the option the installed version has.

Prevent it

Now learn it: break and fix it yourself in Unsupported parameters, then take Level 03 · One command, many hosts. Interactive version

MODULE FAILURE / python3: not found

Error

"module_stdout": "/bin/sh: 1: /usr/bin/python3: not found\r\n", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127

What it means

Ansible ran its module with the Python interpreter it chose for the host, and the shell found no program at that path (rc 127). Most modules need Python on the host.

Causes, most likely first

  1. The host has no Python at all (minimal and container images).

    Check
    ansible <host> -m ansible.builtin.raw -a 'command -v python3 || echo none'   # 'none': no Python
    Fix

    Install it with raw: ansible <host> -m ansible.builtin.raw -a 'apt-get update && apt-get install -y python3' -b (dnf install -y python3 on the RHEL family).

  2. ansible_python_interpreter (or interpreter_python in ansible.cfg) pins a path that does not exist on this host, such as /usr/bin/python where only python3 exists.

    Check
    ansible-inventory --host <host> | grep interpreter   # a pinned path
    ansible-config dump | grep -i interpreter_python
    Fix

    Correct the path, or remove the pin and let discovery find Python.

Prevent it

Now learn it: break and fix it yourself in No Python on the new host, then take Level 03 · One command, many hosts. Interactive version

Failed to import the required Python library

Error

fatal: [db-01]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (psycopg2) on db-01's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

What it means

The module itself ran, but it imports a Python library (here psycopg2) that the named interpreter on the named host does not have. The host and Python path in the message tell you exactly where it is missing.

Causes, most likely first

  1. The library is not installed on the host the module runs on.

    Check
    ssh <user>@<address> "/usr/bin/python3 -c 'import psycopg2'"   # ModuleNotFoundError: not installed for that Python
    Fix

    Install it in a task before the module runs, for example ansible.builtin.package: name=python3-psycopg2.

  2. It is installed for another Python than the one Ansible uses on that host.

    Check
    ansible <host> -m ansible.builtin.setup -a 'filter=ansible_python'   # the executable Ansible uses
    Fix

    Install it for that Python, or set ansible_python_interpreter to the Python that has it.

  3. The module runs on localhost (most cloud modules). If localhost is listed in the inventory, Ansible discovers /usr/bin/python3 for it instead of the virtualenv ansible-playbook runs in.

    Check
    ansible localhost -m ansible.builtin.setup -a 'filter=ansible_python'   # not your virtualenv's Python
    Fix

    Set ansible_python_interpreter: "{{ ansible_playbook_python }}" for localhost, and install the library in that virtualenv.

Prevent it

Now learn it: break and fix it yourself in No Python on the new host, then take Level 03 · One command, many hosts. Interactive version

Using the discovered Python interpreter (warning)

Error

[WARNING]: Platform linux on host web-02 is using the discovered Python interpreter at /usr/bin/python3.10, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.17/reference_appendices/interpreter_discovery.html for more information.

What it means

No interpreter was set for the host, so Ansible found one by itself and warns that installing another Python later could change its choice. The play works. On ansible-core 2.19 and later the wording is Host 'web-02' is using the discovered Python interpreter at ....

Cause

  1. ansible_python_interpreter is not set and interpreter_python is auto, which warns.

    Check
    ansible <host> -m ansible.builtin.setup -a 'filter=ansible_python'   # the Python it found
    Fix

    Set ansible_python_interpreter: /usr/bin/python3 in group_vars, or keep discovery without the warning: interpreter_python = auto_silent in the [defaults] section of ansible.cfg.

Prevent it

Now learn it: break and fix it yourself in No Python on the new host, then take Level 03 · One command, many hosts. Interactive version

No package matching ... is available

Error

fatal: [web-02]: FAILED! => {"changed": false, "failures": ["No package apache2 available."], "msg": "Failed to install some of the specified packages", "rc": 1, "results": []}

What it means

The package manager on the host searched its enabled repositories and found no package by that name. No package apache2 available. is the dnf/yum form; apt says No package matching 'apache2' is available.

Causes, most likely first

  1. Package names differ between OS families (apache2 on Debian, httpd on RHEL), and the task runs on both.

    Check
    ansible <host> -m ansible.builtin.setup -a 'filter=ansible_os_family'   # RedHat for a name from Debian, or the reverse
    Fix

    Load per-OS names with include_vars: "{{ ansible_facts['os_family'] }}.yml".

  2. The repository that has it is not enabled: EPEL or CodeReady Builder on RHEL, an unregistered RHEL host with no repositories, or a missing PPA.

    Check
    ansible <host> -a 'dnf repolist'   # RHEL family: the enabled repositories
    ansible <host> -a 'apt-cache policy <package>'   # Debian family: 'Candidate: (none)'
    Fix

    Enable the repository first (ansible.builtin.yum_repository, ansible.builtin.deb822_repository, or register the host).

  3. The apt cache is stale, so a package added since the last update is unknown.

    Check
    ansible <host> -a 'stat -c %y /var/lib/apt/lists'   # an old date
    Fix

    Add update_cache: true (with cache_valid_time) to the apt task.

  4. A pinned version that the repository does not have.

    Check
    ansible <host> -a 'dnf list --showduplicates <package>'   # or apt-cache madison <package>: the versions on offer
    Fix

    Pin a version the repository offers, or add the repository that has it.

Prevent it

Now learn it: break and fix it yourself in Package name differs by OS, then take Level 05 · Values that change. Interactive version

Failed to update apt cache

Error

fatal: [web-02]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: unknown reason"}

What it means

The apt module ran apt-get update and it failed. unknown reason means the module could not classify the failure; the real reason is in apt's own output.

Causes, most likely first

  1. A repository in /etc/apt/sources.list.d is wrong or gone, for example a PPA with no Release file for this Ubuntu release.

    Check
    ansible <host> -m ansible.builtin.command -a 'apt-get update' --become   # 'does not have a Release file' names the repository
    Fix

    Remove or correct the bad repository (for example ansible.builtin.apt_repository with state: absent), then run again.

  2. A repository's signing key is missing or expired.

    Check
    ansible <host> -m ansible.builtin.command -a 'apt-get update' --become   # NO_PUBKEY or EXPKEYSIG
    Fix

    Install the key with the repository, for example ansible.builtin.deb822_repository with signed_by (ansible-core 2.15 and later).

  3. The host's clock is wrong, so apt rejects Release files as not yet valid or expired.

    Check
    ansible <host> -a 'date -u'   # far from the real time
    Fix

    Fix time sync (chrony or systemd-timesyncd), then update again.

  4. The host cannot reach the mirror (proxy, DNS or firewall).

    Check
    ansible <host> -m ansible.builtin.command -a 'apt-get update' --become   # 'Could not resolve' or 'Connection timed out'
    Fix

    Fix the proxy setting or network path to the mirror.

Prevent it

Now learn it: break and fix it yourself in Could not get lock: apt is busy, then take Level 03 · One command, many hosts. Interactive version

Failed to lock apt for exclusive operation

Error

fatal: [web-01]: FAILED! => {"changed": false, "msg": "Failed to lock apt for exclusive operation: Failed to lock directory /var/lib/apt/lists/: E:Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)"}

What it means

apt keeps lock files so only one apt or dpkg runs at a time. The module could not take the lock. 13: Permission denied means the task did not run as root; Resource temporarily unavailable or a lock held by process means another apt run has it.

Causes, most likely first

  1. The task does not use become, so it runs as a normal user who may not open the lock file.

    Check
    The message says (13: Permission denied)
    ansible <host> -m ansible.builtin.command -a whoami   # not root
    Fix

    Add become: true to the task.

  2. Another apt run holds the lock: unattended-upgrades, apt-daily, or cloud-init on a new host's first boot.

    Check
    ansible <host> -b -m ansible.builtin.command -a 'fuser -v /var/lib/dpkg/lock-frontend'   # the process holding it (fuser is in psmisc)
    ansible <host> -m ansible.builtin.command -a 'cloud-init status'   # 'running' on a new host
    Fix

    Let apt wait: lock_timeout: 600 on the apt task (the default is 60 seconds); on new hosts, run cloud-init status --wait first. Never delete lock files or kill dpkg mid-run: it can leave packages half installed.

Prevent it

Now learn it: break and fix it yourself in Could not get lock: apt is busy · Missing sudo password, then take Level 02 · Getting in. Interactive version

Could not find the requested service

Error

fatal: [web-01]: FAILED! => {"changed": false, "msg": "Could not find the requested service httpd: host"}

What it means

The service module asked the host's init system for the named service (here httpd) and it does not exist there. The trailing : host is part of the module's message, not a host name.

Causes, most likely first

  1. The service name differs by OS (apache2 on Debian, httpd on RHEL).

    Check
    ansible <host> -a "systemctl list-unit-files 'httpd*' 'apache2*'"   # the name this host uses
    Fix

    Use the right name per OS from a per-OS vars file.

  2. The package that provides the unit is not installed, or failed to install.

    Check
    ansible <host> -a 'systemctl list-unit-files <name>.service'   # 0 unit files listed
    Fix

    Install the package before the service task.

  3. A unit file was just written, and systemd has not reloaded its configuration.

    Check
    ansible <host> -a 'systemctl cat <name>'   # 'No files found' although the file exists
    Fix

    Use ansible.builtin.systemd with daemon_reload: true after writing unit files.

  4. The host does not run systemd (many containers), so the module uses another backend that does not know the unit.

    Check
    ansible <host> -m ansible.builtin.setup -a 'filter=ansible_service_mgr'   # not systemd
    Fix

    Start the process another way in containers, or test on a host with the same init system as production.

Prevent it

Now learn it: break and fix it yourself in Package name differs by OS, then take Level 05 · Values that change. Interactive version

Unable to start service: Job for x.service failed

Error

fatal: [web-01]: FAILED! => {"changed": false, "msg": "Unable to restart service app: Job for app.service failed because the control process exited with error code.\nSee \"systemctl status app.service\" and \"journalctl -xeu app.service\" for details.\n"}

What it means

The module asked systemd to (re)start the service and the service's own start command failed. Ansible only passes systemd's message on: the real reason is in the service's journal.

Causes, most likely first

  1. The config file is invalid: literal {{ }} from copy instead of template, or a typo.

    Check
    ansible <host> -b -a 'journalctl -u <service> -n 30 --no-pager'   # the service's own error, often naming the file and line
    Fix

    Fix the file (use template for .j2), then restart.

  2. The service cannot read a file it needs (owner or mode).

    Check
    ansible <host> -m ansible.builtin.stat -a path=<config>   # pw_name and mode: readable by the service user?
    Fix

    Set owner, group and mode on the file task.

  3. SELinux denies the service a file or a port: a file copied with the wrong label, or a non-standard port.

    Check
    ansible <host> -b -a 'ausearch -m AVC -ts recent'   # RHEL family: a denial naming the service
    Fix

    Fix the label (restorecon, or setype on the file task) or allow the port with community.general.seport. Do not disable SELinux.

  4. The port is already in use by another process.

    Check
    ansible <host> -b -a 'ss -ltnp'   # another program on the port
    Fix

    Stop the other process or change the port.

Prevent it

Now learn it: break and fix it yourself in Literal {{ }} in a config file · mode: 600 is not 0600, then take Level 06 · Files that fit each host. Interactive version

uri: Status code was ... and not [200]

Error

fatal: [web-01]: FAILED! => {"changed": false, ... "msg": "Status code was 503 and not [200]: HTTP Error 503: Service Unavailable", "status": 503, "url": "http://localhost:8080/healthz"}

What it means

The uri module requested the URL and got a status other than the one expected (status_code, 200 by default). -1 means no HTTP response at all, for example connection refused. The uri module runs on the managed host, so localhost in the URL is that host.

Causes, most likely first

  1. The app is not running the new configuration yet: its restart handler runs only at the end of the play, after the check.

    Check
    ansible <group> -b -a 'systemctl status <service>'   # started before the config change
    Fix

    Add meta: flush_handlers before the check.

  2. A step the release needs was skipped, for example a migration left out by --tags.

    Check
    ansible-playbook <playbook> --tags <tag> --list-tasks   # the step is missing
    Fix

    Tag the step so it runs, or run it, then check again.

  3. The release itself is broken.

    Check
    ansible <group> -b -a 'journalctl -u <service> -n 30 --no-pager'   # errors from the new version
    Fix

    Roll back to the previous version.

  4. The check uses the wrong port or path.

    Check
    ansible <group> -m ansible.builtin.uri -a 'url=http://localhost:8080/healthz'   # try the right port and path by hand
    Fix

    Correct the URL in the check.

Prevent it

Now learn it: break and fix it yourself in --tags skips a critical task · Handler runs too late · A rolling update takes every host down, then take Level 12 · No downtime. Interactive version

wait_for: Timeout when waiting for host:port

Error

fatal: [web-01]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 127.0.0.1:8080"}

What it means

wait_for tried to connect to the address and port until its timeout (300 seconds by default) and nothing ever accepted. elapsed is how long it waited.

Causes, most likely first

  1. The service failed to start or crashed, so nothing listens on the port.

    Check
    ansible <host> -m ansible.builtin.command -a 'journalctl -u <service> -n 50 --no-pager' --become   # a start error
    Fix

    Fix why the service did not start, then run the play again.

  2. It listens on another address or port than the task checks.

    Check
    ansible <host> -m ansible.builtin.command -a 'ss -ltnp' --become   # the real address and port
    Fix

    Point host: and port: at the address the app really listens on.

  3. It needs longer to start than the timeout.

    Check
    ansible <host> -m ansible.builtin.command -a 'ss -ltnp' --become   # listening now, a little after the failure
    Fix

    Raise timeout:.

  4. With delegate_to, host: is checked from the delegated machine, which may not reach it.

    Check
    grep -n -B5 'wait_for' <playbook>   # a delegate_to on the task
    Fix

    Use an address the delegated host can reach, or drop delegate_to.

Prevent it

Now learn it: break and fix it yourself in A failed deploy leaves the app down, then take Level 12 · No downtime. Interactive version

[Errno 2] No such file or directory

Error

fatal: [web-01]: FAILED! => {"changed": false, "cmd": "/usr/local/bin/lb-ctl disable web-01", "msg": "[Errno 2] No such file or directory: b'/usr/local/bin/lb-ctl'", "rc": 2}

What it means

The command module tried to start the program directly (no shell) and the operating system found no file at that path. rc: 2 is the errno, not the program's exit code: the program never started. On ansible-core 2.19 and later the message starts Error executing command: [Errno 2] ....

Causes, most likely first

  1. The program does not exist on that host.

    Check
    ansible <host> -m ansible.builtin.stat -a path=<program>   # exists: false
    Fix

    Install the program, or use the module that does the job.

  2. The program is on another host, and the task should run there.

    Check
    ansible <other host> -m ansible.builtin.stat -a path=<program>   # exists: true there
    Fix

    Add delegate_to: the host that has it.

  3. The program is installed but not on the PATH of a non-interactive SSH session (for example /usr/local/bin or ~/.local/bin added only in .bashrc).

    Check
    ansible <host> -m ansible.builtin.shell -a 'command -v <program>; echo $PATH'   # shell, because command -v is a shell builtin
    Fix

    Use the full path, or set environment: { PATH: ... } on the task.

  4. The task uses pipes, redirects or &&, which the command module passes to the program as arguments, or the first word is a shell builtin.

    Check
    grep -n -A2 'command:' <file>   # | > && or cd in the command
    Fix

    Use ansible.builtin.shell when you need shell features, or a module.

Prevent it

Now learn it: break and fix it yourself in Load balancer step runs on the wrong host, then take Level 12 · No downtime. Interactive version

Could not find or access ... on the Ansible Controller

Error

fatal: [web-01]: FAILED! => {"changed": false, "msg": "Could not find or access 'app.conf.j2'\nSearched in:\n\t/home/ops/project/roles/app/templates/app.conf.j2\n\t/home/ops/project/roles/app/app.conf.j2\n\t/home/ops/project/templates/app.conf.j2\n\t/home/ops/project/app.conf.j2\n on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

What it means

copy or template looked for src on the control node, in the order listed under Searched in:, and found it nowhere. Nothing reached the managed host.

Causes, most likely first

  1. The file is not in the role's templates/ (for template) or files/ (for copy) directory.

    Check
    ls roles/<role>/templates roles/<role>/files   # the file is elsewhere or missing
    Fix

    Move the file to one of the searched paths.

  2. A typo in src, or a difference in case.

    Check
    Compare the name in the message with the ls output, letter by letter
    Fix

    Correct src.

  3. The file is on the managed host, not the control node.

    Check
    ansible <host> -m ansible.builtin.stat -a path=<file>   # exists: true on the host
    Fix

    Set remote_src: true on copy.

  4. The path is relative to somewhere else than you think: the role, not the playbook or your shell's directory.

    Check
    Read the Searched in: list: those are the only places it looks
    Fix

    Give src relative to the role's files/ or templates/, or build it from role_path or playbook_dir.

Prevent it

Now learn it: break and fix it yourself in Literal {{ }} in a config file, then take Level 06 · Files that fit each host. Interactive version

Destination ... not writable / does not exist

Error

fatal: [web-01]: FAILED! => {"changed": false, "msg": "Destination /etc/nginx not writable"}

What it means

copy or template checked the destination before writing: the directory is not writable by the user the module runs as, or it does not exist.

Causes, most likely first

  1. The task runs as the login user without become, so it cannot write to a root-owned directory such as /etc/nginx.

    Check
    ansible <host> -m ansible.builtin.stat -a path=<directory>   # pw_name: root and not writable by others
    Fix

    Add become: true to the task.

  2. The destination directory does not exist yet (Destination directory ... does not exist).

    Check
    ansible <host> -m ansible.builtin.stat -a path=<directory>   # exists: false
    Fix

    Create it first with ansible.builtin.file: state=directory, with the right owner and mode.

Prevent it

Now learn it: break and fix it yourself in Missing sudo password, then take Level 02 · Getting in. Interactive version

non-zero return code

Error

fatal: [web-01]: FAILED! => {"changed": true, "cmd": ["/opt/app/bin/migrate"], ... "msg": "non-zero return code", "rc": 1, "stderr": "..."}

What it means

The command ran and exited with a status other than 0, which Ansible treats as failure by default. rc is the program's exit code; the reason is in stderr and stdout of the result. On ansible-core 2.19 and later the message reads The command exited with a non-zero return code.

Causes, most likely first

  1. The command really failed.

    Check
    ansible-playbook <playbook> -v   # the full result: read stderr
    Fix

    Fix the command or its input.

  2. The command uses a non-zero exit for a normal case (grep with no match returns 1, diff returns 1 when files differ).

    Check
    The tool's manual page: what each exit code means
    Fix

    Define success with failed_when: result.rc not in [0, 1].

  3. It works in your terminal but not under Ansible: a non-interactive session has another PATH, HOME or locale, and no terminal.

    Check
    ansible <host> -m ansible.builtin.shell -a 'echo $PATH; id; locale'   # compare with your interactive shell
    Fix

    Use full paths and set what it needs with environment: on the task.

Prevent it

Now learn it: break and fix it yourself in A failed deploy leaves the app down · Migrations run on every host, then take Level 07 · Run it twice. Interactive version

The task includes an option with an undefined variable

Error

fatal: [api-01]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'nginx_worker_connections' is undefined. 'nginx_worker_connections' is undefined\n\nThe error appears to be in ..."}

What it means

A task option uses a variable that has no value for this host, so Ansible cannot build the task. The name in quotes is the missing variable. On ansible-core 2.19 and later the message reads Error while resolving value for '<option>': '<name>' is undefined.

Causes, most likely first

  1. The variable is defined for some groups only, and this host is not in them.

    Check
    ansible <host> -m ansible.builtin.debug -a var=<variable>   # VARIABLE IS NOT DEFINED! ('is undefined' on 2.19+)
    ansible-inventory --host <host>   # inventory variables only: no role defaults, play vars or facts
    Fix

    Define it in the role's defaults/main.yml, or in group_vars/all.

  2. A typo in the variable name, in the task or where it is set.

    Check
    grep -rn <variable> group_vars host_vars roles
    Fix

    Use one spelling.

  3. It comes from a fact or a register that did not run on this host (gather_facts: false, a skipped task).

    Check
    ansible-playbook <playbook> -v   # the registering task was skipped for this host
    Fix

    Make sure the task runs, or use | default(...) where a default makes sense.

  4. The group_vars directory is next to neither the inventory nor the playbook, so Ansible never loads it.

    Check
    ls <inventory dir>/group_vars <playbook dir>/group_vars
    Fix

    Move group_vars next to the inventory file or the playbook.

Prevent it

Now learn it: break and fix it yourself in Undefined variable · register in a loop, then take Level 05 · Values that change. Interactive version

'dict object' has no attribute

Error

The error was: 'dict object' has no attribute 'stdout'. 'dict object' has no attribute 'stdout'

What it means

A template asked for a key (here stdout) that the dictionary does not have. On ansible-core 2.19 and later it reads object of type 'dict' has no attribute 'stdout'.

Causes, most likely first

  1. A task registered in a loop stores one result per item under .results, not .stdout.

    Check
    ansible-playbook <playbook>   # with a task: debug: var=<name>, shows a results list
    Fix

    Use <name>.results | map(attribute='stdout') | list, or loop over <name>.results.

  2. with_items over a dictionary gives the whole dictionary as one item.

    Check
    debug: var=item inside the loop   # one item holding every key
    Fix

    Use loop: "{{ my_dict | dict2items }}" with item.key and item.value.

  3. A typo in the key name.

    Check
    debug: var=<name>   # the keys it really has
    Fix

    Use the key the debug output shows.

Prevent it

Now learn it: break and fix it yourself in register in a loop · Looping over a dictionary, then take Level 06 · Files that fit each host. Interactive version

HostVarsVars object has no attribute 'ansible_default_ipv4'

Error

fatal: [lb-01]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_default_ipv4'. ..."}

What it means

A template read hostvars[...] for another host and that host has no such fact in this run. Facts exist only for hosts that gathered them (or come from a fact cache).

Causes, most likely first

  1. The run never gathered facts from those hosts (they are not in any play of this run, or gather_facts is false).

    Check
    ansible-playbook <playbook> --list-hosts   # those hosts are in no play
    Fix

    Add a play that gathers facts from those hosts first, or use inventory values such as ansible_host.

  2. They were excluded by --limit, so their facts were not gathered.

    Check
    The command line or CI job has --limit
    Fix

    Enable fact caching, or include those hosts in the limit.

  3. Facts were gathered with a gather_subset that leaves the network facts out.

    Check
    ansible <host> -m ansible.builtin.setup -a 'filter=ansible_default_ipv4'   # the fact exists when gathered fully
    grep -rn gather_subset <playbook>
    Fix

    Include network in gather_subset for that play.

Prevent it

Now learn it: break and fix it yourself in Facts of other hosts are missing, then take Level 12 · No downtime. Interactive version

Invalid data passed to 'loop', it requires a list

Error

fatal: [web-01]: FAILED! => {"msg": "Invalid data passed to 'loop', it requires a list, got this instead: {'alice': {'shell': '/bin/bash'}}. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup."}

What it means

loop: must be given a list, and it received something else (here a dictionary). On ansible-core 2.19 and later it reads: The loop value must resolve to a 'list', not 'dict'.

Causes, most likely first

  1. The variable holds a dictionary.

    Check
    ansible <host> -m ansible.builtin.debug -a var=<variable>   # { } braces: a dictionary
    Fix

    Use loop: "{{ my_dict | dict2items }}" with item.key and item.value.

  2. The variable holds a string, such as a comma-separated list, or a lookup returned a string.

    Check
    ansible <host> -m ansible.builtin.debug -a var=<variable>   # quotes around the whole value: a string
    Fix

    Store a real YAML list, use query() instead of lookup(), or split(',').

Prevent it

Now learn it: break and fix it yourself in Looping over a dictionary, then take Level 06 · Files that fit each host. Interactive version

The conditional check ... failed

Error

fatal: [edge-001]: FAILED! => {"msg": "The conditional check 'ansible_facts['os_family'] == 'Debian'' failed. The error was: error while evaluating conditional (ansible_facts['os_family'] == 'Debian'): 'ansible_facts' is undefined. ..."}

What it means

Ansible could not evaluate the when: expression at all, so it fails instead of skipping. The part after The error was: names what is missing.

Causes, most likely first

  1. The condition uses a fact that was not gathered (gather_facts: false), or a variable that is undefined for this host.

    Check
    ansible <host> -m ansible.builtin.debug -a var=<variable>   # not defined for this host
    Fix

    Gather the facts you need (gather_subset: [min] is cheap), or add <variable> is defined and ....

  2. A typo in the expression or in a key.

    Check
    ansible <host> -m ansible.builtin.debug -a 'msg={{ <expression> }}'   # the same error, outside the play
    Fix

    Correct the expression.

Prevent it

Now learn it: break and fix it yourself in Fact gathering is slow on 500 hosts, then take Level 06 · Files that fit each host. Interactive version

when runs although the variable is false

Error

TASK [Install debugging tools] ****
changed: [web-01]   (run with -e install_debug_tools=false)

What it means

when: treats any non-empty string as true, and -e key=value always passes strings, so false arrives as the string 'false'. On ansible-core 2.19 and later the task fails instead: Conditionals must have a boolean result.

Causes, most likely first

  1. The value came from -e key=value, which passes strings.

    Check
    ansible <host> -m ansible.builtin.debug -a 'var=<variable>' -e <variable>=false   # shows "false" in quotes: a string
    Fix

    Use when: <variable> | bool, or pass JSON: -e '{"<variable>": false}'.

  2. A quoted 'false' or 'no' in YAML is a string too.

    Check
    grep -rn "<variable>: ['\"]" group_vars host_vars roles   # a quoted value
    Fix

    Remove the quotes, or use | bool.

Prevent it

Now learn it: break and fix it yourself in when is always true, then take Level 06 · Files that fit each host. Interactive version

A variable has the wrong value on some hosts

Error

ansible-inventory --host web-02 shows "app_version": "2.1" while group_vars/web.yml says "2.4"

What it means

Ansible merges variables from many places, and the strongest source wins for each host. A green run with the wrong value means another source beat the one you edited.

Causes, most likely first

  1. A stronger source wins: host_vars beat group_vars, role vars/ beat inventory, set_fact beats both, and -e beats everything.

    Check
    ansible <host> -m ansible.builtin.debug -a var=<variable>   # the value the host really gets (outside a role)
    grep -rn <variable> inventory group_vars host_vars roles   # every place it is set
    Fix

    Remove the stronger value, or set yours at a level that wins; move role tunables from vars/ to defaults/.

  2. group_vars next to the playbook beat group_vars next to the inventory, and ansible-inventory does not read them unless given --playbook-dir.

    Check
    ansible-inventory --host <host> --playbook-dir <playbook dir>   # differs from the same command without --playbook-dir
    Fix

    Keep each variable in one of the two places.

  3. Two groups at the same depth set it; the one whose name sorts last wins unless ansible_group_priority says otherwise.

    Check
    ansible <host> -m ansible.builtin.debug -a var=group_names   # the groups it is in
    Fix

    Set it in one group, or set ansible_group_priority.

  4. A stale override left in host_vars.

    Check
    ls host_vars/<host>*   # a file you forgot
    Fix

    Delete the stale value.

Prevent it

Now learn it: break and fix it yourself in One host stays on the old version · Role vars beat the inventory, then take Level 05 · Values that change. Interactive version

Config file contains literal {{ }}

Error

port = {{ app_port }}   (in /etc/app/app.conf on the host)

What it means

The file on the host still holds Jinja2 text ({{ app_port }}), so it was copied as is, never rendered.

Causes, most likely first

  1. The file was shipped with copy, which does not render Jinja2.

    Check
    grep -rn -B2 '<file name>' roles/*/tasks   # copy: instead of template:
    ansible <host> -b -a "grep -n '}}' <config>"   # only the lines with unrendered braces
    Fix

    Use the template module.

  2. The template sits under files/ and is used by copy, instead of under templates/.

    Check
    ls roles/<role>/files roles/<role>/templates
    Fix

    Move .j2 files to templates/ and use template.

Prevent it

Now learn it: break and fix it yourself in Literal {{ }} in a config file, then take Level 06 · Files that fit each host. Interactive version

template error while templating string

Error

fatal: [web-01]: FAILED! => {"msg": "template error while templating string: Could not load \"to_nice_yml\": 'to_nice_yml'. String: {{ app_config | to_nice_yml }}. Could not load \"to_nice_yml\": 'to_nice_yml'"}

What it means

Jinja2 could not parse or run an expression. Could not load "to_nice_yml" (ansible-core 2.15 to 2.18) or No filter named 'to_nice_yml' (older releases and 2.19+) means no filter by that name is installed here.

Causes, most likely first

  1. A misspelt filter or test (the filter is to_nice_yaml, not to_nice_yml).

    Check
    ansible-doc -t filter -l | grep <name>   # no line: no filter by that name here
    Fix

    Correct the name.

  2. A filter from a collection that is not installed, or used without its full name, such as community.general.json_query.

    Check
    ansible-galaxy collection list   # the collection is missing
    Fix

    Install the collection and use the filter's full name.

  3. Unbalanced braces or quotes in the expression.

    Check
    The String: part of the message: count the {{ and }}
    Fix

    Close every {{ with }} and quote the whole value in YAML.

Prevent it

Now learn it: break and fix it yourself in Unquoted Jinja breaks YAML, then take Level 06 · Files that fit each host. Interactive version

File lookup cannot find the file

Error

fatal: [localhost]: FAILED! => {"msg": "The 'file' lookup had an issue accessing the file 'files/deploy_key.pub'. file not found, use -vvvvv to see paths searched"}

What it means

Lookups run on the control node, so lookup('file', ...) looked for the file there and did not find it. ansible-core 2.15 to 2.18 print the message above, older releases could not locate file in lookup, and 2.19 and later The lookup plugin 'file' failed: Unable to access the file.

Causes, most likely first

  1. The file is on the managed host, but lookups only read the control node.

    Check
    ls -l <file>   # on the control node: not there
    Fix

    To read a file that is on the managed host, use ansible.builtin.slurp instead of a lookup.

  2. A relative path is searched in the role's or playbook's files/ directory, not in your shell's current directory.

    Check
    ansible-playbook <playbook> -vvvvv 2>&1 | grep -i 'looking for\|search'   # every path tried
    Fix

    Put the file in files/ next to the playbook or in the role, or build the path from playbook_dir or role_path.

  3. A typo in the file name.

    Check
    ls -l files/ roles/<role>/files/   # the real name
    Fix

    Correct the name.

Prevent it

Now learn it: break and fix it yourself in Literal {{ }} in a config file, then take Level 06 · Files that fit each host. Interactive version

The requested handler was not found

Error

ERROR! The requested handler 'reload nginx' was not found in either the main handlers list nor in the listening handlers list

What it means

A task with notify: changed something, and no handler has that exact name or a matching listen:, so Ansible stops. The change it made stays on the host.

Causes, most likely first

  1. notify does not match a handler name exactly (case and spaces matter).

    Check
    grep -rn -i '<name>' <playbook> roles/*/handlers   # the handler's real spelling
    Fix

    Make notify match the handler's name, or add listen: <name> to the handler.

  2. The handler is in a role that this play does not load.

    Check
    grep -rln '<name>' roles/*/handlers   # the role it lives in
    Fix

    Add the role to the play, or move the handler.

Prevent it

Now learn it: break and fix it yourself in Handler not found, then take Level 07 · Run it twice. Interactive version

Handler did not run: service still uses old config

Error

TASK [Write nginx.conf] ****
ok: [web-01]   (no RUNNING HANDLER)

What it means

Handlers run only when a notifying task reports changed, and only at the end of the play (or at meta: flush_handlers). Here the task reported ok, so nothing was notified.

Causes, most likely first

  1. A previous run changed the file and failed before handlers ran. This run sees the file already right, reports ok, and notifies nothing.

    Check
    ansible <host> -b -a 'systemctl show <service> -p ActiveEnterTimestamp'   # started before the file changed
    ansible <host> -m ansible.builtin.stat -a path=<file>   # mtime newer than the start time
    Fix

    Restart or reload once by hand.

  2. Handlers run at the end of the play, after the tasks that needed the change.

    Check
    ansible-playbook <playbook> -v   # RUNNING HANDLER only after the last task
    Fix

    Add meta: flush_handlers where later tasks need the change.

  3. The notifying task has changed_when: false, so it never reports changed.

    Check
    grep -rn -B5 'notify' roles <playbook> | grep changed_when
    Fix

    Give it a real changed_when.

Prevent it

Now learn it: break and fix it yourself in Handler never fired · Handler runs too late, then take Level 07 · Run it twice. Interactive version

Task reports changed on every run

Error

web-01 : ok=4    changed=1    unreachable=0    failed=0   (every run)

What it means

A task reports changed even when the host is already right, so the changed count is never 0 and its handlers run every time.

Causes, most likely first

  1. command or shell tasks report changed every time unless told otherwise (changed_when, creates, removes).

    Check
    Run the playbook twice: the task changed on the second run is the one
    Fix

    Use a module, or set changed_when to what really means changed, or creates:.

  2. A shell command appends to a file, so it really changes something every run.

    Check
    ansible <host> -a 'grep -c <line> <file>'   # the same line many times
    Fix

    Use ansible.builtin.lineinfile or blockinfile.

Prevent it

Now learn it: break and fix it yourself in Changed on every run · A check restarts the app every run, then take Level 07 · Run it twice. Interactive version

Check mode skips command and shell tasks

Error

TASK [Raise max_connections] ****
skipping: [db-01]   (with --check)

What it means

In check mode Ansible asks each module to predict its change without making it. command and shell cannot predict, so they are skipped, and --check --diff shows nothing for them.

Causes, most likely first

  1. The change is made by a command or shell task.

    Check
    ansible-playbook <playbook> --check --diff -v   # 'Command would have run if not in check mode' or skipping
    Fix

    Manage the state with a module (lineinfile, template) that supports check and diff.

  2. A later task reads the skipped task's register and fails or misleads.

    Check
    ansible-playbook <playbook> --check -v   # a later failure about the register
    Fix

    Set check_mode: false on read-only commands that later tasks depend on.

Prevent it

Now learn it: break and fix it yourself in Check mode shows nothing, then take Level 11 · Change only what you mean. Interactive version

Attempting to decrypt but no vault secrets found

Error

ERROR! Attempting to decrypt but no vault secrets found

What it means

Ansible met vault-encrypted data and had no vault password at all to try. The data is fine; no password reached this run.

Causes, most likely first

  1. No vault password was given: no -J (--ask-vault-password), --vault-password-file, --vault-id or vault_password_file setting.

    Check
    ansible-config dump | grep -i vault   # DEFAULT_VAULT_PASSWORD_FILE(default) = None and an empty DEFAULT_VAULT_IDENTITY_LIST
    Fix

    Pass --vault-password-file <file> or -J, or set vault_password_file in the project's ansible.cfg.

  2. vault_password_file is set in an ansible.cfg that was not loaded (another directory, a world writable directory), or ANSIBLE_VAULT_PASSWORD_FILE is set in your shell but not in CI.

    Check
    ansible --version   # config file = which ansible.cfg was read
    Fix

    Run from the project directory, or set the variable in the CI job.

Prevent it

Now learn it: break and fix it yourself in No vault secrets found, then take Level 10 · Keeping secrets. Interactive version

Decryption failed (no vault secrets were found that could decrypt)

Error

ERROR! Decryption failed (no vault secrets were found that could decrypt) on /home/ops/project/inventory/group_vars/all/vault.yml

What it means

Ansible had one or more vault passwords and none of them opened this file. The file is named at the end of the message.

Causes, most likely first

  1. The wrong password or password file.

    Check
    ansible-vault view <file> --vault-password-file <password file> > /dev/null && echo opens   # tests the password without printing the secrets
    Fix

    Use the right password.

  2. The file was rekeyed, or encrypted with another vault id's password.

    Check
    head -1 <file>   # $ANSIBLE_VAULT;1.2;AES256;prod: encrypted for the prod vault id
    Fix

    Pass --vault-id <label>@<file> for each password the run needs.

  3. The password file is executable, so Ansible runs it as a script and uses what it prints; a script that prints the wrong value fails here. (A plain file with the x bit fails earlier with Problem running vault password script.)

    Check
    ls -l <password file>   # an x in the mode: it is run, not read
    Fix

    Fix the script, or chmod -x a plain password file.

Prevent it

Now learn it: break and fix it yourself in No vault secrets found, then take Level 10 · Keeping secrets. Interactive version

input is not vault encrypted data

Error

ERROR! input is not vault encrypted data. group_vars/all.yml is not a vault encrypted file for group_vars/all.yml

What it means

ansible-vault view, edit and decrypt work on whole encrypted files, and this file has no $ANSIBLE_VAULT header. On ansible-core 2.19 and later it reads Failed to view 'group_vars/all.yml': Input is not vault encrypted data.

Causes, most likely first

  1. The file is plain text, perhaps decrypted and committed by mistake.

    Check
    head -1 <file>   # no $ANSIBLE_VAULT;1.1;AES256 line
    Fix

    Encrypt it: ansible-vault encrypt <file>; rotate any secret that was committed in plain text.

  2. The file is plain YAML holding inline encrypted values (!vault | from encrypt_string). ansible-vault view and edit do not open those.

    Check
    grep -c '!vault' <file>   # a number above 0: inline values
    Fix

    Leave the file as it is; to change a value, make a new one with ansible-vault encrypt_string and replace it.

Prevent it

Now learn it: break and fix it yourself in Password in plain text in Git, then take Level 10 · Keeping secrets. Interactive version

Secret visible in task output or CI logs

Error

changed: [web-01] => {"changed": true, "cmd": ["/opt/app/bin/app-admin", "set-password", "..."], ...}

What it means

Ansible prints task arguments and results with -v and on failure. A secret passed to a task without no_log: true lands in the terminal and the CI log.

Causes, most likely first

  1. The task handles a secret without no_log: true.

    Check
    ansible-playbook <playbook> -v 2>&1 | grep -ciE 'password|token|secret'   # a count, so the check does not print the secret again
    Fix

    Add no_log: true to every task that receives a secret; rotate the leaked secret.

  2. debug: var= on a secret, or a register of a secret printed later.

    Check
    grep -rn -A2 'debug:' roles <playbook>   # var= on a secret variable
    Fix

    Remove the debug task; rotate the secret.

  3. The secret is a command-line argument. The result's cmd shows it, and the host's process list shows it to other users even with no_log.

    Check
    grep -rnE 'password=|--password' roles <playbook>
    Fix

    Pass it with stdin: or a file with mode 0600, not as an argument.

Prevent it

Now learn it: break and fix it yourself in Secrets printed in the CI log, then take Level 10 · Keeping secrets. Interactive version

mode: 644 without a leading zero sets odd permissions

Error

TASK [Write the config] ****
changed: [web-01]   (the task sets mode: 644)

$ ls -l /etc/app/app.conf
--w----r-T 1 root root 312 Oct 30 10:02 /etc/app/app.conf

What it means

YAML reads an unquoted mode: 644 as the decimal number 644, which is 1204 in octal: the sticky bit plus -w----r--. Ansible applies that, so ls -l shows --w----r-T.

Cause

  1. An unquoted three-digit mode without a leading zero.

    Check
    stat -c '%a %A %n' <path>   # on the host: 1204 --w----r-T
    grep -rnE 'mode: [0-7]{3}\s*$' roles playbooks   # every unquoted mode
    Fix

    Quote it with a leading zero, mode: "0644", or use a symbolic mode such as u=rw,g=r,o=r, then run the play again to correct the files it already wrote.

Prevent it

Now learn it: break and fix it yourself in mode: 600 is not 0600, then take Level 10 · Keeping secrets. Interactive version

the role ... was not found

Error

ERROR! the role 'baseline' was not found in /home/ops/project/roles:/home/ops/project/roles:/home/ops/project

What it means

The play names a role that is in none of the listed directories. The colon-separated list after was not found in is every place Ansible searched.

Causes, most likely first

  1. The role comes from Galaxy or another repository and was not installed on this control node.

    Check
    ansible-galaxy role list   # not listed
    cat requirements.yml   # listed here?
    Fix

    ansible-galaxy install -r requirements.yml.

  2. It is installed, but not in a searched directory: roles_path in ansible.cfg replaces the default list.

    Check
    ansible-config dump | grep -i roles_path   # the paths really searched
    Fix

    Add the install directory to roles_path, or install into one of the searched paths.

  3. The role is inside a collection and is referred to by its short name.

    Check
    ansible-galaxy collection list   # the collection that ships it
    Fix

    Use the full name: namespace.collection.role.

  4. A typo in the role name, or the directory name differs.

    Check
    ls roles/
    Fix

    Match the name to the directory.

Prevent it

Now learn it: break and fix it yourself in The role was not found · A role that does nothing, then take Level 09 · Other people's code. Interactive version

ansible.builtin.include has been removed

Error

ERROR! [DEPRECATED]: ansible.builtin.include has been removed. Use include_tasks or import_tasks instead. This feature was removed from ansible-core in a release after 2023-05-16. Please update your playbooks.

What it means

The old include: action was removed in ansible-core 2.16. The message depends on how it was written: include: file=x.yml gives the text above, the more common include: x.yml gives this task 'include' has extra params, which is only allowed in the following modules: ... on 2.16 to 2.18, and 2.19 and later say The 'ansible.builtin.include' action plugin has been removed.

Causes, most likely first

  1. Tasks still use include: after an upgrade to ansible-core 2.16 or later.

    Check
    ansible --version   # core 2.16 or later
    grep -rnE '^\s*-?\s*(ansible\.builtin\.)?include:' .   # every old include
    Fix

    Replace include: with import_tasks: (static) or include_tasks: (dynamic).

  2. A play-level - include: other.yml that pulls in a playbook.

    Check
    grep -n '^- include:' *.yml
    Fix

    Use - import_playbook: other.yml.

Prevent it

Now learn it: break and fix it yourself in include removed after an upgrade, then take Level 13 · Many hosts, every day. Interactive version

Failed to resolve the requested dependencies map

Error

ERROR! Failed to resolve the requested dependencies map. Could not satisfy the following requirements:
* community.genral:* (direct request)

What it means

ansible-galaxy could not find a set of collection versions that satisfies requirements.yml. The lines after it name each requirement it could not meet and where it came from.

Causes, most likely first

  1. A typo in the collection name (here community.genral).

    Check
    cat requirements.yml   # the name as written
    Fix

    Correct the name.

  2. A version constraint that no release satisfies, or two collections that need conflicting versions of a dependency.

    Check
    ansible-galaxy collection install -r requirements.yml -vvv   # the versions it found and the conflict
    Fix

    Loosen or correct the constraint.

  3. The collection is on Automation Hub or a private Galaxy that server_list does not include, or the control node cannot reach the server (proxy, certificate).

    Check
    ansible-config dump | grep -i galaxy_server   # the servers it asks
    Fix

    Add the server to [galaxy] server_list, or fix the proxy or CA settings.

Prevent it

Now learn it: break and fix it yourself in couldn't resolve module/action, then take Level 09 · Other people's code. Interactive version

Runs take too long on many hosts

Error

Playbook run took 0 days, 0 hours, 53 minutes, 20 seconds

What it means

Ansible runs each task across hosts in batches of forks (5 by default) and waits for the batch, so a big fleet with defaults spends most of its time waiting.

Causes, most likely first

  1. forks is 5 by default, so hosts are handled five at a time.

    Check
    ansible-config dump | grep -i forks   # DEFAULT_FORKS(default) = 5
    Fix

    Raise forks to what the control node's CPU and memory can handle.

  2. Full fact gathering on every play.

    Check
    Set `callbacks_enabled = ansible.posix.profile_tasks` in ansible.cfg   # Gathering Facts near the top of the timings
    Fix

    gather_subset: [min], gather_facts: false where facts are unused, or fact caching.

  3. Pipelining is off, or ssh_args was overridden without the default ControlMaster/ControlPersist options, so every task pays for new connections.

    Check
    ansible-config dump | grep -i pipelining
    grep -n ssh_args ansible.cfg   # a value without ControlPersist
    Fix

    pipelining = True in [ssh_connection]; keep -C -o ControlMaster=auto -o ControlPersist=60s in any ssh_args you set.

Prevent it

Now learn it: break and fix it yourself in Fact gathering is slow on 500 hosts, then take Level 13 · Many hosts, every day. Interactive version

async task did not complete within the requested time

Error

fatal: [db-01]: FAILED! => {"changed": false, "msg": "async task did not complete within the requested time - 300s"}

What it means

The task was started with async: 300, ran longer than 300 seconds, and Ansible stopped waiting and ended the job.

Causes, most likely first

  1. async: is shorter than the task really takes.

    Check
    Time the command in staging
    Fix

    Raise async: above the longest run you expect, and keep poll: at a few seconds.

  2. The command is stuck, for example waiting for input or for a lock.

    Check
    ansible <host> -m ansible.builtin.shell -a 'ps -ef | grep <command>'   # still there long after it should have ended
    Fix

    Make the command non-interactive so it cannot wait for input; find what holds the lock.

  3. The task really needs to run longer than you want to wait.

    Check
    The job's normal duration
    Fix

    Start it with poll: 0, register the job, and check it later with ansible.builtin.async_status in an until loop.

Prevent it

Now learn it: break and fix it yourself in Long task killed by a dropped connection, then take Level 13 · Many hosts, every day. Interactive version

A worker was found in a dead state

Error

ERROR! A worker was found in a dead state

What it means

One of ansible-playbook's worker processes on the control node exited unexpectedly, so the run aborts. It is about the control node, not the managed hosts.

Causes, most likely first

  1. forks is set higher than the control node's memory allows, and the kernel's out-of-memory killer ended a worker.

    Check
    sudo dmesg | grep -i 'killed process'   # on the control node: an ansible-playbook process
    Fix

    Lower forks, or give the control node more memory.

  2. A module or plugin crashed the worker.

    Check
    ansible-playbook <playbook> -vvv   # the last task before the abort
    Fix

    Upgrade or fix that plugin.

Prevent it

Now learn it: break and fix it yourself in Fact gathering is slow on 500 hosts, then take Level 13 · Many hosts, every day. Interactive version

UNREACHABLE: client_loop: send disconnect: Broken pipe

Error

fatal: [db-01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: client_loop: send disconnect: Broken pipe", "unreachable": true}

What it means

The SSH connection was open and then broke while Ansible was still using it. Something between the hosts, or the host itself, ended the session.

Causes, most likely first

  1. A firewall or NAT gateway drops SSH sessions that look idle during a long task.

    Check
    ssh -o ServerAliveInterval=30 <user>@<address> 'sleep 600'   # survives with keepalives, fails without
    Fix

    Add keepalives with ansible_ssh_common_args: '-o ServerAliveInterval=30' (ssh_args replaces the defaults, so keep ControlMaster and ControlPersist in it if you use that), or run the task with async and poll.

  2. The host rebooted or ran out of memory.

    Check
    ansible <host> -a uptime   # a very short uptime: it rebooted
    ansible <host> -b -a 'journalctl -k -n 50 --no-pager'   # out of memory messages
    Fix

    Find why it rebooted or ran out of memory; use ansible.builtin.reboot for planned reboots.

Prevent it

Now learn it: break and fix it yourself in Long task killed by a dropped connection, then take Level 13 · Many hosts, every day. Interactive version

A deploy took every host down at once

Error

fatal: [web-01]: FAILED! ...
fatal: [web-02]: FAILED! ...

NO MORE HOSTS LEFT

What it means

Without serial, each task runs on every host before the next task, so a bad release reaches all hosts before any health check can stop it. NO MORE HOSTS LEFT means every host in the play has failed.

Causes, most likely first

  1. The play has no serial, so there is no batch to stop after.

    Check
    grep -n serial <playbook>   # no output
    Fix

    Roll back with -e version=<previous>, then add serial: 1 (or a percentage) and max_fail_percentage.

  2. No rollback when a step fails.

    Check
    grep -n rescue <playbook>   # no output
    Fix

    Wrap the deploy in block/rescue that restores the previous release.

Prevent it

Now learn it: break and fix it yourself in A rolling update takes every host down · A failed deploy leaves the app down, then take Level 12 · No downtime. Interactive version