Debugging

Tactics for figuring out what ansible-core is doing when it doesn't behave the way you expect.

Verbosity

The single most useful flag is -v. Every CLI accepts up to -vvvvv:

Flag	What it adds
-v	Brief task results, including stdout/stderr
-vv	Plus task path (which playbook/file/line each task came from)
-vvv	Plus connection details (the SSH command, the local path of the AnsiBallZ archive)
-vvvv	Plus connection-plugin internals — full SSH negotiation, sftp transfers
-vvvvv	Plus low-level module bridge debug

Display.vvv(), Display.vvvv(), and Display.vvvvv() (lib/ansible/utils/display.py) gate the output. New code should use these helpers consistently rather than print.

ANSIBLE_KEEP_REMOTE_FILES

When ANSIBLE_KEEP_REMOTE_FILES=1 is set in the environment, the controller skips cleanup of the AnsiBallZ archives on the remote host. The archive paths are printed at -vvv. SSH into the target and run the script directly with python3 AnsiballZ_<module>.py to reproduce just the module-side execution.

The archive is a zipfile with __main__.py, the module source, and any module_utils/ it imports. You can unzip -l it to see exactly what got shipped.

ANSIBLE_DEBUG

ANSIBLE_DEBUG=1 raises the controller's internal log level. It is verbose — most useful when chasing plugin loader resolution or templating issues.

The debug strategy plugin

For step-through-style debugging, set strategy: debug in your play (or ANSIBLE_STRATEGY=debug). Implementation: lib/ansible/plugins/strategy/debug.py. When a task fails, the strategy drops you into a pdb-like prompt with commands such as:

• p task — print the task object.
• p task_vars — print the variable scope.
• r — re-execute the task.
• c — continue.
• q — quit.

The result is interactive failure inspection without re-running the whole play.

breakpoint() in modules

Modules run in a remote Python interpreter; standard pdb doesn't help on the wire. To trace module execution locally:

1. Copy the module into a directory on your $ANSIBLE_LIBRARY (or use -M).

2. Use hacking/test-module.py to run the module standalone:

   ./hacking/test-module.py -m lib/ansible/modules/file.py -a 'path=/tmp/foo state=touch'

   hacking/test-module.py builds the AnsiBallZ archive, prints the path, and optionally invokes it locally with the supplied arguments. You can edit the archive, set breakpoints, and re-run it.

The Display object

Most controller-side code uses a Display singleton:

from ansible.utils.display import Display
display = Display()
display.vvv("loader: trying %s" % path)

Display lives in lib/ansible/utils/display.py. It serializes terminal output across the multiple worker processes and supports display.warning(), display.error(), display.deprecated(), and the per-verbosity vvN helpers. Don't print() from controller code — the output won't be interleaved with the rest of the CLI's output and won't respect --no-color.

Common error-source patterns

Symptom	Likely cause	Where to look
MODULE FAILURE with no useful detail	Remote Python missing or returning non-JSON	Run with -vvv to see stdout/stderr; check the remote ansible_python_interpreter
Could not match supplied host pattern	Inventory plugin not finding the host	ansible-inventory --graph -i <source>
template error while templating string	Untrusted string with {{ }} syntax	Check the value's Origin; templating only runs on TrustedAsTemplate strings
Plugin not found	FQCN mismatch or missing collection	ansible-galaxy collection list; check lib/ansible/config/ansible_builtin_runtime.yml for redirects
Vault decryption error	Wrong --vault-id / --vault-password-file	Format header is $ANSIBLE_VAULT;1.2;AES256;<id>
Failed to import the required Python library from a module	Module's import_required couldn't find the lib on the target	Use ansible -m setup <host> to inspect the target's Python environment
Lookup plugin returns unexpected types	Plugin returning legacy str where new datatag-aware code expects a tagged scalar	The lookup plugin probably needs to opt into the templating engine via lib/ansible/_internal/_templating/_jinja_plugins.py

Inspecting inventory and vars

ansible-inventory -i <inventory> --graph
ansible-inventory -i <inventory> --host <hostname>
ansible -i <inventory> <pattern> -m debug -a 'msg={{ ansible_facts }}'

These three commands cover most "what does Ansible see for this host?" questions. The output of --host is the merged variable view from the perspective of lib/ansible/vars/manager.py:VariableManager.get_vars.

Reading callback output

Different callback plugins surface different detail. default is the human-readable one, minimal and oneline are terser, junit and tree write structured artifacts to disk. Switch with ANSIBLE_STDOUT_CALLBACK=oneline or --stdout-callback=oneline. See Plugins → Callback.

When the test suite fails

AGENTS.md lays out the standard workflow:

1. Read the ansibot PR comment; it summarizes the failure.
2. gh pr checks <pr> → Azure Pipelines URL.
3. hacking/azp/download.py <build_id> → console logs in <build_id>/.
4. grep -r "FAILED\|ERROR\|Traceback" <build_id>/ → narrow to the failing job.
5. Reproduce locally with ansible-test sanity / units / integration against the same target.

See Tooling for full coverage of the CI loop.

Cross-links

• Tooling for hacking/azp/download.py and CI internals.
• Testing for how to write tests that fail in a useful way.
• Systems → Module execution and AnsiBallZ for the on-the-wire format.