Debugging

When something is wrong, where to look.

Python-side debugging

• tf.debugging (tensorflow/python/ops/check_ops.py, tensorflow/python/ops/numerics.py) provides assertion ops: assert_equal, assert_all_finite, etc. They raise eagerly, or insert assertion ops in graph mode.
• tf.debugging.set_log_device_placement(True) — log every op placement to stderr.
• tf.debugging.experimental.enable_dump_debug_info(...) — dump a tfdbg2 directory readable by TensorBoard's Debugger V2 plugin (tensorflow/python/debug/).
• tf.print(...) — graph-mode-friendly printing; prefer it over print() inside @tf.function.
• TF_CPP_MIN_LOG_LEVEL — env var that controls C++ logging: 0 (all), 1 (info-and-up suppressed), 2 (warnings suppressed), 3 (errors suppressed).
• TF_CPP_VMODULE='executor=2,direct_session=2' — selectively crank verbosity for individual files.

tensorflow/python/debug/ (tfdbg) is the larger debugger framework. The CLI tfdbg is gone in 2.x; the Python API now writes to a directory consumed by TensorBoard.

C++-side debugging

• Build with --copt=-O0 --copt=-g for debuggable binaries.
• --config=asan, --config=msan, --config=tsan map onto Bazel sanitizer configs (see .bazelrc).
• LOG(INFO), VLOG(2), etc. from tensorflow/core/platform/logging.h.
• core dumps — set TF_USE_SIGNAL_HANDLER=1 for friendlier backtraces.

Common failure modes

Symptom	Likely cause	Where to look
RuntimeError: tf.function-decorated function tried to create variables on non-first call	Variable created inside @tf.function body without tf.init_scope.	tensorflow/python/eager/polymorphic_function/
Cannot convert a partially known TensorShape	Shape inference missing a case.	tensorflow/core/ops/<op_family>_ops.cc
OOM when allocating tensor with shape ...	GPU memory pressure; check BFC allocator stats.	tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.{cc,h}
Failed to load the native TensorFlow runtime	pip wheel/version mismatch; CUDA version mismatch.	tensorflow/python/platform/sysconfig.py
Test passes locally, fails on CI	Tag mismatch (e.g., test is large but CI runs only medium).	The BUILD file's size, tags.
XLA cluster is empty warning in logs	Auto-clustering decided not to JIT.	tensorflow/compiler/jit/

Tracing performance

• The TF profiler (tf.profiler.experimental.start/stop) writes a directory of trace events readable by TensorBoard.
• C++ side trace events are emitted by tensorflow/core/profiler/lib/traceme.h (TraceMe is the macro you'll see sprinkled through hot paths).
• For GPU: NVIDIA Nsight Systems or nvprof work fine on TF programs.

Where to ask

• Internal logic / runtime issues: open an issue with comp:core.
• Compiler issues: comp:xla or comp:mlir.
• TFLite: comp:lite. Many TFLite issues are answered in tensorflow/lite/g3doc/.
• Forum: discuss.tensorflow.org for usage questions.

Related

• testing — how to run tests with sanitizers.
• tooling — Bazel configs.