TFRT

The newer asynchronous runtime fallback. Lives in tensorflow/core/tfrt/ and tensorflow/core/runtime_fallback/.

Purpose

• Provide an asynchronous, lock-free runtime designed around explicit dataflow ("AsyncValue" per tensor) rather than the original executor's threadpool/dispatcher model.
• Offer better performance for graphs of small ops (where the original runtime has fixed overhead per op).
• Run TF ops by falling back to the original kernel set — TFRT does not reimplement every op.

Status

TFRT is partially shipped: it is the runtime behind some serving paths and Google-internal use cases, but most external users still hit the classic eager / executor path. The tfrt/ and runtime_fallback/ directories see active commits but are not the default runtime for tf.function in the pip wheel.

Directory layout

tensorflow/core/tfrt/
├── runtime/                       # AsyncValue-based runtime core
├── saved_model/                   # SavedModel loader on TFRT
├── graph_executor/                # tf.function execution on TFRT
├── ifrt/                          # IFRT (XLA-derived) integration
├── kernels/                       # TFRT-native kernels (small set)
├── mlrt/                          # MLIR + Runtime — newer compiled runtime
├── transforms/                    # MLIR passes used by TFRT
├── utils/
└── ...

tensorflow/core/runtime_fallback/
├── kernel/                        # Fallback ops that re-enter classic runtime
├── runtime/                       # Glue to call classic OpKernel from TFRT
├── opdefs/                        # MLIR op definitions for fallback
└── ...

How it interacts with the classic runtime

graph LR
    Caller[tf.function call]
    Compiler[MLIR / TFRT lowering]
    TFRTRT[TFRT runtime: AsyncValues, tasks]
    NativeTFRT[Small set of TFRT-native kernels]
    Fallback[Runtime fallback]
    ClassicKernel[Classic OpKernel]

    Caller --> Compiler
    Compiler --> TFRTRT
    TFRTRT -->|small set of ops| NativeTFRT
    TFRTRT -->|most ops| Fallback
    Fallback --> ClassicKernel

A graph compiled to TFRT MLIR runs through TFRT's task scheduler. Most ops dispatch to the runtime fallback, which packs the classic OpKernelContext around the TFRT inputs and calls the existing OpKernel::Compute. The result is an AsyncValue that downstream tasks await.

MLRT

tensorflow/core/tfrt/mlrt/ is a more recent compiled runtime that lowers graphs to a custom byte-code-like representation for cheaper dispatch. It's used for some serving workloads inside Google.

When you encounter TFRT

• If you grep for AsyncValue or RuntimeFallback, you're looking at TFRT plumbing.
• TFRT-related env vars: TF_USE_TFRT, TF_USE_MLRT, TF_TFRT_ENABLE_*.
• Some SavedModel-loading paths in Google's internal serving go through tensorflow/core/tfrt/saved_model/ instead of the classic loader.

Integration points

• MLIR: TFRT consumes MLIR (TF dialect, MHLO) and lowers it to TFRT's runtime ops. See compilers/mlir.
• Runtime fallback: bridges to classic kernels.
• SavedModel: tfrt::SavedModelImpl loads a SavedModel and constructs a TFRT-callable function.

Entry points for modification

• TFRT-native kernels — tensorflow/core/tfrt/kernels/.
• Lowering passes — tensorflow/core/tfrt/transforms/.
• MLRT — tensorflow/core/tfrt/mlrt/.

Caveats

The classic runtime (core-runtime) is still the most-used path; TFRT and the runtime fallback should be considered an alternative dispatcher, not a replacement.

Related

• core-runtime — the classic runtime that fallback delegates to.
• compilers/mlir — MLIR is TFRT's input language.