Mobile and edge

What it is

PyTorch supports on-device inference through two stacks:

1. PyTorch Mobile — the older path; runs .ptl (lite-interpreter) files via a stripped-down JIT runtime. Built into the in-tree CMake/Bazel/Buck via aten/src/ATen/native/mobile, c10/mobile, torch/jit/mobile, and the Android/iOS bindings under android/ and ios/ (now in a separate repo for iOS).

2. ExecuTorch — the modern successor. Lives in a separate repo (pytorch/executorch); consumes ExportedProgram from torch.export and produces a .pte flatbuffer plus a tiny C++ runtime. The PyTorch tree exposes hooks (decomposition tables, kernel registration, custom op surfaces) used by ExecuTorch.

This page focuses on what the PyTorch repo contributes; ExecuTorch itself is documented in its own repo.

Mobile vs. server

The deployment story for on-device inference is fundamentally different from server:

• No Python runtime. The model has to load and run from C++ on iOS/Android.
• Tight memory budgets. Activation memory, weights, and runtime all have to fit.
• Limited ops. Only a curated subset of ATen is supported on each device.
• Heterogeneous hardware. ARM Mali/Bifrost/Mediatek/Apple GPU, CPU NEON SIMD, NPUs/DSPs.
• Quantization is the norm. int8 (XNNPACK) or even int4 weights.

In-tree mobile pieces

Path	What it is
c10/mobile/	Mobile-specific c10 trims
aten/src/ATen/native/mobile/	Mobile-friendly kernels
torch/jit/mobile/	Lite-interpreter Python helper
torch/csrc/jit/mobile/	Lite-interpreter C++ runtime
torch/csrc/jit/serialization/	.ptl reader/writer
android/	Android Gradle build + JNI bindings
aten/src/ATen/native/quantized/cpu/qnnpack/	QNNPACK / XNNPACK integration
aten/src/ATen/nnapi/	Android NNAPI integration

Selective build

Mobile binaries can selectively include only the ops a model uses. The torchgen codegen reads a root_ops YAML and emits a stripped registration file. See torchgen/selective_build/. Selective build is the difference between a 30 MB and a 5 MB PyTorch binary on Android.

XNNPACK

The default mobile CPU backend is XNNPACK (Google's high-performance ARM/x86 NEON/SSE/AVX kernel library). PyTorch ships XNNPACK kernels for the common quantized convs/linears/poolings under aten/src/ATen/native/xnnpack/. The PT2E XNNPACKQuantizer targets exactly these ops.

NNAPI

Android NNAPI bindings let some models offload to NPUs / GPUs via Android's hardware abstraction. Glue lives at aten/src/ATen/nnapi/. NNAPI support has plateaued in favour of vendor-specific delegates accessible through ExecuTorch.

Vulkan and Metal compute

For mobile GPU compute, two paths:

• Vulkan — aten/src/ATen/native/vulkan/. Compiled GLSL compute shaders, used on Android GPUs.
• Metal — historically aten/src/ATen/native/metal/; the modern Apple GPU path uses MPS (which works on macOS, iOS, iPadOS).

ExecuTorch hooks in this tree

When you pip install executorch, the framework consumes:

• torch.export ExportedPrograms (decomposed via the core ATen op table).
• A specific decomposition list controlled by the ExecuTorch backend config.
• Custom op registrations through torch.library for backend-specific ops.

The PyTorch tree at torchgen/executorch/ exposes the codegen pieces ExecuTorch uses to register its op set.

Deployment example (legacy mobile)

import torch
model = MyModel().eval()
example = torch.randn(1, 3, 224, 224)
ts = torch.jit.trace(model, example)
ts._save_for_lite_interpreter("model.ptl")

Then in Android Studio, drop the file in assets/ and load it via org.pytorch.LiteModuleLoader. iOS uses LibTorchLite similarly.

Deployment example (ExecuTorch)

import torch
from torch.export import export
from executorch.exir import to_edge

ep = export(model, (x,))
edge = to_edge(ep)
program = edge.to_executorch()
program.write_to_file("model.pte")

Where to read next

• Systems / JIT — the lite interpreter.
• Features / Quantization — XNNPACK / quantization.
• Features / torch.export and AOTInductor — ExecuTorch's input.
• The ExecuTorch repo at https://github.com/pytorch/executorch.