TiDB Lightning

Lightning is a high-throughput data importer that ingests CSV/SQL/Parquet/Aurora-snapshot files into a TiDB cluster, optionally bypassing the SQL layer by writing SST files directly into TiKV.

Source

• CLI: lightning/cmd/ (entry: lightning/cmd/tidb-lightning/main.go and lightning/cmd/tidb-lightning-ctl/main.go).
• Library: pkg/lightning/.
• Build: make build_lightning and make build_lightning-ctl → bin/tidb-lightning, bin/tidb-lightning-ctl.
• Default config: lightning/tidb-lightning.toml.
• Top-level docs: lightning/README.md.

Directory layout

lightning/
├── cmd/                # tidb-lightning, tidb-lightning-ctl
├── cli/                # CLI wiring
├── context/            # Top-level context construction
├── log/                # Lightning-specific log helpers
├── export/             # Export side of physical mode
└── tests/              # Shell-driven integration tests

pkg/lightning/
├── backend/            # Pluggable backends
│   ├── local/          # Physical (SST) backend
│   ├── tidb/           # Logical (SQL via TiDB) backend
│   ├── kv/             # KV backend helpers
│   ├── encode/         # Row → KV encoding
│   └── external/       # External-storage SST stage
├── common/             # Misc types and helpers
├── config/             # Lightning config schema
├── duplicate/          # Duplicate detection / resolution
├── importdef/          # Schema describing IMPORT INTO jobs
├── log/                # Logging
├── manual/             # Manual / pause / resume helpers
├── membuf/             # Memory-buffered KV writer
├── metric/             # Lightning Prometheus metrics
├── mydump/             # Source-file parsers (CSV, SQL, Parquet, Aurora)
├── tikv/               # TiKV-side helpers (SST ingest, region split)
├── verification/       # Per-table checksum verification
└── worker/             # Worker pool for parallel imports

Backends

Lightning supports two backends with very different performance characteristics:

Backend	Path	Throughput	When to use
Physical (local)	pkg/lightning/backend/local/	Highest. Encodes rows into SST files and ingests them via TiKV's Region-level SST ingest API. Bypasses SQL.	Initial bulk-load of a fresh cluster, large migrations.
Logical (tidb)	pkg/lightning/backend/tidb/	Lower (bound by SQL). Issues INSERT statements through the SQL layer.	Online imports, smaller datasets, clusters that already have traffic.

config.Backend selects between them at runtime.

How a physical-mode import runs

graph LR
  src[Source files<br/>CSV / SQL / Parquet] --> mydump[mydump parser]
  mydump --> rows[Row iterator]
  rows --> encode[backend/encode<br/>row → KV pairs]
  encode --> sort[External sort + SST build<br/>backend/external + local]
  sort --> ingest[TiKV SST ingest API]
  ingest --> tikv[(TiKV)]
  ingest --> verify[Checksum + analyze]

1. Parse: pkg/lightning/mydump/ reads source files and produces a row stream. Format detection picks CSV/SQL/Parquet/Aurora-snapshot parsers.
2. Encode: each row is converted to MVCC KV pairs using the table codec (pkg/tablecodec/) and the destination's auto-id allocator (pkg/meta/autoid/).
3. Stage and sort: KVs are buffered in membuf/ and the external-sort pipeline in backend/external/. Sorted output is written to local disk as SST files.
4. Ingest: backend/local/ and pkg/lightning/tikv/ issue Region-level SST ingest RPCs to each TiKV peer. PD schedulers are paused for the table's regions during the import.
5. Verify: verification/ runs ADMIN CHECKSUM TABLE and compares against expected checksums computed during encode.
6. Switch back: schedulers resume; final stats and indexes are validated.

Logical-mode import

backend/tidb/ opens a connection (or many) to the target TiDB cluster and bulk-inserts rows. It is much slower than physical mode but is online and respects every SQL constraint, including FK validation.

Duplicate detection

pkg/lightning/duplicate/ records collisions seen during physical-mode encode. It supports two strategies:

• none — fail on the first duplicate.
• record — record duplicates to a side table for manual resolution.

Logical mode relies on TiDB's normal unique-constraint handling.

Integration with the rest of the repo

• The local backend is shared with BR restore and the DDL ADD INDEX distributed reorg path. All three produce SST files and call into TiKV's ingest API.
• IMPORT INTO (a SQL statement available since TiDB 7.x) runs Lightning logic in-process inside the TiDB server. Entry points are pkg/executor/import_into.go, pkg/executor/importer/, and pkg/lightning/importdef/.
• cmd/importer/ is a separate, smaller "importer" tool used by tests.

tidb-lightning-ctl

A small companion CLI (lightning/cmd/tidb-lightning-ctl/main.go) used to inspect or repair a Lightning import: pause/resume, show progress, force checksum, list duplicates, etc.

Tests

• Unit tests live alongside each pkg/lightning/... package.
• lightning/tests/ contains shell-driven integration tests that build a fresh cluster and exercise both backends end-to-end.

Build and run

make build_lightning
bin/tidb-lightning -config tidb-lightning.toml

End-user docs and operational tuning are at https://docs.pingcap.com/tidb/stable/tidb-lightning-overview; the in-repo README points at the same content.