CSV and JSON

Active contributors: Antoine Pitrou, Felipe Aramburu, Sutou Kouhei

The CSV and JSON readers/writers live next to each other in cpp/src/arrow/csv/ and cpp/src/arrow/json/. They share the same architectural pattern: lex the input into rows, infer or coerce types, build Arrow arrays.

CSV

Files

cpp/src/arrow/csv/
├── api.h                # Public API
├── options.{h,cc}       # ReadOptions, ParseOptions, ConvertOptions, WriteOptions
├── parser.{h,cc}        # The vectorized lexer
├── lexing_internal.h    # Per-character classifications
├── chunker.{h,cc}       # Splits the input into row-aligned chunks for parallel parsing
├── column_decoder.{h,cc}  # Decodes one chunk's column to typed bytes
├── column_builder.{h,cc}  # Builds Arrow arrays from decoded values
├── converter.{h,cc} (~32 KB)  # Per-type conversion (string → int, string → date, ...)
├── inference_internal.h  # Type inference for unknown columns
├── reader.{h,cc} (~49 KB)  # Top-level Reader: orchestrates everything
├── writer.{h,cc} (~31 KB)  # CSV writer
├── invalid_row.h        # Custom error handling for malformed rows
└── fuzz.cc, generate_fuzz_corpus.cc  # Fuzzing

Reader pipeline

graph LR
    Bytes["Read bytes from InputStream"] --> Chunker["Chunker splits at row boundaries"]
    Chunker --> Parser["BlockParser tokenizes rows in parallel"]
    Parser --> Decoder["ColumnDecoder reads one column from a block"]
    Decoder --> Builder["ColumnBuilder builds Arrow array"]
    Builder --> Output["RecordBatch / Table"]

The reader runs the pipeline on the IO and CPU thread pools. chunker.cc is the trickiest piece: it scans for unquoted line terminators that can serve as chunk boundaries so each chunk can be parsed in parallel. parser.cc (~25 KB) is a careful state machine that handles quotes, escaped quotes, embedded newlines, and trailing whitespace.

Three option bags

CSV reading takes three option bags so that each piece (block size, parsing rules, type conversion) can be tuned independently:

• ReadOptions — block size, column names from headers vs explicit, skip rows.
• ParseOptions — delimiter, quoting, escape character, newline handling, ignore-empty-lines.
• ConvertOptions — column types (or auto-infer), string → bool/null/decimal mappings, include columns, strings_can_be_null.

Type inference

When ConvertOptions::auto_dict or implicit type inference is on, inference_internal.h runs a try-cast cascade per column: try int → try float → try date/timestamp → fall back to string. Inference happens during the first chunk and the inferred types are then used for the rest of the file (with promotion if a later chunk has a larger range).

Writing

writer.cc is straightforward by comparison: serializes a RecordBatch to a CSV stream with options for delimiter, quoting, EOL.

JSON

Files

cpp/src/arrow/json/
├── api.h                # Public API
├── options.{h,cc}       # ReadOptions, ParseOptions
├── parser.{h,cc} (~40 KB)  # Streaming JSON parser (uses RapidJSON internally)
├── chunker.{h,cc}       # Chunks at object boundaries
├── chunked_builder.{h,cc}  # Builds arrays from chunks
├── from_string.{h,cc} (~40 KB)  # Convert JSON → Arrow scalars/arrays without going through a record batch
├── object_parser.{h,cc}, object_writer.{h,cc}  # Streaming key-value parser
├── converter.{h,cc}     # Scalar conversions
├── reader.{h,cc} (~23 KB)  # Top-level Reader
├── rapidjson_defs.h     # RapidJSON configuration
└── test_common.h

What's supported

The reader handles newline-delimited JSON (one record per line) — the format used in many ETL pipelines (jsonl). Each line is a JSON object whose top-level fields become columns. Nested objects and arrays become Arrow Struct/List columns.

from_string.cc is a separate path for parsing a single JSON value into an Arrow scalar/array — useful for tests, REPL use, and small payloads.

Pipeline

The pipeline mirrors the CSV reader: chunk at object boundaries (newlines outside strings), parse chunks in parallel using RapidJSON, build Arrow arrays. Type inference handles ints vs floats vs strings; nested types are inferred from the first record encountered.

What's not supported

• Pretty-printed JSON (multi-line objects). Use from_string or pre-process.
• Streaming arbitrary JSON arrays. The reader expects an object per line, not a JSON array of objects.
• A JSON writer. Arrow → JSON conversion happens through pretty_print.cc or per-language wrappers (e.g. pyarrow.Table.to_pylist() and json.dumps).

Test infrastructure

CSV: parser_test.cc (32 KB), reader_test.cc (22 KB), column_builder_test.cc (27 KB), converter_test.cc (37 KB), writer_test.cc (~21 KB).

JSON: parser_test.cc (13 KB), reader_test.cc (38 KB), from_string_test.cc (65 KB), chunked_builder_test.cc (17 KB).

Both have fuzz.cc LibFuzzer harnesses and corpus generators.

Language wrapper integration

• PyArrow. python/pyarrow/_csv.pyx (58 KB), python/pyarrow/_json.pyx (13 KB). Public APIs pyarrow.csv and pyarrow.json.
• R. r/R/csv.R (~36 KB), r/R/json.R. CSV is by far the most-used reader from R.
• C-GLib + Ruby. Both formats covered by the arrow-glib and red-arrow packages.

Entry points for modification

• Adding a new option: extend the appropriate *Options struct, plumb it through the reader/writer.
• Improving CSV throughput: parser.cc's state machine and column_decoder.cc's value-decoder are the hot paths. SIMD lexing is on the roadmap (some pieces already use vectorized character classification).
• Supporting non-newline-delimited JSON: the chunker would need to be rewritten to find object boundaries inside arrays. Discuss on the dev mailing list before tackling this.