s3select

internal/s3select/ is MinIO's S3 Select implementation: a SQL evaluator that runs over CSV, JSON, and Parquet payloads inside the server, returning only the rows/columns the client asked for.

Layout

internal/s3select/
├── select.go                    # Top-level entry: parse XML request, build pipeline
├── csv/                         # CSV parser (uses minio/csvparser)
├── json/                        # JSON line and document parsers (uses simdjson-go)
├── parquet/                     # Parquet reader (fraugster/parquet-go)
├── sql/                         # SQL parser (alecthomas/participle)
├── simdj/                       # simdjson glue
├── jstream/                     # Streaming JSON helper
└── unused-errors.go

How it works

graph LR
    REQ[POST /?select XML] --> P[select.New]
    P --> SQL[sql.Parse]
    P --> R{format?}
    R -- csv --> CSV[csv.Reader]
    R -- json --> JSON[json.Reader]
    R -- parquet --> PARQUET[parquet.Reader]
    SQL --> EVAL[Per-record evaluator]
    EVAL --> WRITE[CSV/JSON output framer]
    WRITE --> CLIENT

• The HTTP layer (cmd/object-handlers.go) detects an S3 Select request and forwards to internal/s3select.New.
• The XML body specifies the query, input format, output format, and progress reporting cadence.
• The reader streams the object payload from xl-storage and feeds the evaluator one record at a time.
• The evaluator emits matching records back to the client, framed in either CSV or JSON.

SQL features

The supported subset (defined in sql/) includes:

• SELECT with column projection.
• WHERE predicates with most comparison operators.
• LIMIT, OFFSET.
• A small set of built-in functions (SUBSTRING, CAST, LOWER, UPPER, TRIM, ...).

Joins, group-by, and aggregations beyond COUNT(*) are not supported.

Format-specific notes

• CSV. Header detection, custom delimiters, quote handling. Uses github.com/minio/csvparser.
• JSON. Document and line-oriented variants. simdjson-go is used when the payload is small enough; otherwise a streaming parser falls back.
• Parquet. Column projection is pushed down so unused columns are skipped at the file level.

Integration points

• Driven entirely by cmd/object-handlers.go (the SelectObjectContent handler).
• Reads from xl-storage via ObjectLayer.GetObject.
• Encryption is transparent: the handler decrypts the payload before passing to the format reader.

Entry points for modification

• New SQL function. Add it under internal/s3select/sql/. Watch test coverage in the same directory.
• New format. Mirror the existing format directories; the reader interface is small.
• Performance. Profile against simdj and Parquet; the bottleneck is usually disk I/O, not the evaluator.