I/O and filesystem

Active contributors: Antoine Pitrou, Tianchen Ding, Sutou Kouhei, Felipe Aramburu

The I/O layer is split into two complementary subsystems:

• cpp/src/arrow/io/ defines the abstract stream interfaces (InputStream, OutputStream, RandomAccessFile) and concrete implementations (memory, file, buffered, compressed, slow-wrapper).
• cpp/src/arrow/filesystem/ defines arrow::fs::FileSystem — a higher-level abstraction with semantics like "list directory", "create directory", "delete file", "open input stream", and ships local-disk + cloud implementations.

I/O streams

Interfaces

cpp/src/arrow/io/interfaces.h defines the base classes:

class InputStream {
  virtual Status Close();
  virtual Result<int64_t> Read(int64_t nbytes, void* out);
  virtual Result<std::shared_ptr<Buffer>> Read(int64_t nbytes);
  virtual bool closed() const = 0;
  ...
};

class OutputStream {
  virtual Status Close();
  virtual Status Write(const void* data, int64_t nbytes);
  virtual Result<int64_t> Tell();
  ...
};

class RandomAccessFile : public InputStream, public Seekable {
  virtual Result<int64_t> GetSize();
  virtual Result<int64_t> ReadAt(int64_t position, int64_t nbytes, void* out);
  virtual Result<std::shared_ptr<Buffer>> ReadAt(int64_t position, int64_t nbytes);
  virtual Future<std::shared_ptr<Buffer>> ReadAsync(IOContext, int64_t, int64_t);
  ...
};

Concrete implementations

File	Class	Purpose
io/memory.cc	BufferReader, BufferOutputStream, MockOutputStream, FixedSizeBufferWriter	In-memory streams. BufferReader is the universal "read from a Buffer" entry point.
io/file.cc	ReadableFile, MemoryMappedFile, FileOutputStream, OSFile	Local file streams via read/write/mmap.
io/buffered.cc	BufferedInputStream, BufferedOutputStream	Adds a fixed-size in-memory buffer to any stream.
io/compressed.cc	CompressedInputStream, CompressedOutputStream	Wraps a stream with a compression codec from cpp/src/arrow/util/compression.h.
io/caching.cc	RangeCache, ReadRangeCache	Coalesces small reads into bigger ones. Used by Parquet.
io/hdfs.cc	HdfsReadableFile, HdfsOutputStream	Hadoop FS via libhdfs. (Largely supplanted by filesystem/hdfs.cc.)
io/slow.cc	SlowInputStream, SlowRandomAccessFile	Latency injectors for testing.
io/transform.cc	TransformInputStream	Apply a callback to bytes as they pass through.
io/stdio.cc	StdoutStream, StderrStream, StdinStream	Reading from / writing to standard FDs.

Concurrency

cpp/src/arrow/io/concurrency.h provides internal::Concurrency helpers — RAII wrappers that ensure thread safety for streams that aren't thread-safe by default. The IOContext carries an io thread pool reference; readers like the dataset scanner schedule async reads through it.

Filesystem

cpp/src/arrow/filesystem/filesystem.h defines arrow::fs::FileSystem — the higher-level abstraction. Subclasses implement:

class FileSystem {
  virtual Result<FileInfo> GetFileInfo(const std::string& path);
  virtual Result<std::vector<FileInfo>> GetFileInfo(const FileSelector&);
  virtual Status CreateDir(const std::string&, bool recursive = true);
  virtual Status DeleteDir(const std::string&);
  virtual Status DeleteFile(const std::string&);
  virtual Status Move(const std::string&, const std::string&);
  virtual Status CopyFile(const std::string&, const std::string&);
  virtual Result<std::shared_ptr<io::InputStream>> OpenInputStream(const std::string&);
  virtual Result<std::shared_ptr<io::RandomAccessFile>> OpenInputFile(const std::string&);
  virtual Result<std::shared_ptr<io::OutputStream>> OpenOutputStream(const std::string&);
  virtual Result<std::shared_ptr<io::OutputStream>> OpenAppendStream(const std::string&);
};

Concrete implementations

File	Class	Notes
filesystem/localfs.cc	LocalFileSystem	Local POSIX filesystem. Defaults to use mmap for files large enough.
filesystem/s3fs.cc (~137 KB)	S3FileSystem	AWS S3 (and S3-compatible: MinIO, Wasabi, Ceph). Uses the AWS C++ SDK.
filesystem/gcsfs.cc (~38 KB)	GcsFileSystem	Google Cloud Storage. Uses google-cloud-cpp.
filesystem/azurefs.cc (~152 KB — the largest single source file)	AzureFileSystem	Azure Blob Storage + Data Lake Gen2. Uses the Azure SDK.
filesystem/hdfs.cc	HadoopFileSystem	HDFS via the existing libhdfs integration.
filesystem/mockfs.cc	MockFileSystem	In-memory filesystem for tests.

Each cloud filesystem is heavily configurable: anonymous access, role-based credentials, custom endpoints, regional overrides, request timeouts, retry policies, and many filesystem-specific knobs (e.g. S3's multipart upload thresholds, GCS's bucket-level metadata).

URIs and registry

cpp/src/arrow/filesystem/filesystem.cc implements FileSystemFromUri — parses a URI like s3://bucket/key or gs://bucket/key and returns the appropriate FileSystem instance. The registry is extensible via FileSystemRegistry so applications can register their own schemes.

The dataset scanner (cpp/src/arrow/dataset/) and Parquet reader use the FileSystem abstraction so that the same code reads from local disk, S3, GCS, Azure, or HDFS without modification.

Performance

S3 in particular is heavily optimized:

• Coalesced range reads via io/caching.cc to avoid per-page round trips.
• Parallel multipart uploads (s3fs.cc uses Aws::Transfer::TransferManager).
• Read-ahead via the IO thread pool.
• Configurable connection pooling.
• Optional CRC32C checksumming.

cpp/src/arrow/filesystem/s3fs_benchmark.cc ships benchmarks against MinIO. cpp/src/arrow/filesystem/s3fs_narrative_test.cc is a manual end-to-end test against real S3.

Test infrastructure

• s3fs_test.cc (~67 KB) tests against a local MinIO instance launched via compose.yaml's minio service.
• gcsfs_test.cc (~62 KB) uses gcs-server (the Google Cloud Storage testbench).
• azurefs_test.cc (~138 KB) uses Azurite — Microsoft's Azure storage emulator.
• localfs_test.cc (~22 KB) covers the local filesystem.

Key abstractions cheat-sheet

Need	Use
Read from a byte buffer	io::BufferReader
Read from a local file	io::ReadableFile::Open
Read with mmap	io::MemoryMappedFile::Open
Read from S3	arrow::fs::S3FileSystem(...).OpenInputFile(...)
Coalesced range reads	io::ReadRangeCache (Parquet uses this)
Buffer a slow stream	io::BufferedInputStream::Create
Compress on write	io::CompressedOutputStream::Make

Entry points for modification

• Adding a new filesystem: subclass arrow::fs::FileSystem, implement the virtual methods, register it via RegisterFileSystemFactory. Each existing filesystem is a useful template.
• Adding a new compression codec: extend arrow::util::Codec (cpp/src/arrow/util/compression.h) and register in compression.cc.
• Tuning S3 / GCS / Azure performance: each has an *Options struct in its *.h. Look at S3Options, GcsOptions, AzureOptions.