Encoding (M17N)

Ruby's strings are encoding-aware: every String carries an Encoding, and operations like length, chars, regex matching, and case conversion respect that encoding. This is called M17N (multilingualisation, "M" + 17 letters + "N"). M17N landed in Ruby 1.9 and has been quietly central ever since.

Purpose

• Let Ruby strings represent text in any of ~100 supported encodings without forcing UTF-8 conversion.
• Provide explicit transcoding via String#encode, IO#set_encoding, and Encoding::Converter.
• Make encoding-aware operations (length in characters, regex matching, case folding, comparison) work correctly without programmer effort.

Files

File	Purpose
encoding.c	The Encoding class, encoding registry, ASCII-compat helpers. ~52 KB.
transcode.c	String#encode and Encoding::Converter. ~145 KB.
enc/	Per-encoding tables and methods.
enc/encdb.c	The encoding database.
enc/<name>.c	Code for each encoding (utf_8.c, shift_jis.c, euc_jp.c, ...).
enc/trans/	Transcoding tables (Shift-JIS ↔ UTF-8, etc.).
enc/unicode/data/	Unicode Character Database tables (case folding, normalization).
encindex.h	Compile-time index numbers for built-in encodings.
tool/transcode-tblgen.rb	Generates enc/trans/*.c from .trans source.
tool/enc-unicode.rb	Generates the Unicode tables in enc/unicode/data/.

Encodings as objects

Every encoding is a singleton instance of Encoding:

Encoding::UTF_8       # frozen, single instance per encoding
Encoding::Shift_JIS
Encoding::ASCII_8BIT  # the binary / "no encoding" sentinel

Internally each encoding is an OnigEncodingTypeST (the regex engine's encoding type) extended with extra fields:

struct rb_encoding {
    OnigEncodingType *base;
    rb_encoding **prev_official_encoding;
    /* + flags: ASCII-compat, dummy, fixed-width, ... */
};

The encoding's "ASCII-compatible" flag means bytes 0x00–0x7F are valid one-byte characters and have ASCII meaning. Most encodings have it; UTF-16/UTF-32 don't.

A String stores its encoding as an integer index into a global encoding table (encoding.c::enc_table). RSTRING_LEN(s) gives bytes; rb_str_strlen(s) (or the Ruby-level String#length) decodes characters.

ASCII-only flag

A string can carry an "ASCII-only" flag separate from its encoding:

• Source: any string whose bytes are all <= 0x7F.
• Behaviour: treated as compatible with any ASCII-compatible encoding.

This lets "hello" + japanese_string work without manual coercion: the literal is ASCII-only, so it inherits the encoding of the right operand.

Encoding compatibility

encoding.c::enc_compatible_latter decides what encoding two strings produce when concatenated:

graph LR
    A[Both same encoding] --> A1[Use it]
    B[ASCII-only + ASCII-compat] --> B1[Use the ASCII-compat one]
    C[ASCII-8BIT + ASCII-only] --> C1[Use the ASCII-compat one]
    D[Otherwise] --> D1[Encoding::CompatibilityError]

Encoding.compatible?(a, b) exposes this from Ruby.

Transcoding

String#encode(target) uses Encoding::Converter (transcode.c):

• For pairs that have a direct conversion table (Shift-JIS → UTF-8), one lookup per character.
• For pairs without a direct table, an intermediate UTF-8 hop ("Universal Newline" mode also routes through this).
• Options: :invalid => :replace, :undef => :replace, :replace => '?', :newline => :universal, etc.

Encoding::Converter is a streaming API:

ec = Encoding::Converter.new("Shift_JIS", "UTF-8")
src = +"...".force_encoding("Shift_JIS")
dst = +""
ec.primitive_convert(src, dst)

The converter holds a state machine; primitive_convert returns one of :source_buffer_empty, :destination_buffer_full, :incomplete_input, :undefined_conversion, :invalid_byte_sequence, :finished.

The conversion graph is described declaratively; transcode.c::transcode_search_path performs Dijkstra over the graph to find the shortest conversion sequence between two arbitrary encodings.

Built-in encodings

The first ~few dozen encodings are statically compiled in (encindex.h):

enum ruby_preserved_encindex {
    RUBY_ENCINDEX_ASCII_8BIT = 0,
    RUBY_ENCINDEX_UTF_8,
    RUBY_ENCINDEX_US_ASCII,
    /* + ~30 more for common European/Asian encodings */
};

These are guaranteed to be present and have stable index numbers (used by C code as a fast path).

The rest of the encodings are loaded on demand via Encoding.find("name"), which dynamically opens enc/<name>.so.

Magic comments

A Ruby file's encoding can be set per-file with a magic comment on the first or second line:

# encoding: UTF-8
# frozen_string_literal: true

parse.y and prism/ both consume these. The default since 2.0 is UTF-8.

Default external/internal encodings

Encoding.default_external  # encoding for IO from disk/network
Encoding.default_internal  # if set, IO transcodes to this encoding

Set at startup from LANG/LC_* (Linux), chcp (Windows), or via -E / --external-encoding=/--internal-encoding= flags. The defaults change the behaviour of File.read, ARGV, ENV, etc.

Onigmo integration

The regex engine (Onigmo, see regexp.md) is encoding-aware: every Regexp carries an encoding, and matching uses the engine's per-encoding character class tables. enc/<name>.c provides those tables (Unicode case-folding, character categorization, etc.) for each supported encoding.

Common pitfalls

• Mixing UTF-8 and binary: appending a UTF-8 String to an ASCII-8BIT String raises Encoding::CompatibilityError if either has non-ASCII bytes. Use force_encoding carefully.
• force_encoding is a no-op tag change: it changes the encoding label without converting bytes. To actually convert, use encode.
• String#bytes vs String#chars: bytes are always raw; chars are encoding-decoded.
• Source file encoding mismatch: a UTF-8 source file with an # encoding: shift_jis magic comment will mis-decode literals.

Entry points for modification

• Add a new encoding: drop a new file under enc/, register it in enc/encdb.c, and add transcoding tables under enc/trans/. Run the build (make srcs).
• Fix a string operation: most string-encoding code is in string.c and encoding.c. The pattern is to switch on rb_enc_asciicompat(enc) and rb_enc_str_asciionly_p(s).
• Update Unicode: bump the Unicode version in tool/enc-unicode.rb's URL list, run make update-unicode, regenerate enc/unicode/data/.

See systems/regexp.md for the regex engine and core-classes/string-and-symbol.md for String itself.