langchain-text-splitters

A small standalone package that provides the document chunkers LangChain uses for RAG. Source: libs/text-splitters/langchain_text_splitters/. PyPI: langchain-text-splitters. Current version: 1.1.2. ~14 source modules, ~9,800 lines of Python.

Purpose

Splitting long documents into chunks that fit within model context windows is one of the most common operations in RAG pipelines. The splitters in this package are factored out of the main langchain package so they can be installed and used independently:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_text(long_text)

Directory layout

libs/text-splitters/langchain_text_splitters/
├── __init__.py
├── base.py                # TextSplitter, Tokenizer, TokenTextSplitter, Language enum
├── character.py           # CharacterTextSplitter, RecursiveCharacterTextSplitter
├── html.py                # HTMLHeaderTextSplitter, HTMLSectionSplitter, HTMLSemanticPreservingSplitter
├── json.py                # RecursiveJsonSplitter
├── jsx.py                 # JSFrameworkTextSplitter
├── konlpy.py              # Korean (KoNLPy) splitter
├── latex.py               # LatexTextSplitter
├── markdown.py            # MarkdownHeaderTextSplitter, MarkdownTextSplitter, ExperimentalMarkdownSyntaxTextSplitter
├── nltk.py                # NLTKTextSplitter
├── python.py              # PythonCodeTextSplitter
├── sentence_transformers.py
├── spacy.py               # SpacyTextSplitter
└── xsl/                   # XSLT stylesheets used by HTML splitters

Key abstractions

Symbol	File	Description
TextSplitter	libs/text-splitters/langchain_text_splitters/base.py	Base class with split_text, split_documents, create_documents
Tokenizer	libs/text-splitters/langchain_text_splitters/base.py	Tokenizer-aware splitter wrapper
TokenTextSplitter	libs/text-splitters/langchain_text_splitters/base.py	Splits by tokens (using tiktoken by default)
Language	libs/text-splitters/langchain_text_splitters/base.py	Enum with code-language-specific separator presets (Python, JS, Java, Go, Rust, Markdown, HTML, …)
CharacterTextSplitter	libs/text-splitters/langchain_text_splitters/character.py	Splits at a single separator
RecursiveCharacterTextSplitter	libs/text-splitters/langchain_text_splitters/character.py	The default for most users; recursively splits on a list of separators
MarkdownHeaderTextSplitter	libs/text-splitters/langchain_text_splitters/markdown.py	Splits by Markdown header level, preserving header context
HTMLHeaderTextSplitter, HTMLSectionSplitter, HTMLSemanticPreservingSplitter	libs/text-splitters/langchain_text_splitters/html.py	HTML-aware splitters; HTMLSemanticPreservingSplitter is ~1,400 lines because real HTML is brutal
RecursiveJsonSplitter	libs/text-splitters/langchain_text_splitters/json.py	Splits a JSON tree without breaking objects
PythonCodeTextSplitter, JSFrameworkTextSplitter, LatexTextSplitter	libs/text-splitters/langchain_text_splitters/python.py, jsx.py, latex.py	Code-aware splitters
NLTKTextSplitter, SpacyTextSplitter, KonlpyTextSplitter, SentenceTransformersTokenTextSplitter	corresponding *.py files	Tokenizer-library-backed splitters (require optional deps)

How it works

The base TextSplitter exposes:

def split_text(self, text: str) -> list[str]: ...
def split_documents(self, docs: Iterable[Document]) -> list[Document]: ...
def create_documents(self, texts: list[str], metadatas: list[dict] | None = None) -> list[Document]: ...

RecursiveCharacterTextSplitter is the default for most users. It tries a list of separators in order (["\n\n", "\n", " ", ""] by default), falling back to coarser ones until each chunk fits the configured chunk_size. It also supports chunk_overlap so that consecutive chunks share context.

For code, the Language enum carries language-specific separator presets:

splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON, chunk_size=1000
)

This uses Python-aware separators (\nclass , \ndef , \n\tdef , \n\n, \n, …).

MarkdownHeaderTextSplitter and HTMLHeaderTextSplitter work differently: they split on heading boundaries while attaching ancestor headings as metadata, so a chunk in a deeply nested section knows its full breadcrumb.

Integration points

• Used by langchain-core's documents/ module via the Document.page_content interface.
• Re-exported from libs/langchain/langchain_classic/text_splitter.py for backwards compatibility.
• Optional deps for the tokenizer-library-backed splitters: nltk, spacy, sentence-transformers, konlpy. They are not pinned by the package itself; users install whatever they need.

Entry points for modification

• To add a new splitter, create a new module under libs/text-splitters/langchain_text_splitters/ and re-export it from __init__.py's __all__.
• To add a new programming language preset, extend the Language enum in base.py and add the separator list to _get_separators_for_language.
• The xsl/ directory holds XSLT stylesheets for the HTML splitters — extend or add stylesheets there if you need different sectioning rules.

Related

• packages/core — defines the Document type that splitters operate on
• packages/langchain-classic — re-exports these splitters for legacy users