Generation

Purpose

generate is the universal autoregressive decoding entry point for causal-LM, encoder-decoder, image-to-text, audio-to-text, and any other model that produces a sequence one token at a time. It supports greedy, beam, contrastive, sample, assisted (speculative), and DoLa decoding, plus continuous batching for production serving.

Key abstractions

Class / function	File	Role
GenerationMixin	src/transformers/generation/utils.py (3,887 LOC)	Adds generate() to model classes
GenerationConfig	src/transformers/generation/configuration_utils.py (102K LOC)	Decoding hyperparameters
LogitsProcessorList, LogitsProcessor	src/transformers/generation/logits_process.py (150K LOC)	Mutate logits each step
StoppingCriteriaList, StoppingCriteria	src/transformers/generation/stopping_criteria.py (29K LOC)	Decide when to stop
BaseStreamer, TextStreamer, AsyncStreamer	src/transformers/generation/streamers.py	Push tokens to consumers
CandidateGenerator, AssistedCandidateGenerator	src/transformers/generation/candidate_generator.py (66K LOC)	Speculative / assisted decoding
Watermarker	src/transformers/generation/watermarking.py	Output watermarking
ContinuousBatchingScheduler	src/transformers/generation/continuous_batching/	Mix prefill & decode across requests
Cache hierarchy	src/transformers/cache_utils.py (1,574 LOC)	KV cache state — see Cache

How generate runs

graph TD
    Start[generate kwargs] --> Conf[Resolve GenerationConfig]
    Conf --> Strat{Decoding strategy}
    Strat -->|greedy / sample| Loop1[Token-by-token loop]
    Strat -->|beam / group-beam| Loop2[Beam search loop]
    Strat -->|contrastive| Loop3[Contrastive search]
    Strat -->|assisted| Loop4[Assisted decoding loop]
    Strat -->|DoLa| Loop5[DoLa loop]
    Loop1 --> Step[Forward pass + KV cache update]
    Loop2 --> Step
    Loop3 --> Step
    Loop4 --> Step
    Loop5 --> Step
    Step --> LP[LogitsProcessorList]
    LP --> Sample[Sample / argmax / beam expand]
    Sample --> Stop{StoppingCriteria}
    Stop -->|continue| Step
    Stop -->|done| End[Return GenerationOutput]

Decoding strategies

Strategy	Trigger	Use case
Greedy	do_sample=False, num_beams=1	Deterministic
Sampling	do_sample=True	Diverse outputs
Beam search	num_beams > 1	Higher-quality summarization/translation
Group-beam search	num_beam_groups > 1	Diverse beams
Contrastive search	penalty_alpha > 0, top_k > 1	Reduce repetition
Assisted (speculative)	assistant_model=...	Accelerate using a smaller draft model
DoLa	dola_layers=...	Decoding by Contrasting Layers
Watermarked	watermarking_config=...	Statistical output marking

Most strategies share the per-step machinery and differ only in how candidates are scored and selected.

Logits processors

logits_process.py is the second-largest file in the library (150K LOC). Each processor is a callable on (input_ids, scores) and returns modified scores. The most-used:

• MinLengthLogitsProcessor, MinNewTokensLengthLogitsProcessor — force min length.
• TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper, TypicalLogitsWarper, MinPLogitsWarper, EpsilonLogitsWarper, EtaLogitsWarper — sampling distortions.
• RepetitionPenaltyLogitsProcessor, EncoderRepetitionPenaltyLogitsProcessor, NoRepeatNGramLogitsProcessor.
• BadWordsLogitsProcessor, ForcedBOSTokenLogitsProcessor, ForcedEOSTokenLogitsProcessor.
• SuppressTokensLogitsProcessor, SuppressTokensAtBeginLogitsProcessor — Whisper-style.
• LogitNormalization, EncoderNoRepeatNGramLogitsProcessor.
• ClassifierFreeGuidanceLogitsProcessor — for some VLMs.
• Schema-guided: JsonSchemaConstrainedLogitsProcessor, RegexConstrainedLogitsProcessor.

The order of processors matters; GenerationConfig builds the list deterministically.

Stopping criteria

• MaxLengthCriteria, MaxNewTokensCriteria, MaxTimeCriteria.
• StopStringCriteria — stops when a literal string is produced.
• EosTokenCriteria — stops on EOS (multiple EOS ids supported).
• ConfidenceCriteria — stops when probability of selected token drops below a threshold.

Caches and generate

For autoregressive decoding to be fast, generation reuses a Cache (src/transformers/cache_utils.py) across iterations. generate instantiates the right cache based on config.cache_implementation:

• dynamic (default) — DynamicCache.
• static — StaticCache, required for torch.compile.
• sliding_window, hybrid — for sliding-window or hybrid attention models.
• quantized, offloaded — memory savers.

See Cache.

Continuous batching

src/transformers/generation/continuous_batching/ adds production-grade serving. The scheduler interleaves prefill steps and decode steps from many concurrent requests in a single forward pass, using a paged KV cache. It is what powers transformers serve --continuous-batching (CLI).

The PR that introduced this (#40426, Aug 2025) added cb_block_size, cb_num_blocks, cb_max_batch_tokens knobs.

Streaming

from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
streamer = TextStreamer(tok)
model.generate(**tok("Hello", return_tensors="pt"), streamer=streamer, max_new_tokens=128)

For async use cases, AsyncTextStreamer exposes an async for interface used by transformers serve.

Integration points

• All causal-LM and seq2seq classes mix in GenerationMixin.
• Pipeline subclasses for text generation, ASR, image-to-text, etc., delegate to model.generate.
• transformers serve (src/transformers/cli/serve.py) wraps generate in an OpenAI-compatible HTTP server.
• Trainer does not call generate during training, but Seq2SeqTrainer.predict does for evaluation.

Entry points for modification

• New decoding strategy → add a method to GenerationMixin and route from _get_generation_mode. Update tests in tests/generation/.
• New logits processor → add a class to logits_process.py, register it in GenerationConfig if it is a configurable knob.
• New cache → add a class in cache_utils.py and register in CACHE_MAP.
• For cache + attention combinations, also update the relevant entry in src/transformers/integrations/*_paged.py.