eval_cli

Active contributors: rtfeldman, benbrandt

Purpose

crates/eval_cli is a headless CLI for evaluating AI-driven features (the agent and edit prediction). It replays scripted scenarios against language models and reports outcomes, used both in CI and ad-hoc for prompt tuning.

Directory layout

crates/eval_cli/
├── Cargo.toml
└── src/
    └── main.rs        # Single-file entry point

crates/eval_utils carries shared helpers for setting up project fixtures, running the agent in eval mode, and scoring results.

How it works

graph LR
    fixtures[(Fixtures)] --> CLI[eval_cli]
    CLI -->|start| HP[Headless agent]
    HP -->|tool calls| FS[FakeFs / repo clones]
    HP -->|completion| LLM[LLM provider]
    LLM --> HP
    HP -->|results| Scorer[Scorer]
    Scorer --> Out[stdout / JSON]

The CLI usually:

1. Loads a fixture (a small repo + a task description).
2. Spins up the agent in headless mode (no GPUI window).
3. Runs the agent against a target LLM.
4. Compares produced edits or answers against expectations.
5. Emits scores.

Integration points

• Drives crates/agent and crates/edit_prediction from the outside.
• Talks to the language-model providers via crates/language_models.
• Used by CI workflows to detect regressions in AI behaviour.

Entry points for modification

• New evaluation scenario — drop a fixture in the eval data path the CLI scans.
• New scoring metric — implement in crates/eval_utils and surface via the CLI flags.

Related pages

• AI agent
• Edit prediction