Regression testing (for LLMs)
Last verified: June 2026· definition
Regression testing for LLM systems means running your eval suite automatically whenever a prompt, model, or retrieval component changes — and blocking the change if quality drops, just like a failing unit test blocks a merge.
It’s what keeps cost optimizations safe: you can downgrade a model on a route only because the regression suite proves quality held.
Wired into CI/CD, regression evals turn “did this change hurt quality?” into a question answered before deploy instead of after a customer notices.
Why LLM systems regress silently
Conventional software fails loudly: a broken change throws an error or fails a test. An LLM system that regresses keeps returning fluent, well-formed answers that are simply worse — less accurate, less grounded, subtly off-format. Nothing crashes and no exception surfaces. This is why regression testing matters more here than in ordinary software rather than less: without it, quality degradation is invisible until users notice, and by then attributing it to a specific change is difficult.
The triggers are broader than code changes
A prompt edit, a model version change, a retrieval tuning adjustment, a chunking change, or a provider updating a model behind a stable name can all move quality without a deploy in the conventional sense. Regression suites need to run on all of these, which means treating prompts and retrieval configuration as versioned artefacts that trigger the suite. Teams running evals only on application code changes miss the majority of the causes.
Fixed datasets and production sampling do different jobs
A fixed dataset gives you a stable comparison point — the same inputs every run, so a score change means a real change. Production sampling catches drift and inputs your dataset never anticipated. A suite with only the first passes reliably while the real distribution moves away from it; one with only the second has no stable baseline to compare against. Mature setups run both, and treat divergence between them as its own signal.
Gate on the metrics you trust
A regression suite is only useful if a failure blocks something. That requires deciding in advance which metrics gate a deploy and what threshold constitutes a regression rather than noise. Suites that report without gating tend to be watched attentively for a few weeks and then ignored, which is worse than not having them, because the appearance of coverage discourages other caution.
Common misconceptions
MythRegression testing is the same as having evals.
RealityEvals measure quality; regression testing is running them systematically on every change so degradation is caught automatically. A suite that exists but runs manually catches very little.
MythOnly prompt changes need re-testing.
RealityModel updates, retrieval configuration, chunking, and provider-side changes to a model behind a stable name all move quality without touching your prompts.
MythSmall score movements should be investigated.
RealitySampling and model non-determinism produce noise. Without an established threshold separating real regression from variance, teams either chase noise or learn to ignore the suite entirely.
Frequently asked questions
What should trigger a regression run?
Any change that can move quality: prompt edits, model version changes, retrieval or chunking configuration, tool definitions, and application code touching how requests are assembled. In practice the cleanest approach is treating prompts and retrieval config as versioned artefacts in the repository so a change to any of them runs the suite automatically, the same way a code change does.
How do I distinguish a real regression from noise?
Establish the noise floor first. Run the suite several times against an unchanged system and observe how much scores vary from sampling and model non-determinism alone. That variance sets the threshold below which a movement is not meaningful. Without it teams either investigate every fluctuation or dismiss genuine regressions as noise, and both erode trust in the suite.
How large should a regression suite be?
Large enough to cover the cases that matter and small enough to run on every change without anyone resenting it. A suite that takes too long or costs too much gets moved to a nightly run, then to a weekly one, then to nobody looking. Starting with a focused set on the highest-stakes routes and expanding as it proves useful works better than aiming for coverage first.
Go deeper
How to build LLM evals
Evals turn 'the model feels better' into a number you can gate a deploy on — they are the test suite that makes every prompt, model, and pipeline change safe to ship.
LLM evalsLLM observability: what to instrument in production
LLM observability is the practice of recording enough about every model call that you can answer, after the fact, what it cost, how long it took, whether the output was any good, and what the model was actually shown. Standard APM answers none of those, because a call that returns HTTP 200 in 900ms can still be a complete failure.