“Our AI feature breaks whenever we change a model or a prompt.”
Last verified: October 2026· engagement
Every prompt tweak or model upgrade is a blind deploy, and regressions are found by customers. The fix is an evaluation suite built from real traffic, a release gate that blocks regressions in CI, and monitoring that catches drift in production — sized to the failure classes that actually hurt you.
Is it worth fixing?
It usually is when several of these are true:
- You change prompts, models or retrieval often enough that regressions keep reaching users.
- A model deprecation or upgrade is forcing a change you cannot currently test.
- Nobody can say whether last month’s change made answers better or worse.
- The feature matters enough — revenue, support load, compliance — that a silent regression is expensive.
The decision, one question at a time
Each step links to the page that answers it. You can get most of the way without talking to anyone.
- 1
- 2
- 3
- 4
- 5
What a first engagement looks like
Free read
A written read on the single eval you are most exposed without, from a description of the feature and how you test it today.
Paid diagnostic
A fixed-price failure-analysis sprint: a labelled sample of real traffic, the failure classes that matter, and a sized plan for the gate. Fixed price, quoted after the free read.
Stop or go
If changes are rare and failures are cheap, we recommend a lightweight golden set you can run yourselves instead.
The accepted outcome is the test above: a release gate and production monitoring that catch the failure classes agreed at kickoff, proven by catching an injected regression.The full acceptance criteria, exclusions and pricing terms are written out onthe engagement page, so you and procurement can see the finish line before any call.
Access and security
- A sample of production traffic for the feature, redacted to your policy.
- CI access to add the gate; no production credentials needed to assess fit.
When we’d tell you not to buy
Our scarce resource is senior engineering attention, so we would rather say no early. We decline or refer when:
- Changes are rare and failures cheap — a small golden set run by hand is enough.
- No production traffic can be sampled, even redacted.
- The ask is to certify the model as accurate rather than to catch regressions.
Not there yet? Start withHow to build LLM evals, LLM observability or You shipped AI with no evals.
The engagements behind it
Questions
How big does the eval suite need to be?
Big enough to cover the failure classes that cost you, and no bigger. That is usually tens to low hundreds of examples per class, drawn from real traffic.
Will this let us switch models safely?
That is the main use. With a gate in CI, a model or prompt change has to clear the same bar as the version it replaces.
Do you fix the failures the evals find?
We prioritise them and can scope fixes separately. The engagement itself delivers the gate and the monitoring.
Related guides
How to build LLM evals
Evals turn 'the model feels better' into a number you can gate a deploy on — they are the test suite that makes every prompt, model, and pipeline change safe to ship.
LLM evalsLLM observability: what to instrument in production
LLM observability is the practice of recording enough about every model call that you can answer, after the fact, what it cost, how long it took, whether the output was any good, and what the model was actually shown. Standard APM answers none of those, because a call that returns HTTP 200 in 900ms can still be a complete failure.
LLM evalsWhat is RAG (retrieval-augmented generation)?
RAG retrieves the passages relevant to a question and puts them in the prompt, so the model answers from your data with citations instead of relying on what it happened to memorise during training.
Find out whether it’s worth fixing
A few structured questions, no call required. You get a written decision brief back — including whether we think you should buy anything at all.
Start the diagnostic