LLM observability: what to instrument in production

Why your existing APM does not cover this

Application performance monitoring was built for systems where success is legible from the outside. A request either errors or it does not; a query is either slow or it is not. Model calls break that assumption in a way that is easy to underestimate. The single most important property of an LLM response — whether the answer was correct, relevant, and safe — is invisible to every layer of your existing stack. A hallucinated citation, a refusal, a JSON blob with the right shape and the wrong contents, and a perfect answer are indistinguishable to an HTTP status code. Your dashboards will be green through an outage that only your users can see.

Cost behaves differently too. Conventional infrastructure cost is roughly a step function of capacity: you provision, and the bill is flat until you provision again. LLM cost is a linear function of usage measured in tokens, so it moves continuously with traffic, prompt length, and product decisions nobody labelled as a cost decision. Adding three lines of instructions to a system prompt raises the marginal cost of every call from then on. Because the provider bills at the account level, not the feature level, the connection between that edit and the invoice is invisible unless you record it yourself at the moment of the call.

The third gap is causal. When latency rises in a normal service, the trace tells you which downstream call got slower. When an agent loop gets slow or expensive, the cause is usually structural — it re-read the same context on every step, or retried after a timeout that a shorter prompt would have avoided — and you cannot see that from aggregate metrics. You need the individual, reconstructable story of one request, including how many model calls it made and what each one was shown. That is a tracing problem, and it is the reason LLM observability is a distinct discipline rather than a dashboard you bolt onto what you already run.

The four signals worth capturing

Cost is the easiest to get right and the most commonly botched. Record input and output tokens separately, because they are priced differently and output usually dominates; record reasoning or thinking tokens as their own field where the provider reports them, since they bill as output but behave like an internal cost you can sometimes turn down. Record cached versus uncached input tokens as distinct counters — conflating them makes a prompt-caching win look like no change at all. Compute the money figure at query time from the recorded token counts and the model identifier, never at write time, or every historical row becomes wrong the day prices change.

Latency needs more than one number. Time to first token and total completion time answer different questions: the first governs whether a streaming interface feels responsive, the second governs throughput and timeout budgets. Record both, and record queue or retry time separately from provider time, otherwise a rate-limit backoff shows up as the model being slow. For agent loops, the number that actually matters to a user is wall-clock time for the whole task, which no single call reports.

Quality is the signal teams skip because it looks unmeasurable, and it is the one that justifies the rest. You will not score every response, and you do not need to. Capture the cheap structural signals on everything — did the output parse, did it satisfy the schema, did it come back empty, did the model refuse, did it hit the token ceiling and truncate — and sample a small fraction of traffic for expensive scoring, whether that is a judge model or human review. Capture implicit user signals too, since a retry, an edit, an abandonment, or a thumbs-down is a quality label your users are handing you for free.

Safety covers the things that are not about correctness at all: whether a tool call was attempted outside its allowed scope, whether retrieved content contained instruction-like text, whether output tripped a policy filter, whether a response contained something that looks like a secret. These events are rare, which is exactly why they need to be recorded as first-class fields rather than buried in a log line. You want to be able to ask "has this ever happened" and get a trustworthy answer.

What a single span should carry

Start with the fields you will want to group by, because a field you did not record is a question you cannot ask later. The most valuable of these is a stable route identifier: a short, human-chosen name for the logical job this call performs, such as support.summarise_thread or ingest.classify_document. It must not be derived from the prompt text, or it will change when someone edits a word and your historical comparison silently breaks. Nearly every attribution question a finance or engineering conversation produces is answerable with a group-by on this one field, and unanswerable without it.

Then record the model identifier including its version or snapshot, the sampling parameters, and the prompt or template version. Model behaviour changes between snapshots, and "quality dropped last Tuesday" is only debuggable if you can see that the pinned version moved. Record the resolved prompt — the actual text sent, after templating and retrieval — subject to the redaction rules below, because the template plus the variables is not the same thing as what the model saw, and the difference is where most bugs live. Record the finish reason, which distinguishes a complete answer from one cut off at the token limit, and record whether the call was a retry and of what.

For anything agentic, the individual call is the wrong unit. Group calls under a trace that represents the whole task, with parent and child relationships, so you can see that a task made fourteen model calls and that eleven of them re-sent the same growing context. Record the step index and the tool name on tool calls. This is where the quadratic cost pattern in naive agent loops becomes visible: the tokens per step climb steadily, and once you can see it on a chart the fix is usually obvious.

Redaction: decide before you switch logging on

Prompt and completion text is the most useful thing in your trace store and the most dangerous. It routinely contains customer records, support ticket contents, source code, internal documents, and credentials that ended up in a payload nobody audited. A trace store is typically far more widely readable inside a company than the production database the data came from, and it is often a third-party SaaS. Turning on full payload capture without a decision about this is how teams accidentally create a second, less-governed copy of their sensitive data.

Make redaction a property of the instrumentation layer, applied before anything leaves the process, rather than a cleanup job on the store. Practically, that means a small set of choices per route: capture full text, capture a hash plus a length, capture a structural summary such as which template and which retrieval document identifiers were used, or capture nothing. Many teams land on full capture in development and structural capture in production for regulated surfaces, which preserves most of the debugging value. Where you do capture text, run detectors for the specific things you know are catastrophic — keys, tokens, national identifiers — and accept that a detector is a mitigation rather than a guarantee.

Set retention deliberately and separately from your other telemetry. Traces are useful for debugging for days and for eval-building for weeks; keeping raw prompt text for a year is almost never justified by its value, and is very easy to justify against you later. A common shape that works: short retention on full payloads, long retention on the numeric and categorical fields, which are the ones your cost and latency analysis actually needs.

Keeping the telemetry from becoming its own problem

Observability data for LLM systems is unusually heavy, because the interesting payload is text measured in kilobytes rather than a handful of numbers. A moderately busy feature can generate more bytes of trace than of application data, and the bill for storing and indexing it can become a meaningful fraction of the model spend it was meant to control. The instinct to capture everything forever is the same instinct that made the original problem, pointed at a different vendor.

The shape that holds up is tiered. Capture the numeric and categorical fields — tokens, latency, model, route, finish reason, error class — on one hundred percent of traffic, because they are small, and because aggregates built on a sample are exactly the thing people stop trusting during an incident. Sample the heavy payload text, but sample it non-uniformly: keep everything that errored, everything that tripped a safety signal, everything that a user reacted badly to, and a small random slice of the rest. Random-only sampling reliably discards the traces you most want.

Watch cardinality, which is the usual way a metrics bill explodes. A route identifier drawn from a fixed, code-defined list is fine; a label built from a user identifier, a document identifier, or any part of the prompt is not, and will produce millions of series. Keep high-cardinality facts in the trace, where they are stored once, and keep metric labels to a small, enumerable set.

Closing the loop into evals

Observability that only produces dashboards is worth much less than observability that produces test cases. The most valuable artefact your production traffic generates is a stream of real inputs your system handled badly, and those are precisely the inputs your offline eval suite lacks, because nobody imagines their own blind spots when writing test cases by hand. The loop to build is simple to state and rare to find: an interesting production trace becomes a candidate eval case, gets a reviewed expected answer, and joins the suite that runs in CI.

Define "interesting" narrowly enough to be actionable. Parse failures, refusals, truncations, safety events, unusually expensive completions for a route, and anything a user retried or corrected are all strong candidates. Give someone a queue of these and a way to promote a case with one click, or the loop will not survive contact with a busy week. The suite that results has a property hand-written evals never do: every case is something that actually happened to a real user, so a regression on it is a regression that matters.

This is also how you make a cost reduction safe. The reason teams are frightened to route a route to a cheaper model is that they have no way to know whether quality held. With a suite built from real traffic and a judge or rubric that has been checked against human labels, the downgrade becomes a measurable experiment rather than a leap of faith. The observability layer, the eval suite, and the cost work are not three projects; they are one, and doing them in that order is what makes the third one possible.

What to alert on, and what to leave on a dashboard

Most LLM metrics are diagnostic rather than actionable, and paging on them produces alert fatigue quickly. The test for an alert is whether a human would do something differently in the next hour on receiving it. Provider error rate and sustained rate-limiting pass that test. So does a collapse in cache hit rate, which usually means someone made a prompt prefix dynamic and just multiplied the cost of that route. So does a spike in parse or schema failures, which usually means a model version moved under you or a prompt edit went out. So does any safety event, because the right volume for those is zero and you want to know immediately.

Spend rate deserves an alert but needs care in how it is expressed. An absolute daily threshold fires on every successful launch and gets muted within a fortnight. Cost per unit of work — per conversation, per document, per resolved ticket — is the stable version: it stays flat as traffic grows and moves only when something structural changed, which is exactly when you want to hear about it. Alert on a sustained change in that ratio, not on the raw total.

Leave the rest on a dashboard reviewed on a cadence: token distribution by route, latency percentiles, model mix, sampled quality scores, the ratio of tool calls to completions in agent paths. These are for the weekly conversation about where the system is drifting, not for the pager. The discipline of keeping that separation is what makes the pager credible when a genuine incident does fire.

Limitations and honest caveats

Observability tells you what happened, not what should have happened. It will show you that a route is expensive, that a model refuses on a class of input, that quality scores dipped after a deploy. It will not tell you whether the feature should exist, whether the prompt is the right prompt, or what the correct answer was. Teams sometimes buy a tracing product expecting it to resolve a disagreement about quality standards, and no amount of instrumentation settles a question nobody has defined.

Sampled quality metrics are noisier than they look, and judge-model scores carry their own systematic biases — toward longer answers, toward answers that resemble the judge's own style. Treat a movement in a sampled score as a prompt to investigate rather than a conclusion, and validate any judge against human labels on a real sample before you trust it to gate anything. A quality number that has never been checked against a person is a number with unknown units.

Finally, none of this is free to run. Instrumentation is code you maintain, redaction rules drift out of date, and the storage bill is real. On a small system with a handful of routes and low traffic, a modest structured log with token counts and a route name will get you most of the value at a fraction of the effort, and moving to a full tracing stack before you have the volume to justify it is a reasonable thing to defer. The floor, though, is not zero: recording tokens and a route identifier is cheap enough that there is no good reason to run any production LLM feature without it.