How to measure whether AI coding tools are working

Why the metrics you are given mislead

Acceptance rate is the most widely quoted and the least informative. It records that an engineer pressed tab, which happens when a suggestion looks right at the moment it appears. It says nothing about whether the code survived review, whether it was rewritten an hour later, or whether it introduced a defect that surfaced in a month. A tool that produces confident, plausible, subtly wrong code scores well on acceptance and badly on everything that matters. Worse, the metric is gameable in both directions without anyone intending to game it: engineers who accept and then heavily edit look identical to engineers who accept and ship.

Lines generated is worse, because volume and value are not merely uncorrelated in software but often inversely related. The best change is frequently the smallest one, and a tool that makes it cheap to produce a large diff has made review more expensive, not development cheaper. Any metric that rewards output volume will, given a quarter, produce more volume. It is worth saying plainly to anyone proposing it as a headline number.

Seats active measures that a licence is being used, which is a procurement fact rather than an engineering one. It is genuinely useful for one narrow purpose — telling you whether a rollout reached people at all, which matters because an evaluation of a tool nobody adopted measures nothing. Treat it as a precondition check, not as evidence of value. The pattern to avoid is the quarterly slide where adoption is up and to the right and no one asks whether anything shipped faster.

Start from the delivery metrics you already have

The four delivery metrics that the DORA research programme popularised are a reasonable spine for this, partly because they are well understood and partly because most organisations already collect them, which removes the temptation to invent a bespoke scorecard nobody trusts. Change lead time — from first commit to running in production — is the one most likely to move, and it is honest about the fact that writing code is only one segment of the path. Deployment frequency catches whether the team is shipping in smaller pieces. Change failure rate and time to restore are the counterweights that stop a speed gain being celebrated when it was bought with instability.

The reason this framing is worth insisting on is that it locates the effect in the right place. If an agent halves the time to write a change but the change then waits four days for review, lead time barely moves, and that is the correct answer rather than a measurement failure. It tells you the constraint is review capacity, which is actionable, and it heads off the common mistake of scaling up generation into a bottleneck downstream. A rollout that increases the volume of code arriving at an already-saturated review process makes delivery worse, and only a delivery metric will show you that.

Add one more that is not in the standard four and is specific to this question: review load, measured as time spent in review per change and per reviewer per week. It is the cost most reliably shifted rather than removed by AI coding tools, it lands on a small number of senior people, and it is invisible in every other metric here. If your tooling cannot produce it directly, an approximation from pull-request timestamps is good enough to spot a trend.

The questions that are specific to agent work

Once agents rather than autocomplete are in play, a few additional things are worth instrumenting, because they distinguish a workflow that is compounding from one that is quietly costing more than it saves. The first is task completion without human rescue: of the tasks delegated to an agent, what share landed reviewable without an engineer taking over mid-flight? This is the number that tells you whether delegation is real. It also segments usefully — by repository, by task type, by whether the target module has tests — and the segmentation is usually more actionable than the average, because it points at which surfaces are ready.

The second is rework rate: how often agent-authored code is substantially rewritten or reverted within some window after merge. Distinguish it from ordinary iteration by looking for changes that touch the same lines with a different approach rather than extending them. A high rework rate alongside a high acceptance rate is the classic signature of a tool producing plausible code that does not survive contact with reality, and it is precisely the pattern the vendor dashboard cannot show you.

The third is escaped defect attribution: when an incident happens, can you tell whether the change that caused it was agent-authored, and was it reviewed? This is uncomfortable to set up and politically sensitive, so it is worth being explicit that the purpose is to find which surfaces need better guardrails, not to assign blame to individuals. In practice a commit trailer or a pull-request label is enough, and the resulting data has settled more arguments in our engagements than any productivity figure.

Running a fair comparison without an experiment

You will not run a randomised trial. Teams are not interchangeable, engineers talk to each other, and holding a control group back from a tool everyone wants is both impractical and corrosive. What you can do is make the comparison honest about its own weakness, which is mostly a matter of controlling for the obvious confounds and saying out loud which ones you could not control.

The strongest available design is usually a staggered rollout: enable the tooling for one group, then another, then another, over a quarter or more, and compare each group against its own baseline rather than against the groups that have not started. Comparing a team to its own past controls for the team, which is the single largest source of variance. It will not control for seasonality, a reorganisation, or the large migration that landed in week six — so record those events on the same timeline as the metrics, and expect to explain rather than to isolate.

Set the window at a quarter at minimum. Shorter windows measure novelty: the first fortnight of any new tool shows an effect that is partly enthusiasm and partly people choosing easy tasks to try it on. A quarter is also roughly the time it takes for the second-order costs to show up, and those are the ones that determine whether the first-order gain was real. Be similarly careful about which work is included — comparing a greenfield service built with an agent against maintenance work on a legacy system measures the codebase, not the tool.

Counting the costs that get left out

The licence fee is the least of it and the only one most business cases include. Token spend is the first omission, and it is easy to get wrong in the same way production LLM spend is: it accrues per call, scales with how much context each agent run consumes, and nobody notices until the invoice. It needs attributing per team and ideally per task type, for the same reason production cost needs attributing per route — an unattributed total tells you nothing about which use is worth it.

Review and supervision time is the largest hidden cost and falls disproportionately on senior engineers, whose time is the scarcest thing in the organisation. If generation gets cheaper and review does not, the constraint moves to review and the throughput gain is smaller than the code-writing gain suggests. Measure it rather than assuming it: this is where an apparently positive rollout turns out to be net neutral.

Then the slower-moving ones. Rework is time spent twice. Incident cost from changes that should not have merged belongs on this side of the ledger, and so does the setup and maintenance of the platform itself — the rules files, the MCP servers, the guardrails, the person who owns them. Finally there is a cost with no good unit: comprehension debt, where a codebase accumulates code that works and that nobody on the team can confidently explain. We do not have a way to measure it and will not pretend otherwise; what we can say is that teams who invest in review standards early report less of it, and that a rollout evaluated on speed alone has no mechanism to notice it accruing.

Interpreting what you find

The most common honest finding after a quarter is a modest improvement in lead time, flat or slightly worse change failure rate, and a noticeable increase in review load. That is not a disappointing result; it is a diagnosis. It says generation improved, verification did not keep up, and the next investment should go into the parts of the system that make review cheaper — faster tests, tighter types, clearer boundaries, better pull-request hygiene — rather than into more seats.

The second common finding is that the effect is wildly uneven across the organisation, and the variation is explained by the codebase rather than by the people. Teams working in well-bounded, well-tested services see real gains; teams in the tangled legacy service see almost none and sometimes a regression. This is the most useful result the exercise produces, because it converts an abstract argument about whether AI coding works into a concrete, prioritised list of which repositories to make ready first.

A third pattern worth naming: gains that show up in throughput but not in outcomes. More changes shipped, no movement in whatever the team exists to move. That usually means the tooling accelerated work that was not the constraint, which is a portfolio question rather than an engineering one, and is worth escalating as such rather than solving with better prompts.

Reporting it without overclaiming

There is heavy pressure to produce a single percentage, and the single percentage is almost always indefensible. When a board hears that AI made engineering thirty percent faster, someone eventually asks whether thirty percent more shipped, and no it did not, and the credibility of the entire programme goes with the answer. The version that survives scrutiny separates what moved, by how much, over what window, for which teams, and states plainly what was not controlled for.

Report the counterweights alongside the gains, in the same table, at the same prominence. Lead time down, change failure rate flat, review hours up, token spend at a stated figure. A report that shows only the improving metric will be assumed to have been selected for, and it usually has been. Showing the full picture also protects the next decision, because the flat and rising numbers are what justify the follow-on investment in tests and guardrails.

Frame the conclusion as a decision rather than a verdict: given this data, here is where we expand, here is where we hold, here is what we fix first. Leaders are generally comfortable with directional evidence and uncomfortable with precision that turns out to be fake. Saying "we cannot isolate the effect, and here is the best read we have and why we believe it" earns more room to keep investing than a confident number that does not survive the second question.

Limitations of any measurement you build

Everything above produces correlation over a period in which many things changed. Team composition shifts, the product roadmap changes shape, the codebase itself is being modified by the very tools under evaluation. You are not going to isolate the causal contribution of one tool, and a measurement programme that promises to will produce a number and lose an argument later.

Measurement also changes what it measures. Announce that agent-authored changes will be tracked for defects and some engineers will stop labelling them; make review hours a headline metric and reviews get shallower. The mitigations are the ordinary ones — aggregate rather than individual reporting, an explicit statement that this informs investment and not performance review, and a preference for metrics that are awkward to game. None of them are complete.

Finally, some of the most important effects have no metric at all. Whether engineers enjoy the work more, whether the team understands its own system, whether a junior is learning or being carried — these matter to retention and to the two-year trajectory, and they do not appear in any of the numbers here. Ask about them directly in a survey or a skip-level, keep the answers next to the quantitative read, and resist the urge to convert them into a score.