Measuring honestly: accept rate, PR attribution windows, and the metrics that reward the wrong behaviour

Three weeks into the rollout Lesson 12.3 scoped, someone asks for the number. You open the dashboard and there is a good one: 34% of merged PRs tagged with Claude Code, up from nothing. You put it in a slide.

You have just made a claim you cannot defend. Not because the number is wrong — it is exactly what the dashboard says — but because you do not yet know what it counted, over what window, with which files excluded, or which of your team's merges it silently dropped. Someone will ask. This lesson is so you have an answer that survives the question.

Every level so far has drawn the same line: a claim is not evidence until you know how it was produced. Lesson 5.5 applied that to a green test run. This is the same discipline pointed at a chart with your team's name on it.

Three measurement surfaces, three different questions

Pick the wrong one and you spend a quarter collecting a number that cannot answer the question you were asked.

The docs put a gate in front of the choice:

Choose monitoring based on what you need to report on. The dashboards, APIs, and spend controls differ between Claude for Teams or Enterprise plans and Claude Console organizations, so check the Availability column before you plan your reporting around a capability. — Set up Claude Code for your organization

Check it before the plan, because documented gates can remove the interesting half of the dashboard entirely:

What accept rate actually measures

Suggestion accept rate: percentage of times users accept Claude Code's code editing suggestions, including Edit, Write, and NotebookEdit tool usage — Analytics

Read the unit. The denominator is tool uses, not sessions, not lines, not tasks. A one-character typo fix and a 400-line file rewrite each count as one decision. Three tool calls to land one change count as three.

So the number moves for reasons that have nothing to do with quality of output:

• A person who asks for one large edit and a person who asks for six small ones produce different accept rates from identical work.
• The scope is code editing suggestions. The docs name Edit, Write and NotebookEdit, and on the telemetry counter the tool_name attribute is exactly those three. A Bash command, a Read, an MCP call, the agent's whole plan: none of them is documented to reach this number.
• Where the decision is recorded, it is recorded as accept or reject and nothing more. Rejecting because you thought of something better and rejecting because the edit was wrong are the same datum.

What lines of code accepted leaves out

Lines of code accepted: total lines of code written by Claude Code that users have accepted in their sessions. This excludes rejected suggestions and does not track subsequent deletions. — Analytics

That second sentence is the whole lesson in miniature. Deletion is invisible to it. Code accepted in a session and deleted forty minutes later is still counted. A branch abandoned, a spike thrown away, a file generated and reverted — each one was accepted, and nothing afterwards subtracts it.

Which makes it a measure of volume produced, not of value shipped. It is not the same metric as the contribution figure beside it, either:

"Effective lines" is defined, and the definition is worth knowing before you compare two teams: only lines with more than 3 characters after normalization count, excluding empty lines and lines with only brackets or trivial punctuation. A codebase whose style puts closing braces on their own line does not score higher for it.

How a PR gets attributed, and the window that decides it

The tagging rule is more generous than most people assume:

PRs are tagged as "with Claude Code" if they contain at least one line of code written during a Claude Code session. — Analytics

One line. So "34% of merged PRs" means 34% of PRs contained at least one attributed line — it is a reach number, not a share of code. The share of code is a different reading of the same chart: the Pull requests breakdown toggles to a Lines of code view that splits the same merges by line count instead of PR count. The two are not interchangeable, and quoting the PR percentage while the room hears the line share is the misreading this section exists to stop.

The matching itself has three properties that decide your result:

The 21-day window is the one to write on the plan. A team that batches releases, or that keeps feature branches open for a month, will under-report against a team that merges daily, with no difference in how either uses the tool.

The exclusions: rewrites, generated files, long lines

Three categories of code never reach the numerator.

Code you improved. "Code substantially rewritten by developers, with more than 20% difference, is not attributed to Claude Code" (Analytics). Read that as a behaviour rule, not a footnote: the more carefully a developer reworks what Claude produced, the less of it the dashboard credits.

Auto-generated files, excluded as a category: lock files, Protobuf output, build artifacts and minified files, dist/, build/, node_modules/, target/, test fixtures, snapshots and mock data — plus "lines over 1,000 characters, which are likely minified or generated".

Anything outside the window, per the table above.

The docs are explicit about the direction of the resulting error:

These metrics are deliberately conservative and represent an underestimate of Claude Code's actual impact. Only lines and PRs where there is high confidence in Claude Code's involvement are counted. — Analytics

Which cuts both ways, and the second way is the one people miss. A conservative metric is a floor, so a low number proves nothing about low usage — but it also means the number is not a precise measure of impact in the other direction, and a rise in it is not automatically a rise in real output. It is a floor that moved. Say "at least", and you can defend every number you print.

Telemetry when you need more than the dashboard

Per-user token counts and cost estimates are not on the Teams or Enterprise dashboard. The docs send you elsewhere for them — to OpenTelemetry, or to the spend report in your organization's analytics settings, which lists token usage and estimated usage-credit spend per user and per model. (Console is the exception: its team insights table does show per-user spend and lines for the current month.) Neither of those is documented to break spend down by skill. OpenTelemetry is where that breakdown lives:

Break down by type (input/output), user, team, model, skill.name, plugin.name, or agent.name — Monitoring, on claude_code.token.usage

The cost counter carries the same attribution, which makes "what did the plugin we shipped in Lesson 12.2 actually cost us" a query rather than a guess. One constraint before you quote a figure from it:

Cost metrics are approximations. For official billing data, refer to your API provider (Claude Console, Amazon Bedrock, or Google Cloud's Agent Platform). — Monitoring

Quoted in a budget conversation, that is an estimate presented as an invoice. Label it, or pull the provider's number before anyone plans against it.

Know what you are collecting about people, too. Prompt content and assistant response text are not collected by default — only their length — and both are opt-in environment variables. When authenticated via OAuth, user.email is included in telemetry attributes; the docs' own suggestion is to work with your backend to filter or redact it if that is a concern. Decide before the collector is live, not after.

One setup failure belongs with that decision, for its shape rather than its detail. On the http/protobuf and http/json protocols, Claude Code declares a Content-Length on every export request. A band of versions before v2.1.212 sent them with chunked transfer encoding instead, and endpoints that insist on a declared length — Azure Monitor among them — rejected the lot as 411 Length Required or 400 (Monitoring). That one is fixed. Keep the failure mode, because the next transport bug will wear it: the exporter is rejected, no metric arrives, and the symptom is an empty dashboard nobody reports rather than an error anybody sees. Confirm one real number has travelled the whole path before the measurement plan from Lesson 12.3 starts depending on it — and confirm it again on the far side of a fleet upgrade.

Metrics that reward the wrong behaviour

Everything above is documented. What follows is this course's reading of it, and you should hold it to a lower standard of authority than the quotes — but the mechanism in each case is visible in the metric's own definition.

Lines of code accepted, as a target. It excludes rejections and never subtracts deletions. The cheapest way to move it is therefore to accept more and prune less. A developer who deletes 200 lines of dead code that Claude wrote last week improves the codebase and cannot lower the metric by doing so; a developer who accepts a generated 400-line scaffold nobody needed raises it. Nothing here is a bug. It is a volume counter being asked to be a value counter.

The leaderboard, as a ranking. It shows the top 10 users by contribution volume. The docs frame its purpose narrowly — find people who can share techniques, give feedback, and help onboard others. That is a talent-spotting tool. The moment it is read as a performance ranking, the people it names are the people who merged the most attributed lines, which is not the same population as the people who did the most valuable work.

Accept rate, as a proxy for quality. On the telemetry counter, each decision carries a source attribute, and two of its values are not people:

source: Where the decision came from. One of "config", "hook", "user_permanent", "user_temporary", "user_abort", or "user_reject". — Monitoring

"config" means the call was "decided automatically without prompting, based on project settings, allow or deny rules in the user's personal settings, enterprise managed policy, --allowedTools or --disallowedTools flags, the active permission mode" and more — and the event does not indicate which of those matched. So on that counter, an "accept" is not evidence that a human looked at anything. Broadening an allowlist, or a team standardising on a less interruptive permission mode, changes the population of decisions being counted. No doc states that this raises the dashboard's accept rate, and you should not assert that it does; what the attribute set establishes is that the number is not purely a record of human judgement, which is enough to disqualify it as a target.

The rewrite threshold, as an incentive. Attribution drops code a developer changed by more than 20%. A team told to raise "PRs with Claude Code" has been handed a reason to edit Claude's output less. That is precisely backwards from the verification discipline of Level 5 and the review workflows of Lesson 11.5.

The honest construction is the one the docs themselves recommend: use these numbers alongside something you already trusted.

Use alongside DORA metrics, sprint velocity, or other engineering KPIs to understand changes from adopting Claude Code — Analytics

A Claude Code metric paired with an outcome metric that predates the rollout is hard to game, because moving the first without moving the second is exactly the failure you were watching for.

The version that changed under your chart

One more source of dishonest numbers, and it is nobody's fault: the metric definitions move.

Of mcp_server.name on the cost and token counters:

Before v2.1.222, Claude Code set this attribute on every request after an MCP tool call, not only on requests that consumed a tool result, so dashboards that aggregate it show a step down after you upgrade. — Monitoring

Cost and token metrics have their own instance. Claude Code now counts each streaming response exactly once, including when a gateway or proxy behind ANTHROPIC_BASE_URL streams usage progressively across several frames — but before v2.1.214, streams that carried usage in more than one frame inflated claude_code.cost.usage and claude_code.token.usage by roughly one extra full request per extra frame. A step in your chart on the week of a fleet upgrade is a measurement change until proven otherwise. Record the version the baseline was taken on; Lesson 12.6 is where the team's version routine gets built.

Read both of those from where your data is standing rather than from where you are. Both corrections move the numbers down, and down is the dangerous direction: a fall in cost per developer in the week of an upgrade has exactly the shape of the result the rollout was hoping for, and it will be quoted before anyone checks. Running a build that has both corrections does not put the distortion behind you, because the numbers they changed are still sitting in the history your chart draws through — a cost baseline taken before v2.1.214 is the inflated one, and MCP attribution collected before v2.1.222 is the over-broad one. Write the version beside the baseline on the day you take it.

Writing a falsifiable claim

A measurement claim is defensible when someone who disagrees with you could go and check it. Most slide-deck claims cannot be checked at all:

The defensible version of the third one, written out in full:

In August, at least 34% of merged PRs in payments-api contained one or more
lines attributed to Claude Code (21-day attribution window, generated files
and lock files excluded, conservative matching, so treat it as a floor).
Over the same month, median PR cycle time fell from 31h to 24h, measured in
our own GitHub data, which we have had since March and did not change.
If cycle time had not moved, we would have reported the 34% as adoption and
made no productivity claim at all.

Longer than anyone wants a bullet to be. Write it once, in the document behind the slide; then the slide can say the short version honestly, because there is something to point at when the question comes.

The standard, restated: name the number, name what it counts, name the window, and name the result that would have made you say the rollout did not work. A claim with no falsifier is not a measurement. It is a preference with a percentage attached.