Daily degradation trackers are too small to trust

Someone drops a chart in your team channel. A public tracker has been running a benchmark against Claude Code every day, and the line went down this week. The implied next move is to change something — pin an older model, rewrite your CLAUDE.md, switch providers, or just work less confidently.

Before you change anything, ask one question about the chart: how many tasks, how many times a day?

Why the sample size decides everything

A daily tracker that runs 50 tasks once per day is measuring a pass rate out of 50. The uncertainty on that number is large enough to swallow most of the movement you can see on the chart.

Rough 95% intervals for a pass rate near 50%, which is the noisiest case:

Read the last column, not the third one. A chart of daily scores is a chart of differences, and the difference between two noisy numbers is noisier than either. At 50 tasks a day, a 15-point drop overnight is inside the range you would expect from a coin-flip process that never changed at all.

The bar one benchmark author gave

On the Hacker News thread about one of these trackers, a co-author of SWE-bench argued the setup was underpowered and gave his own rough bar: on the order of 300 tasks, run five to ten times a day, and averaged. Below that, he said, day-to-day movement is ordinary noise — including noise from the provider's servers being loaded rather than anything about the model.

Treat that as a practitioner's rule of thumb stated in a forum comment, not as a published power calculation. The useful part is the shape of it: repeated runs of the same task set on the same day, averaged, is what separates "the model changed" from "today was a Tuesday."

The confound the chart cannot see

Even a well-powered tracker measures the whole system, not the model. A real drop in your day-to-day experience is consistent with several causes the chart cannot separate:

• A model change.
• A harness change — tool definitions, retry behavior, context assembly.
• A system-prompt change.
• Server load, which the official docs acknowledge indirectly: CLAUDE_CODE_RETRY_WATCHDOG exists specifically for capacity errors under load.
• Your own session. The docs are explicit that performance degrades as the context window fills, and that this is the constraint most best practices are built around. A long session that felt worse at hour three is the documented case, not the mysterious one.

That last bullet is the one worth checking first, because it is the only one you control.

What to do instead of reacting to the chart

1. Check the tracker's task count and runs per day before you read its conclusion. If it is not published, treat the chart as an anecdote with axes.
2. Compare the swing to the interval the tracker itself reports. If the swing is smaller, there is nothing to explain.
3. Look at your own session before blaming the model — context fill, a bloated CLAUDE.md, a pile of loaded MCP tools.
4. If you genuinely suspect a regression, keep a small fixed set of your own tasks and run it repeatedly on the same day, not once. Your own repeated runs on five real tasks tell you more than someone else's single run on fifty synthetic ones.

What is unverified here

Be clear about which half of this is solid.

Solid: the arithmetic. The interval widths in the table are the standard binomial half-width, 1.96 * sqrt(p*(1-p)/n) at p = 0.5, and the last column is that multiplied by sqrt(2) because a difference of two independent noisy numbers carries both their variances. Work it yourself and you get the same table; it does not depend on trusting anyone.

Not verified: everything about the specific tracker. Nobody here has audited its harness, confirmed its task count, or checked whether it re-runs the same tasks each day. The "300 tasks, 5–10 runs" bar is one person's stated threshold in a comment thread, not a result.

A real qualifier: if a tracker runs the same tasks every day, the day-over-day comparison is paired, and a paired comparison is tighter than the independent-sample column above — how much tighter depends on how many tasks flip outcome between runs, which is exactly the number these trackers rarely publish. That is an argument for asking about the design, not an argument that the design is fine.

What would settle it: the tracker publishing per-task, per-run raw results, so anyone could compute the paired interval directly instead of arguing about which formula applies.

Nothing in the official Claude Code docs addresses third-party degradation trackers, benchmark sample sizing, or how to evaluate a claim that the model changed. The docs' only nearby statement is that performance degrades as context fills, which is a different phenomenon.

Checkpoint

• A chart shows a 12-point drop from a benchmark of 50 tasks run once a day. What is the first number you ask for, and what do you conclude?

  ›Show answerAnswer

  Ask how many tasks and how many runs per day — which here you already have: 50 and one. At that size, near a 50% pass rate, the interval on a day-over-day change is about ±20 points, so a 12-point drop is inside what a process that never changed would produce. Nothing to explain, and nothing to act on.

  Two honest limits on that conclusion. The ±20 is the independent-sample case; if the tracker re-runs the same tasks daily the comparison is paired and can legitimately be tighter, which is why the move is to ask about the design rather than to declare the chart refuted. And underpowered cuts both ways — a tracker that cannot prove a regression cannot rule one out either.

• Name two causes other than a model change that would produce the same drop.

  ›Show answerAnswer

  Any two of: a harness change — tool definitions, retry behavior, context assembly; a system-prompt change; server load, which the docs acknowledge indirectly by shipping CLAUDE_CODE_RETRY_WATCHDOG for capacity errors under load; ordinary sampling noise at that sample size; and your own session's context filling up, which the docs state plainly degrades performance.

  Check the last one first. It is the only cause on the list you control, and a long session that felt worse at hour three is the documented case rather than the mysterious one. Note also that harness and system-prompt updates ship constantly and can move real quality with no model change at all — exactly what a coarse daily tracker is least able to detect or attribute.

• Why is the interval on a day-over-day change wider than the interval on a single day's score?

  ›Show answerAnswer

  Because a chart of daily scores is a chart of differences, and the difference of two independent noisy numbers carries both their variances. The single-day binomial half-width 1.96 * sqrt(p*(1-p)/n) gets multiplied by sqrt(2), which turns about ±14 points into about ±20 at 50 tasks. You can work the table yourself; it does not rest on trusting anyone.

  The qualifier is that this holds for independent samples. Re-running the same tasks every day is a paired comparison, tighter by an amount that depends on how many tasks flip outcome between runs — a number these trackers rarely publish, and the one to ask for.