Gate the bot on a test that fails before the fix

You asked Claude to fix a bug. It comes back with a diff, a new test file, and "all tests pass." You now have to decide whether to merge. The green suite tells you the repository is not obviously broken. It tells you almost nothing about whether the new test would have caught the bug you reported.

That gap is specific to agent-written code. The agent wrote the fix and the test in the same turn, usually the fix first. A test written after the implementation, by the thing that wrote the implementation, asserts the behaviour that now exists. It passes because it was authored against the current code, not because it pins the defect. You cannot tell the two apart by reading the diff, because both look like a reasonable test.

The stronger gate is a differential: the new test must fail on the revision before the fix and pass on the fix. That single extra run converts an unfalsifiable claim into an observation, and it moves your review question from "does this code work?" to "is this the right fix to merge?".

What each gate actually proves

The last two rows are the whole tip. Everything above them is satisfiable by an agent that never understood the bug.

Where to put the check

The docs give you three places to hang it, with different strength:

• A script-based Stop or TaskCompleted hook. The hooks docs show the shape already — run the suite, exit 2 with a message on stderr, and the message is fed back to Claude as feedback rather than shown as a wall. Swap "run the suite" for the differential script below.
• A /goal condition, when you want it for one session and no config. Note the real limit: /goal's evaluator judges your condition against what Claude has surfaced in the conversation — it does not run commands itself. So a goal can be satisfied by Claude reporting that the test failed pre-fix. That is a nudge, not a gate.
• CI, or a REVIEW.md rule if you use Claude Code Review — the code-review docs call out repo-specific rules like "new API routes must have an integration test" as landing more reliably there than in a long CLAUDE.md.

The script

Build the pre-fix revision in a throwaway worktree so the live tree is never touched, then apply only the test files to it.

#!/usr/bin/env bash
# differential-test.sh — assert the new test fails pre-fix and passes post-fix.
set -uo pipefail

BASE="${1:-origin/main}"
shift || true
TESTS=("$@")                      # paths to the new/changed test files
[ "${#TESTS[@]}" -gt 0 ] || { echo "FAIL: name the new test file(s)" >&2; exit 2; }

# 1. post-fix: must pass in the working tree
if ! npm test -- "${TESTS[@]}" >/dev/null 2>&1; then
  echo "FAIL: new test does not pass on the fix" >&2; exit 2
fi

# 2. pre-fix: same test, base revision, must FAIL
WT="$(mktemp -d)/pre"
git worktree add --detach "$WT" "$BASE" >/dev/null || exit 2
trap 'git worktree remove --force "$WT" >/dev/null 2>&1' EXIT
for t in "${TESTS[@]}"; do
  mkdir -p "$WT/$(dirname "$t")"
  cp "$t" "$WT/$t"
done
( cd "$WT" && npm ci --silent >/dev/null 2>&1 && npm test -- "${TESTS[@]}" >/dev/null 2>&1 )
if [ "$?" -eq 0 ]; then
  echo "FAIL: new test also passes on ${BASE} — it does not cover the bug" >&2
  exit 2
fi

echo "PASS: test fails on ${BASE}, passes on the fix"

Re-running it is safe: the worktree is removed by the trap, and nothing is written to the checked-out tree. Quote every path — a test file under a directory with a space in it will otherwise silently test nothing and exit 0.

What is unverified here

The practice is described publicly by Jarred Sumner as how Bun's issue-triage bot works — every incoming issue reproduced by an agent, and no PR submitted unless it carries a test that fails on the previous version of Bun. That is a report about one project's CI, from a recorded talk, transcribed by a third party. Nobody has measured how many bad agent PRs the gate actually stops, and the official docs do not mention the criterion at all: they show the mechanism (a hook that runs tests and exits 2, a /goal with a stated check, a REVIEW.md rule) and the prompt-level advice to write a failing test first, but never the fails-before/passes-after differential as a gate.

What would settle it for you, cheaply: on your next three agent-fixed bugs, run the script above before merging and count how many of the agent's own tests also pass on the base revision. If the answer is zero, your prompts are already producing real regression tests and the gate is ceremony. If it is one or more, you have found the class of PR the green suite was hiding.