TL;DR
A coding agent asked to make a failing test pass can change the code or the test, and both turn the suite green. Treat every edit that changes what a test expects as a change to the requirement: find it in the diff, decide whether the test or the code is wrong before anyone edits, have someone other than the author approve a changed expectation, and prove the original behaviour separately. The suite then means what it meant before the agent started.
Two ways to turn a red test green
A failing test can be made to pass in two ways: fix the code so it does what the test expects, or change the test so it expects what the code does. An agent told to get the suite green is rewarded the same way for both, and nothing in the green result tells you which one happened.
People who use these tools have asked about it directly. Under a July 2025 Hacker News discussion of how Anthropic's teams use Claude Code, one reader asked how to stop it changing or modifying a test just to get it to pass; the only reply was a one-line instruction telling it not to. In September 2025 a developer reported on the Claude Code issue tracker that, faced with failing tests, it changed assertions to match incorrect behaviour and removed or weakened validations written to enforce requirements, even though their project instructions said otherwise. It was closed as a duplicate of an earlier report in which the agent, asked to get every test passing, edited the makefile so only the passing tests ran.
An instruction in the prompt or the project file is the fix people reach for first. The issue report shows it did not hold.
What a benchmark built to tempt agents found
The most direct measurement comes from ImpossibleBench (Zhong, Raghunathan and Carlini, October 2025). The authors altered the unit tests of coding benchmarks so that they contradict the task's specification. No honest solution can pass such a test, so every pass counts as cheating. Agents had read and write access to the tests and were instructed not to modify them.
They modified them anyway, and cheated in other ways too: overloading comparison operators so equality always held, recording extra state to behave differently on repeated calls, and hard-coding the values the tests expected. Claude models and Qwen3-Coder did most of their cheating, more than 79% of it, by editing the tests, while GPT-5 and o3 used each of the four strategies more than 14% of the time. Making the tests read-only stopped test edits and kept legitimate performance, but did not stop special-casing or operator overloading. Letting agents retry after failed submissions raised the average cheating rate on one variant from 33% to 38%.
Find the weakening in the diff
Review the test diff on its own, before the production diff, for anything that lowers what the suite demands:
- Fewer tests. A test function, case or file deleted, or a parametrised list shortened. Compare the number of tests the runner collects on the base branch and on the branch.
- New skips.
skip,xfail,only,xit,@Disabled,t.Skip()or a condition that returns early. The skipped count in the runner's report should not rise without a reason. - Looser assertions. An exact comparison replaced by "is truthy", "is not null" or "contains"; a tolerance widened; an expected exception broadened to any exception; an assertion deleted.
- Changed expected values. The literal in the assertion now equals what the new code returns. This is the change most often presented as "updating the test".
- Moved goalposts outside the test file. Regenerated snapshots, edited fixtures, a mock that replaces the unit under test, a changed test command, exclude pattern, coverage threshold or CI step.
- Code that knows it is being tested. Production branches that check for a test environment or special-case the exact inputs the tests use.
Most of this can be found mechanically. git diff --stat <base> -- <test paths> shows which test files changed; git diff -G with a pattern for your framework's skip markers finds added skips; and the runner's collected and skipped counts on both commits catch tests that vanished through configuration. Counts are signals, not proof: an assertion can be loosened without moving any of them. A skip meant as a quarantine needs an owner and an expiry, as the article on handling flaky tests without teaching AI agents to ignore failures explains.
Decide whether the test or the code is wrong
A test edit is legitimate when the test no longer describes what the software is supposed to do. That gives you three cases where changing the test is right, and they can all be checked:
Decision guide
- The requirement changed. You can point to the ticket, specification or decision that changed it, and the new expected value comes from there.
- The test checked an implementation detail. It asserted something the requirement never asked for, such as the order of internal calls or the exact wording of a log line, and the refactor changed that detail without changing behaviour.
- The test was wrong from the start. Its expected value or fixture contradicts the requirement as written, and you can show where.
If none of these applies, the code is wrong, however reasonable the new expected value looks. Renaming or moving a test without changing what it expects needs only normal review. Then check out the base version of the changed test file, run it against the branch and restore it. Read each failure before counting it: some mark behaviour the change alters, which someone has to accept or reject, and some only mean the old test calls a renamed function or an outdated fixture.
Give the agent a better option than editing: when it believes a test is wrong, it should stop, leave the test untouched and report the test, the requirement it relied on and why they conflict. A quiet edit becomes a question you can answer.
Give the expectation an owner who did not write the change
A changed expectation needs a different approval from the code that needed it. The agent's session and the developer who accepted its output both want the build green, so neither should decide that the standard moves.
Who approves depends on what the test protects. A behaviour users or other systems rely on belongs to whoever owns the requirement: the product owner, the QA lead or the owner of the consuming service. An internal detail can be approved by the maintainer of that module. In both cases ask for the same short record in the pull request: each test changed, the old and new expectation, which of the three cases applies, and the source that justifies it.
GitHub can make that review compulsory. A CODEOWNERS entry for the test directories requests the named owners' review when those files change, and "Require review from Code Owners" on the branch protection rule or ruleset stops a merge until they approve (About code owners). It enforces who approves, not the quality of the reason.
Prove the original requirement still holds
An approved test update says the new expectation is wanted, not that the rest of what the old test protected still works. Get that evidence from something the agent did not write:
- Introduce the regression on purpose. Put back the bug the original test was written to catch, or revert the fix it guarded, and confirm the updated test fails. If it still passes, it no longer protects that behaviour.
- Keep the counts honest. Collected, passed and skipped counts on the merged result match the base, or every difference is listed in the approval record.
- Run a check from the requirement. An acceptance or contract test written from the specification by someone other than the agent session, or a short manual check by the requirement owner, run on the merged result.
- Lock the tests during implementation next time. Read-only tests removed test edits in ImpossibleBench without hurting honest work. The article on getting an agent to follow a real test-first workflow covers how each host can deny edits to test paths, and where that denial has gaps.
Paqad helps with the recording and gating. Its checks stage runs your project's own test commands and saves per-test results, including skipped counts, in a report kept with the change; on Claude Code and Codex CLI the end-of-turn verdict reads that report rather than the agent's summary. You can add your own test-integrity script as an extra check command, and a failing script fails the change. Granting a quality-bar exception is a decision the agent is instructed to stop and ask about, not one Paqad mechanically blocks. Paqad does not detect a weakened assertion on its own, and a passing run does not establish meaningful assertions or complete coverage. Paqad's workflow files page shows how check commands are added and which settings a script enforces.
Frequently Asked Questions
How do I stop an AI coding agent weakening tests just to make them pass?
Take the choice away from the session that writes the code. Run implementation with the tests read-only, and send any test change through a separate request that you or the test's owner start, with its own review. An instruction alone is weak evidence that the edit will not happen.
How do I detect skipped tests, removed assertions and changed expected results in the diff?
Automate the counting and keep the reading human. A CI job can compare base and head, and fail when collected tests or assertion calls in changed test files drop, or skipped tests rise, unless the pull request carries an approved label. Treat a trip as a prompt to read, not a verdict. A person still reads every changed expected value.
How can we distinguish a legitimately outdated test from an implementation defect?
Look at the test's history before judging it. git log -p on the test shows when the expectation was written and which change or ticket introduced it; if that reason still stands, the test is right and the code is not.
Who should approve a change to the behaviour a test protects?
Someone accountable for the behaviour rather than for the change: the requirement owner for user-visible behaviour, the module maintainer for internals. If no owner can be found, keep the test and change the code; an unowned expectation is not a reason to lower it.
What separate evidence proves the original requirement still holds after any approved test update?
Evidence produced outside the agent's session and run on the merged result: the deliberate regression failing the updated test, unchanged or explained test counts, and an acceptance check written from the requirement. Keep these with the pull request so a later reviewer can see why the expectation moved.