How should a team handle flaky tests without teaching its AI agents to ignore failures?

TL;DR

Browser regression suites fail intermittently, and the quick fixes, retries and skipped tests, also teach every reader of the results, coding agents included, that a red run can be waved through. Treat each intermittent failure as an open question instead: reproduce it under recorded conditions, give it one owner and a written list of the release decisions it blocks, tell agents to report and stop rather than retry, and close it only when the same reproduction stops failing. The signal stays visible while the work happens.

When a retried failure reports as a pass

Playwright, a common runner for browser regression suites, sorts results into three outcomes: passed, failed, and flaky, which means a test failed and then passed on a retry (Playwright retries). Unless the run uses --fail-on-flaky-tests, a flaky result does not fail the run (Playwright CLI). Anyone reading only the exit code sees green, including an agent that runs the suite and reports back.

Martin Fowler described the human side of this in 2011: once people stop investigating failures in unreliable tests, failures in the healthy tests start being ignored as well (Eradicating Non-Determinism in Tests). Coding agents add a second channel. Claude Code loads CLAUDE.md at the start of every session, and its auto memory saves corrections you give it so they load in later sessions too (Claude Code memory). "Ignore the checkout test, it's flaky" can stop being a remark in one chat and become a standing instruction.

The question itself is old. In 2017 a tester on Software Quality Assurance Stack Exchange asked for general strategies for intermittent failures in a Protractor browser suite, noting that the usual outcome was disabling tests to keep moving. In February 2026 someone building browser automation tooling asked on Hacker News how teams manage flaky end-to-end tests at scale, how QA and engineering split the responsibility, and whether flakiness is a test problem, a product problem or infrastructure noise.

How often a flaky test is pointing at the product

Google reported in 2016 that about 1.5% of its test runs gave a flaky result, that almost 16% of its tests showed some flakiness, and that about 84% of the pass-to-fail transitions it observed involved a flaky test. The same post says developers sometimes dismiss a failure as flaky and later find it was a legitimate failure caused by the code (Google Testing Blog). In a study of 200 flaky tests that Mozilla developers had fixed, concurrency was the most common cause, and in 34% of those concurrency cases the developers had to examine production code rather than the test (Eck et al., ESEC/FSE 2019).

The evidence that separates them is mostly in a Playwright trace: the page state before and after each action, with console and network activity (Trace viewer).

Reproduce it before anyone retries or quarantines it

Retry and quarantine decisions should follow a reproduction record, not precede it:

Fifty is an example; a rare failure needs more runs, or timing made worse on purpose, as the last section shows.

Quarantine with a name and a list of blocked releases

Quarantine is a reasonable way to stop one unstable test from blocking everyone, as long as it has limits. Fowler suggested capping both the number of quarantined tests and how long each may stay there. Each entry needs:

Blocked decisions are the line most often left blank. If the quarantined test is the only automated check on a release-critical flow such as login or checkout, the release needs a substitute check or a named person who accepts the risk in writing. A quarantined test on a rarely used settings page may block nothing.

Rules an agent can follow without learning to shrug

Agents need the same discipline, written so they can follow it:

• On a failure, rerun the failing test once, alone and unchanged, before editing anything.
• If it passes, report it as suspected flaky with both results and stop. Do not edit the test, add waits or retries, or skip it.
• Never add a test to quarantine or raise a retry count; propose it, with the reproduction record, to the owner.
• Treat the quarantine list as tracking, not permission. A quarantined test failing in a change that touches its flow is a reason to look.

Claude Code's documentation is explicit that instruction files are context, not enforced configuration, so back the parts that matter with something the agent cannot change alone. Put the retry settings and the quarantine list under a CODEOWNERS entry and require code-owner approval before merging (GitHub code owners). Keep "known flaky" notes out of instruction files and agent memory; an exception belongs in the quarantine entry, with its expiry date. The article on what to save for a new AI coding session explains why durable notes and current state should be kept apart.

Paqad applies the first rule mechanically in its checks stage. Each test that fails in the parallel run is re-run by itself: one that fails alone blocks the change with its name, file and line, and one that passes alone is recorded in a flaky registry with suspected causes and, by default, only adds a warning to the change's result; a setting can make it fail the change instead. That tie-breaker separates failing alone from failing in a crowd. It cannot say whether a crowd-only failure is a real concurrency defect, and it assigns no owner. For the investigation itself, Paqad's root-cause analysis workflow writes one record with fixed sections and tells the agent never to present a guess as a confirmed cause. A well-organized record still cannot guarantee the diagnosis is right. Paqad's workflows overview lists that workflow beside the stages and records of an ordinary change.

Show the fix removed the mechanism, not just the symptoms

Fewer visible failures after a fix can mean the fix worked or that conditions changed. Two checks tell them apart.

First, make the failure deterministic before fixing it, when the trace allows. If it points at a slow response, hold that response in a page.route handler longer than the page expects (Playwright network). A test that failed now and then should now fail every time, so the fix is checked against a failure you can produce on demand, and the delayed version can stay in the suite. The article on getting an AI coding agent to follow a real test-first workflow covers reading why a deliberately failing test failed.

Second, rerun the original reproduction under the same conditions with retries off for that test, using test.describe.configure({ retries: 0 }) (Playwright test API). Zero failures in N runs is not proof of zero. As a rule of thumb it bounds the failure rate below about 3/N with 95% confidence, so 50 clean runs say "below about 6%", but only if the runs are independent and comparable. Repetitions on one machine share timing, data and load, so they say less than the arithmetic suggests. If the reproduction had shown, for example, 3 failures in 200 runs, you would need a few hundred clean runs before the result meant anything. The owner then closes the entry with both records attached.

Frequently Asked Questions