How can I tell whether AI-generated tests catch real bugs or just increase coverage?

TL;DR

A coverage report tells you which lines ran during the tests, not whether any test would notice those lines doing the wrong thing. When an agent writes both the code and its tests, find out by breaking the behaviour on purpose and checking that the suite fails. Read each test for what it asserts, add rejected inputs and stored-state checks, seed faults by hand or with mutation testing, and use coverage only as a map of what nothing runs.

Three tests, every line covered, two broken versions passing

The quickest way to see the problem is a small example. Below is a complete Express handler for a signup endpoint, written for this article: it validates an email address and a seat count from 1 to 10, rejects a duplicate email, and saves the signup to a store.

app.post('/signups', (req, res) => {
  const { email, seats } = req.body ?? {};
  if (typeof email !== 'string' || !email.includes('@') ||
      !Number.isInteger(seats) || seats < 1 || seats > 10) {
    return res.status(400).json({ error: 'invalid signup' });
  }
  if (store.has(email)) {
    return res.status(409).json({ error: 'already signed up' });
  }
  const signup = { id: store.size + 1, email, seats };
  store.set(email, signup);
  return res.status(201).json(signup);
});

Next to it sits a thin suite of the kind the questions below worry about, also written for this example: a valid signup expecting status 201 and an id, a request without an email expecting 400, and a duplicate expecting 409. Then two single-line faults were seeded, one at a time: seats < 1 became seats < 0, so a signup for zero seats is accepted, and the store.set line was commented out, so nothing is saved.

One handler says nothing about how often generated tests look like this. It does show the same coverage figure for a working endpoint and for one that loses every signup.

What the coverage number measures

In October 2025 a Cursor user asked on the Cursor forum whether tests written by the agent are really of better quality than the code. Almost all of their code is agent-written, they keep coverage at 75% or more so the agent can catch its own bugs, and the product has been solid, but the tests come from the same agent. In September 2026 a developer asked on Stack Overflow how to tell whether an AI tool's fix really implements a bug report when every existing test passes. Their example: the tests check that users A and B can each read their own records, and nothing checks that B cannot read A's.

Both questions come down to what a passing, well-covered suite is evidence of. Line, branch and function coverage count what executed. An assertion that only checks the status code executes the same lines as one that checks the stored record. One large study, by Inozemtseva and Holmes (ICSE 2014), generated 31,000 test suites for five large Java systems and found only a low to moderate correlation between coverage and fault detection once suite size was controlled for. Stronger kinds of coverage did not do better. The authors conclude that coverage is useful for finding under-tested code and should not be used as a quality target. They measured effectiveness with mutation testing, the technique recommended below, not with real-world bugs.

Read each test for what it would notice

Before running anything, ask of each assertion: if this behaviour were wrong, would this line fail? These patterns run lines without checking much:

Then list the requirement's user behaviours, including the refusals, and mark which ones a test would fail on. The unmarked ones are the gap, whatever the coverage report says.

Add the negative cases and state checks that pin behaviour down

The four tests that caught both faults look at different things rather than reaching more lines. On the correct handler they reported 100% line coverage and 88.89% branch coverage, slightly less than the suite that missed everything. They check:

The second test is the one the zero-seat fault failed:

test('rejects zero seats and stores nothing', async () => {
  const res = await post({ email: 'a@example.com', seats: 0 });
  assert.strictEqual(res.status, 400);
  assert.strictEqual(store.size, 0);
});

For a real endpoint, derive the cases from each rule rather than from the code: a value just inside and just outside every limit, a check of stored and unchanged state for every write, and for every permission the case the Stack Overflow question was missing, one user reaching for another's data. The article on defining done so an agent cannot mistake a partial feature for finished covers writing criteria that a placeholder would fail; the same criteria make good negative tests.

Break the code on purpose

A seeded fault answers the question directly. Pick a behaviour the change is meant to deliver, make the smallest edit that breaks it, run the suite, and write down what happened before you revert. If the suite stays green, no test detected that fault with the inputs the suite uses, however many lines ran. Confirm the edit really changes the behaviour first: an edit with no observable effect proves nothing either way. Keep the list of faults with the change: the boundary moved, the save removed, the permission check skipped.

Mutation testing does the same thing mechanically. StrykerJS, the JavaScript and TypeScript tool, changes the code in many small ways and reruns the tests against each change (Stryker documentation). A mutant is killed when at least one test fails, survived when all tests pass, and reported as no coverage when no test runs that code at all; the mutation score is the percentage detected (mutant states and metrics). Incremental mode reruns only mutants in changed code and tests, which suits a pull request, though it cannot see dependency, environment or snapshot changes (incremental mode).

The survivors matter more than the score. Each names a line and a change no test noticed, a concrete request to hand back to the agent: write a test that fails on this mutant. Meta describes doing this at scale in its ACH system, which generates faults specific to a concern such as privacy and then has a language model write tests that catch them (Foster et al., 2025); engineers accepted 73% of its tests in internal test-a-thons. Some survivors are equivalent mutants that cannot change behaviour, and mutation operators are small edits, not the bugs people actually write, so review survivors rather than chasing 100%.

What to put next to coverage before calling a feature ready

Keep the coverage report for what it is good at, showing code that no test runs, and ask for evidence about behaviour as well:

• A behaviour list mapped to tests, including the refusals, with any behaviour that has no failing-on-error test named as a gap.
• Seeded-fault results for the changed behaviour: each fault, the test that failed, and any fault that survived with the reason it was accepted.
• Surviving mutants in the changed files, each either killed by a new test or marked as equivalent, where the project runs mutation testing.
• A test review by someone who did not generate the code and the tests in the same session, reading assertions rather than counting them. The article on getting an agent to follow a real test-first workflow covers keeping tests ahead of the implementation and checking that a new test fails without the change.

Paqad keeps a green run in proportion. Its checks stage runs the project's own test command and records per-test results, and on Claude Code and Codex CLI the end-of-turn verdict reads that saved report rather than the chat; its documentation states that this evidence does not prove test coverage is sufficient. Each module's health record keeps coverage next to a mutation score, filled only when mutation testing has run and left empty rather than guessed otherwise. A module with high coverage but no defect history yet reads as fragile, not stable. Paqad does not judge whether an assertion is meaningful or write the missing tests. Paqad's module health page lists what each record holds and where its numbers come from.

Frequently Asked Questions